# Two is better? Combining EEG and fMRI for BCI and neurofeedback: a systematic review

Mathis Fleury, Patrícia Figueiredo, Athanasios Vourvopoulos, Anatole Lecuyer

► To cite this version:

Mathis Fleury, Patrícia Figueiredo, Athanasios Vourvopoulos, Anatole Lecuyer. Two is better? Combining EEG and fMRI for BCI and neurofeedback: a systematic review. Journal of Neural Engineering, 2023, Journal of Neural Engineering, 20 (5), pp.051003. 10.1088/1741-2552/ad06e1. hal-04483806

HAL Id: hal-04483806

https://hal.science/hal-04483806v1

Submitted on 7 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Two is better ? Combining EEG and fMRI for BCI and Neurofeedback : A systematic review

Mathis Fleury*<fnref n="1" />,<fnref n="2" />, Patrícia Figueiredo<fnref n="2" />, Athanasios Vourvopoulos<fnref n="2" />, and Anatole Lécuyer<fnref n="1" />

Correspondence*:

Mathis Fleury
IST / Lisboa
ISR-Lisboa/LARSyS and Department of Bioengineering
Universidade de Lisboa, Lisbon
mathis.fleury@inria.fr

## ABSTRACT

Electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) are two commonly used non-invasive techniques for measuring brain activity in neuroscience and brain-computer interfaces (BCI). While EEG has high temporal resolution and low spatial resolution, fMRI has high spatial resolution and low temporal resolution. In this review, we focus on the use of EEG and fMRI in neurofeedback (NF) and discuss the challenges of combining the two modalities in order to improve understanding of brain activity and achieve more effective clinical outcomes. Advanced technologies have been developed to simultaneously record EEG and fMRI signals in order to better understand the relationship between the two modalities. However, the complexity of brain processes and the heterogeneous nature of EEG and fMRI present challenges in extracting useful information from the combined data. We will survey existing EEG-fMRI combinations and recent studies that exploit EEG-fMRI in NF, highlighting the experimental and technical challenges. We will also identify remaining challenges in this field.

**Keywords:** EEG, fMRI, BCI, NF, brain imaging, multimodal, review

## 1 INTRODUCTION

The study of the simultaneous combination of the EEG and fMRI is an expanding area of research, particularly under the impetus of studies on sleep or epilepsy (Jorge et al., 2014). Their integration into studies on neurofeedback (NF) is rather recent and was pioneered by Zotev et al. (Zotev et al., 2014). By using multiple modalities together, we can overcome limitations of individual modalities, gather more detailed and precise information about brain activity, and create more effective neurofeedback approaches. Specifically, using both electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) is particularly promising as it combines the strengths of both methods in terms of temporal and spatial resolution.

To date, EEG is the most common non-invasive brain signal acquisition technology, especially in the domain of brain-computer interfaces (BCI) and neurofeedback where it remains the most popular brain

<footnote n="1">¹Univ Rennes, Inria, CNRS, Inserm, Empenn ERL U1228, Rennes, France</footnote>

<footnote n="2">²ISR-Lisboa/LARSyS and Department of Bioengineering, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal</footnote>
imaging method. Scalp EEG uses surface electrodes for capturing —over a layer of bone and tissue— the combined electrical activity of populations of excitable pyramidal neurons (da Silva, 2013). These “firing” neurons produce electrical and magnetic fields that generate ionic currents that are often modelled as electric dipoles. These dipoles, which can be characterised by their electromagnetic field, can be measured respectively by EEG and magnetoencephalography (MEG). Brain activity can also be measured indirectly thanks to the haemodynamic activity, in fact the cerebral activity induces changes in oxygen concentrations in order to supply energy to the neurons, known as the haemodynamic response (HRF), which is characterised by changes in blood flow and oxygenation aimed to supply the required energy to the neurons (Logothetis, 2008). The relationship between oxygen-rich and oxygen-poor blood, generates changes that can be detected by fMRI through the blood oxygen level dependent (BOLD) contrast or functional near-infrared spectroscopy (fNIRS) (Bandettini et al., 1992).

However, these modalities have different properties; their spatial and temporal resolution varies greatly. fMRI is characterised by a very high spatial resolution of the order of a millimetre but a low temporal resolution (about 1s) as opposed to EEG and MEG which have a high temporal resolution but have a low spatial resolution (a little less with MEG). This complementarity, which already seems to be taking shape, goes even further because EEG and MEG measurements are mostly sensitive to more superficial brain sources while fMRI can detect activity from deep areas of the brain.

Because of the complementarity (in time and space) of EEG and fMRI, advanced technologies focused on the integration and simultaneous recording of EEG and fMRI signals to provide bi-modal setting (Nunez and Silberstein, 2000; Lei et al., 2012). This new technology opened a new research field increasing knowledge about the spatio-temporal dynamics of brain function (Jorge et al., 2014). In order to better illustrate the problem linked to this multimodal integration, we present on the Venn diagram by Biessman and colleagues, which shows easily the benefits and pitfalls of the integration of EEG and fMRI (see Figure 1). Indeed, the integration of these two modalities also has its limits, due to their very heterogeneous nature and the fact that brain processes are very complex systems that depend on many latent phenomena meaning that simultaneously extracting useful information from them is not an evident task. That is why the challenge of understanding the relationship between EEG and fMRI is not fully accomplished (Dong et al., 2018).

In this review, we will therefore focus on the contribution of the two most widely used non-invasive imaging methods in the field of neurosciences and more precisely for NF and BCI. Indeed, although fMRI answers the questions of *where* and EEG answers the question of *when*, then what is the real scientific question that implies the fusion of EEG/fMRI? and how can we use this fusion to report brain activity and lead to more efficient neuroadaptive changes and more effective clinical outcomes (Stoeckel et al., 2014).

The remaining of this review is organised as follows: firstly, we provide an survey of existing EEG-fMRI combinations in Section 2. Secondly, we surveyed recent studies exploiting EEG-fMRI as neuroimaging modalities in NF in Section 3, showing the experimental and technical challenges. These articles are then discussed in Section 4, where we also identify some remaining challenges.

# 2 WHY AND HOW SHOULD EEG AND FMRI BE COMBINED FOR NF PURPOSE ?

## 2.1 General Properties of EEG & fMRI Modalities

This section describes the properties of EEG and fMRI, their advantages and disadvantages. The aim of this section is thus to shed light on the limitations of both modalities and advantages of their combined use.
<figure><img src="image_2.png" /></figure>

**Figure 1.** Venn diagram of EEG-fMRI neuroimaging analysis methods (adapted from (Biessmann et al., 2011)); certain aspects of the brain activity are reflected in electrophysiological recordings (EEG) and others in hemodynamic measurements (fMRI). While some aspects such as fast neuronal oscillations are only detectable in electrophysiological signals (area 1), other aspects (such as activity in deep brain structures) are easier to investigate using BOLD signal (area 2). Aspects that are reflected in both modalities can be subdivided into signals originating from neural activity (area 3) and non-neural physiological processes reflected in both modalities, such as muscle contractions that lead to head movement (area 4). Besides these common artefact sources, there are many artefacts that are reflected in one modality only (area 5 and 6).

### 2.1.1 EEG modality

EEG is one of the most widely used non-invasive brain imaging technique for studying brain activity (the most widely used for NF). EEG measures electrical brain activity caused by the flow of electric currents during synaptic excitations and inhibitions of neuronal dendrites, mainly in the superficial layers of the cortex. It is therefore a direct measure of electrical activity. The propagated electrical signals are measured by electrodes located on the scalp and each of them allows to measure a spatio temporally smoothed version of the local field potential (Nunez and Silberstein, 2000), integrated over an area of 10 cm² or more (Buzsáki et al., 2012).

Scalp EEG activity shows oscillations at a variety of frequencies and several of them show some characteristics in terms of frequency ranges, and spatial distributions and are associated with different states of brain functioning (e.g., waking and the various sleep stages). In the literature, EEG is typically described in terms of activity types namely rhythmic activity and transients. These rhythmic activities can be divided into certain frequency bands which are shown to have certain biological significant or certain distribution over the scalp. Six types of frequency bands can be identified: Delta (0.5-4 Hz) which is associated with deep sleep and wake up states; Theta (4-7 Hz) which is generated with idling, creative inspiration, unconscious material, drowsiness, and deep meditation; Alpha & Mu (8-13 Hz) which is associated with relaxation, concentration, and sometimes in attention and Mu is a centrally located alpha frequency that represents the sensorimotor cortex, it should be noted that while it resembles the alpha rhythm, it is not affected by eye opening; Beta (12-30 Hz) which is associated with motor behaviour and is generally attenuated during active movements; and Gamma (>30 Hz) which could be detected at somatosensory cortex and is also shown during short term memory matching of recognised objects, sounds, or tactile sensations (Tatum IV, 2014).
Generally, neuroimaging modalities are considered to be divided into two categories: invasive and non-invasive. One way to record better signals is to use electrodes implanted in the brain. This invasive technique allows the exploration and recording of electrical events in deeper regions thanks to metal or glass electrodes (Electrocorticography). However, the use of invasive electrodes implies significant drawbacks due to the risk of performing surgery and the apparent gradual degradation of the recorded signals.

EEG has been widely used in NF over the years to induce long-lasting behavioural changes thanks to its relatively low cost and portability, both in healthy volunteers and in patients (Gruzelier, 2014).

### 2.1.2 fMRI modality

In the field of cognitive neuroscience, fMRI has become the go-to mainstay as a non-invasive brain imaging method. Thanks to a much higher spatial resolution than EEG, fMRI provides unparalleled access to detailed patterns of activity in the human brain (both cortical and subcortical target regions). Here we will focus not on fMRI in general but rather on its real time aspect, real-time fMRI (rtfMRI), which permits a non-invasive view of brain function (thanks to BOLD contrast) in vivo and in *real time*. We talk about *real time*, but in reality it is rather the ability to capture the brain signal every 1-2 s. In addition, blood oxygenation level changes 2 to 6 seconds after the stimulus. HRF is used to model BOLD signal (LaConte, 2011; Logothetis et al., 2001).

Indeed, contrary to the EEG which directly measures neuronal activity, fMRI is an indirect measure of brain activity because it does not measure neuronal activity exactly but rather the consequences of neuronal activity. The pathway from neural activity to the fMRI activity map is schematised in Figure 2.

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Schematic of the detection of neural response to a stimulus with fMRI BOLD signal. From (Arthurs and Boniface, 2002)</figcaption></figure>

In general, it is common to consider that the spatial resolution of fMRI is high, but it is especially high compared to the EEG since its spatial resolution is typically of up to 2 mm³ (for each voxel); a volume that would encompass approximately 187 134 neurons in cortex (Lent et al., 2012). However, this value can be improved by suppressing macrovascular signals and contrasting different experimental conditions appropriately (Sitaram et al., 2017).

These properties can be valuable for NF applications (Stoeckel et al., 2014). Recent studies suggest that fMRI is a mature technology to use in the context of NF training (Ruiz et al., 2014; Weiskopf, 2012) NF-based fMRI (or fMRI-NF) works by providing a feedback representing HRF in a given ROI. The advantages of fMRI in terms of spatiality are substantial for NF because it makes it possible to reinforce engagement or regulation of these specific ROIs (Ramot et al., 2017). A review on the design of fMRI-NF studies can be found in (Fede et al., 2020).
## 2.2 EEG-fMRI multimodal Integration for NF

### 2.2.1 General properties of Fusion of EEG and fMRI features

In this section, we will discuss the different multimodal analysis methods for EEG-fMRI features. The studies mentioned below are therefore not all related to the NF but could underline the dearth of studies. A review on simultaneous EEG-fMRI can be found in (Lei, 2019). This categorisation is based on the work of Biessmann and colleagues (Biessmann et al., 2011), who have classified these methods according to the type of analysis used. Two main fusion features are reported: Asymmetric and symmetric fusions, that is dividing analysis methods in those that bias one modality towards features extracted from the other modality (asymmetric approaches) and methods that try to analyse both modalities at the same time (symmetric approaches); and activation and connectivity analysis, that is dividing single brain regions and network analysis.

#### 2.2.1.1 Asymmetric and symmetric fusion :

Multimodal data integrations are categorised as symmetrical and asymmetrical. Asymmetric approaches mainly use one of the modality information to bias the brain activity estimates of another modality (Figure 3). Most asymmetric methods can be seen as regression, a form of supervised learning (Biessmann et al., 2011). The most influential asymmetric fusion methods include time prediction, that is: asymmetric EEG Based fMRI analyses where fMRI analysis is based on the temporal information of EEG where a specific EEG feature is convoluted with the HRF to model the fMRI waveform (Debener et al., 2005); or spatial constraints, that is: asymmetric fMRI Based EEG Analyses where the EEG imaging is based on the spatial prior of fMRI: the EEG source reconstruction is constrained by the spatial activity information obtained from the fMRI (Lei et al., 2011, 2012). Symmetric approaches try to analyse both modalities at the same time by establishing a common generative model or the use of interactive information to explain two modalities (Trujillo-Barreto et al., 2001; Valdes-Sosa et al., 2009). The interest of this approach is to overcome the problems associated with asymmetrical approaches, since during the selection of the characteristics of the "primary" modality (the one used to bias the estimation of the other modality), characteristics may be selected that are not reflected at all in the other ("secondary") modality. For example, a large vessel might lead to highly active voxels in an fMRI time series, but the dipole source of interest does not necessarily coincide with this very location (Biessmann et al., 2011). Different methods can be used in symmetric analyses to address this problem, such as model driven and mutual information (Valdes-Sosa et al., 2009; Daunizeau et al., 2007, 2009; Ostwald et al., 2011).

#### 2.2.1.2 Activation and Connectivity Analyses :

Another categorisation to be taken into account in the EEG-fMRI analyses is the distinction between studies that locate activation patterns and those that investigate functional connectivity between regions. Although the majority of the studies are activation based, it is also interesting to see how specific brain regions interact together. Indeed, many cognitive processes require more than one active brain area and if brain areas interact they will show correlated activity. In fact, most of the processes so far examined with fMRI studies (e.g., emotion processing, motor response, language, pain perception, etc.) include the coordinated activity of several brain regions (Ruiz et al., 2014, 2011). Most functional connectivity studies in the context of fMRI are based on correlations between voxels (see, e.g., (Biswal et al., 1995; van de Ven et al., 2004)). In (Biswal et al., 1995), univariate correlation coefficients are used to quantify functional connectivity. In (Friston et al., 1993), Principal component analysis (PCA) is used to reveal connectivity patterns. Also Independent component analysis (ICA) is often used for functional connectivity (van de Ven et al., 2004).
<figure><img src="image_4.png" /></figure>

**Figure 3.** Multimodal methods as categorised into asymmetric or symmetric approaches (grey indicates optional nodes); in asymmetric analyses features from one modality are used to improve brain activity estimates of another modality, sometimes via a generative model of the latter modality (From (Biessmann et al., 2011)).

## 2.2.2 Classification of NF-based EEG-fMRI studies

In the literature, we can find different ways of combining EEG and fMRI for NF, simultaneously or not. In order to classify these different combinations, we will base ourselves on the taxonomy elaborated by Perronnet et al. (Perronnet, 2017). In this taxonomy, we can find two categories, the one that uses one modality as NF signal, such as Passive fMRI during EEG-NF (pfMRI) or Passive EEG during fMRI-NF (pEEG), EEG-informed fMRI NF or EEG Finger-Print NF (iEEG) and fMRI before/after EEG-NF (efMRI); and the one that uses both modalities as NF signals, such as EEG-fMRI-NF that refers to simultaneous EEG-fMRI-NF (EEG-fMRI). These combinations do not have the same technological implications and difficulties. But share sometimes similar objectives, such as overcoming the cost-intensive, cumbersome and tiring aspects of fMRI, while keeping a good specificity in EEG-NF training (Thibault et al., 2018).

### 2.2.2.1 fMRI before/after EEG-NF (efMRI) :

Using fMRI before and after EEG-NF can be used to evaluate functional outcomes of the EEG-NF training such as connectivity or change in cerebral plasticity. This can be done during the resting state (Ros et al., 2013).

### 2.2.2.2 Passive fMRI during EEG-NF (pfMRI) :

Recording passive fMRI during EEG-NF allows to evaluate and validate the EEG-NF protocol and to find BOLD correlation of the EEG-NF training. The main disadvantage is that MR artefacts affect the EEG
signal and therefore the quality of the online signal could be lower.

### 2.2.2.3 Passive EEG during fMRI-NF (pEEG):

Recording passive EEG during fMRI-NF allows to evaluate the fMRI-NF protocol and to identify electrophysiological correlates of the fMRI-NF training. In this configuration EEG artefact correction is performed offline. This approach can be used to explore EEG correlates of fMRI-NF that could be used as potential targets for EEG-NF or EEG-fMRI-NF.

### 2.2.2.4 EEG-informed fMRI NF or EEG Finger-Print NF (iEEG):

In literature, two terms can be found: EEG-informed fMRI or EEG Finger-Print (EFP). The all-around term used in the literature is EEG-informed fMRI, but that sometimes one can find EFP what defines the same thing. The term EEG-informed fMRI refers to methods extracting features from EEG signals in order to derive a predictor of the associated BOLD signal in a specific ROI (Cury et al., 2020) (Figure 4). The interest is to be able to reproduce EEG-fMRI-NF in real time with EEG only in order to increase the quality of EEG-NF sessions. To export fMRI information outside the scanner, most of the methods intend to predict the fMRI BOLD signal activity in a specific ROI by learning from an EEG signal recorded simultaneously inside the fMRI scanner (Meir-Hasson et al., 2014; Simoes et al., 2020). For a comprehensive review on EEG-informed fMRI (Abreu et al., 2018; Murta et al., 2015).

<figure><img src="image_5.png" /><figcaption>**Figure 4.** Schematic of EEG-informed fMRI from Cury et al. (Cury et al., 2020). The idea of this method is to use data from NF-based EEG-fMRI sessions to create an NF-fMRI or NF-EEG-fMRI predictor to add missing information during EEG sessions only.</figcaption></figure>

### 2.2.2.5 EEG-fMRI-NF (EEG-fMRI):

EEG-fMRI-NF refers to the combined use of simultaneous EEG and fMRI features for NF. Here the EEG must be cleaned of all artefacts (Balistocardiogram artifacts (BCG), gradient artefacts and electric signal caused by radio-frequency pulse (pulse artefacts) (Wu et al., 2016)) in real time and not offline (Mayeli et al., 2016; Klovatch-Podlipsky et al., 2016). This type of protocol can be seen as the combination of EEG-NF and fMRI-NF protocol. From a technical point of view we refer to the work from Mano and colleagues (Mano et al., 2017) who describe how to build an NF based EEG-fMRI platform. An open source hardware and software system for acquisition and real time processing can be found in (Purdon et al., 2008). In addition to the technical difficulty of implementing the EEG inside the MR system, these
studies must also answer the question of the form that takes the feedback. How to represent two features
corresponding to different neuromarkers so that the subject can regulate them optimally?

<figure><img src="image_6.png" /><figcaption>**Figure 5.** Photography of the preparation of a simultaneous EEG-fMRI-NF session with a 3T MRI and a 64 EEG cap. (a) EEG subsystem installation and impedance check outside the MR environment, (b) Installation of the MR coil and EEG impedance recheck, (c) Placement of the amplifiers, battery and LCD display.</figcaption></figure>

3 NF-BASED EEG-FMRI

The following section describes the state-of-the-art of NF applications based on EEG-fMRI neuroimaging modalities. This review of the literature was conducted in accordance with PRISMA guidelines (Moher et al., 2009). We conducted a PubMed search using the following key-words: "EEG" AND "fMRI" AND "NF" resulted in 81 publications. We then screened out conference proceedings, articles only using EEG or fMRI modality, articles unavailable in english, and others that were falsely identified as NF studies by the search engine. Within the remaining publications, we identified articles reporting original research studies. This excluded reviews, opinion pieces, methods only papers, and the ones using other non-fMRI/EEG modalities. During the process of reviewing the articles (described below), several additional research studies missed in the initial literature search were identified and added to the review. Articles reporting secondary analysis or reusing participant data were not included so as not to over represent single studies (final number of unique studies n = 15). Two main families of paradigms were found: the network-based emotion paradigm with applications for Major depressive disorder (MDD) or for Post-traumatic stress disorder (PTSD) patients; and the Motor-Imagery (MI) paradigm with applications for stroke patients.

**3.1 Emotion Network Paradigm**

To date, the amygdala is highly represented as a feature in the EEG-fMRI-NF studies, with various combinations of EEG-fMRI such as simultaneous EEG-fMRI and passive EEG during fMRI-NF or EFP. Amygdala-based NF refers to the fact that the amygdala is the region of emotions and seems to be a very good neuromarker for rehabilitation. This feature is very suitable for the EEG-fMRI as it is a fairly deep region in the brain and difficult to reach with EEG only. The first study that reported amygdala as fMRI feature for EEG-fMRI-NF was designed by Zotev and colleagues (Zotev et al., 2014) thanks to the anterior study from Johnston and colleagues (Johnston et al., 2010), in which they claimed that after localising emotional network using fMRI only, subjects could upregulate target areas, including the insula and amygdala. These studies used positive autobiographical memories as a task during NF trials.
An extensive body of research in both humans and experimental animals has established that the amygdala plays a central role in several aspects of emotion processing, such as recognition of both positively- and negatively-balanced emotional stimuli, reward learning, and appetitive or aversive conditioning. The involvement of the amygdala during mood self-induction has been reported in several studies (Phelps et al., 2001; Schneider et al., 1998). Therefore, the possibility of volitional modulation of left amygdala activity using fMRI-NF training provides a valuable tool to study neurophysiological regulation within neural networks involved in emotional processing (Zotev et al., 2011). Several fMRI-NF studies have also revealed the potential of emotion regulation for clinical utility to reduce symptoms associated with chronic pain (DeCharms et al., 2005), smoking cessation (Hartwell et al., 2016), anxiety (Zilverstand et al., 2015), PTSD (Nicholson et al., 2016), and MDD (Linden et al., 2012).

In simultaneous EEG-fMRI-NF the frontal asymmetry is used as an EEG feature. For instance, (Zotev et al., 2014) proposed a fMRI-EEG-NF based on the frontal EEG power asymmetry in the high-beta band (FBA; 21-30 Hz) and upregulation of BOLD fMRI activation in the left amygdala (9 healthy subjects) (See Figure 6). This EEG feature would seem to be an important and widely used feature for emotion and emotional reactivity (Davidson, 1992); indeed, many EEG studies have indicated that depression and anxiety are associated with reduced relative activation of the left frontal regions and increased relative activation of the right frontal regions (meta-analysis from (Thibodeau et al., 2006)). They claimed that the combined protocol could be more efficient than either the EEG-NF or the fMRI-NF protocol performed separately. In a follow-up study with the same paradigm (Zotev et al., 2020), the authors conducted a proof-of-concept study with MDD patients (experimental group n = 16, control group n = 9). Participants demonstrated significant upregulation of the left amygdala BOLD activity, Frontal alpha asymmetry (FAA), and FBA during the EEG-fMRI-NF task. Their results also demonstrated that FAA and frontal beta asymmetry FBA showed temporal correlations with amygdala BOLD activity.

In EEG-informed fMRI NF the BOLD activity of the amygdala is predicted thanks to a time-frequency representation extracted from the EEG data (Meir-Hasson et al., 2014), yielding an EEG model of weighted coefficients (Figure 4). It has also been shown that this prediction model can reliably probe amygdala BOLD activity and, that compared with sham-NF, EEG-informed fMRI (called amygdala EEG finger print in the paper) can lead to improved amygdala BOLD downregulation capacities via fMRI-NF (Keynan et al., 2016a). In a recent study with a double-blind randomised controlled trial and a large sample (n = 180), Keynan and colleagues (Keynan et al., 2019) demonstrated in a follow-up fMRI-NF (approximately 1 months after the training) greater amygdala BOLD downregulation and amygdala-ventromedial prefrontal cortex functional connectivity following EEG-informed fMRI NF relative to the no-NF. It is interesting to note that the control group who received an NF based on the alpha/theta ratio did not undergo this last fMRI-NF session.

## 3.2 Motor Imagery Paradigm

Motor Imagery (MI) combined with EEG-NF or fMRI-NF is a very popular approach and is considered to be a promising for neurorehabilitation training of stroke patients (Cervera et al., 2018). In simultaneous EEG-fMRI, two neuromarkers need to be considered, one for the EEG, the other for the fMRI. Compared to the amygdala-based NF, the regions involved for the MI are cortical and are therefore easily detectable even for the EEG. When imagining movement (or executing movement), amplitude desynchronisations are detected in the alpha and beta bands (8-30 Hz) from the sensorimotor cortices: also known as Event-Related Desynchronisation (ERD) (Cheyne, 2013; Pfurtscheller and Neuper, 2001). On the fMRI-NF side, most MI paradigm involve primarily upregulation of Primary motor cortex (M1) or premotor (SMA) (Berman et al., 2012; Fede et al., 2020), but the choice of the ROI is still controversial (Fede et al., 2020). Indeed,
<figure><img src="image_7.png" /><figcaption>**Figure 6.** EEG-fMRI-NF experimental protocol for emotional self-regulation described (from Zotev, Phillips, et al. (Zotev et al., 2014)).</figcaption></figure>

Berman and colleagues (Berman et al., 2012) found that M1 regulation was possible during finger tapping but not motor imagery, while Mehler and colleagues (Mehler et al., 2019) found that neurofeedback was associated with a decrease in primary motor but an increase in SMA engagement activity during MI.

In simultaneous EEG-fMRI-NF studies, ERD over sensorimotor cortex and upregulation of BOLD motor areas have been used as features. These studies have been conducted by Perronnet and colleagues (Perronnet, 2017; Perronnet et al., 2020), who performed the first EEG-fMRI-NF with healthy subjects (n = 10) with MI paradigm (Perronnet, 2017). In this study, MI-based EEG-fMRI-NF was compared to unimodal MI-based EEG-NF and MI-based fMRI-NF. The authors reported that MI-related hemodynamic activity was higher during EEG-fMRI-NF than during EEG-NF, suggesting that EEG-fMRI-NF could indeed be more specific or more engaging than EEG-NF. It also highlighted that during bimodal EEG-fMRI-NF subjects could happen to regulate one modality more than the other, hence supporting the hypothesis that different neural mechanisms are involved during regulation of fMRI or EEG activity with NF (Zotev et al., 2014). In a follow-up study, the same authors concentrated on the representation of the EEG-fMRI feature. A two-dimensional feedback (2D) (See Figure ??) in which each dimension depicted the information from one modality was compared to an uni-dimensional feedback (1D) that integrated both types of information. It was reported that 1D and 2D integrated feedback are both effective but online fMRI activations were significantly higher in the 1D group than in the 2D group.

In passive fMRI during EEG-NF, a study from Zich and colleagues (Zich et al., 2015), showed that MI-based EEG-NF allows subjects to generate enhanced cortical activation in EEG but also higher BOLD activity compared to MI with no feedback. Interestingly, the study revealed that the contralateral activity in EEG and fMRI were correlated while the laterality patterns were not. The revelation that EEG and fMRI signatures of MI are not redundant suggests a potential for bimodal EEG-fMRI-NF.
In EEG-informed fMRI NF, a recent study from Cury and colleagues proposed a new model able to exploit EEG only to predict fMRI-NF or EEG-fMRI-NF during MI tasks. They showed that predicting NF-fMRI scores from EEG signals adds information to NF-EEG scores and significantly improves the correlation with bi-modal NF sessions compared to classical NF-EEG scores.

# 4 DISCUSSION

Integrating complementary sources of information about neural activity in a meaningful way can significantly increase the overall amount of information extracted. Data fusion techniques have been highly successful in neuroimaging in general and in NF / BCI in particular. This is why NF based EEG-fMRI studies have gained interest in recent years. These studies employed different ways of combining these modalities, the most used being simultaneous EEG-fMRI and EEG-informed fMRI but other combinations exist such as passive EEG during fMRI / passive fMRI during EEG or fMRI before / after EEG. In the literature, researchers using EEG-fMRI as a means to provide NF have focused on two main paradigms: MI paradigm and Emotion Network Paradigm. In each of these paradigms, clinical applications have been addressed with promising results. However, there are still limitations and challenges that must be addressed by the NF community. In this section, we will discuss some of these points regarding the study design or new NF applications and limitations of current solutions.

In this state-of-the art, we focused on the EEG-fMRI as NF modalities. Many studies have been focused on the contribution of this bimodality. Indeed, their complementarity is no longer justified as many studies and reviews have shown the potential behind the achievement of a very high spatiotemporal resolution. However, there are still limits that are intrinsic to the nature of this modality: the portability. Indeed, fMRI is both the brake and engine of this research, its high temporal resolution being counterbalanced by its high cost and stationary aspect. However, some studies have addressed this problem thanks to the EEG-informed fMRI, the aim is to approximate the BOLD signal of a specific ROI during EEG-fMRI sessions in order to be able to render it during EEG only sessions. With the limitation that this method is still individual and specific to each subject (Cury et al., 2020). A recent study has used this learning method with subjects who were not involved in the construction of the model (Keynan et al., 2019), but without taking into account the progression of the subject through the sessions because "a change in strategy for the task might impact the learned model, as the relation between the EEG and fMRI signals may change" (Cury et al., 2020). A possible improvement of this method would be to predict the evolution of the score through the sessions.

In order to overcome this problem of portability while keeping the possibility of measuring the haemodynamic response would be to use the fNIRS, that measures infrared light absorption of haemodynamic signals in the brain by scalp optodes at a spatial resolution of 2–5 cm² (Kohl et al., 2020). Moreover, EEG can be used simultaneously with NIRS without major technical difficulties. There is no influence of these modalities on each other and a combined measurement can give useful information about electrical activity as well as hemodynamics at medium spatial resolution. Several NF studies have already made full use of fNIRS as brain imaging modalities as revealed by this review (Kohl et al., 2020). Some studies developed fully integrated wireless wearable EEG-NIRS bimodal acquisition system (Safaie et al., 2013), that could be used for NF applications with multiple sessions (Kassab et al., 2018) and even rehabilitation (Rieke et al., 2020). However, fNIRS cannot be used to measure cortical activity more than 4 cm deep due to limitations in light emitter power and has more limited spatial resolution. A review on the use of EEG-fNIRS for BCI can be found in (Fazli et al., 2015).

A fusion of EEG-fMRI that has not been studied in NF is fMRI-informed EEG, where EEG electromagnetic source reconstruction benefits from the spatial information of fMRI signals; in this approach, the ill-posed
<table><caption>**Table 1.** NF-based EEG-fMRI studies *Note*. S: Simultaneous EEG-fMRI; pfMRI: Passive fMRI during EEG-NF; pEEG: Passive EEG during fMRI-NF; iEEG: EEG-Informed fMRI-NF or EEG fingerprint; efMRI: fMRI before/after EEG-NF NP: Number of Patient; NS: Number of subject; SP: Stroke patients; PTSD: Post-traumatic stress disorder patients; MDD: Major depressive disorder patients; FAA: Frontal Asymmetry in alpha band; T/A: Theta/Alpha (4-7 Hz)/(8-13 Hz) Band; FBA: Frontal Asymmetry in high beta band; RA/LA: Right/Left Amygdala; rACC: left rostral anterior cingulate cortex; pSTS: right posterior Superior Temporal Sulcus; FEPN: Facial Expressions Processing Network; HEI: Happy Emotion Induction; MI: Motor imagery;</caption><thead><tr><th>Combination</th><th>fMRI-NF feature</th><th>EEG-NF feature</th><th>Task</th><th>Paper purpose</th><th>NP/NS</th><th>Reference</th></tr></thead><tbody><tr><td>EEG-fMRI</td><td>LA</td><td>Frontal Asymmetry in the high-beta band (21-30 Hz)</td><td>HEI</td><td>Research</td><td>-/9</td><td>(Zotev et al., 2014)</td></tr><tr><td>EEG-fMRI</td><td>LA + rACC</td><td>FAA (7.5-12.5 Hz) + FBA (21-30 Hz)</td><td>HEI</td><td>Rehabilitation</td><td>16-8/- MDD</td><td>(Zotev et al., 2020)</td></tr><tr><td>EEG-fMRI</td><td>M1</td><td>Laterality index between C1 and C2 in the alpha band (8-12 Hz) over sensori-motor cortex</td><td>MI</td><td>Research</td><td>-/10</td><td>(Perronnet et al., 2017)</td></tr><tr><td>pEEG</td><td>LA</td><td>N/A</td><td>HEI</td><td>Rehabilitation</td><td>13/- MDD</td><td>(Zotev et al., 2016)</td></tr><tr><td>pEEG</td><td>LA</td><td>N/A</td><td>HEI</td><td>Rehabilitation</td><td>20-11/- PTSD</td><td>(Zotev et al., 2018)</td></tr><tr><td>pEEG</td><td>pSTS</td><td>N/A</td><td>Mental imagery of facial expression</td><td>Research</td><td>-/13</td><td>(Sim, 2015) &amp; (Simoes et al., 2017)</td></tr><tr><td>pfMRI</td><td>N/A</td><td>SMR</td><td>MI</td><td>Research</td><td>-/24</td><td>(Zich et al., 2015)</td></tr><tr><td>pfMRI</td><td>N/A</td><td>SMR</td><td>N/A (control the ball)</td><td>Rehabilitation</td><td>9-8/- CP</td><td>(Terrasa et al., 2020)</td></tr><tr><td>pfMRI</td><td>N/A</td><td>T/A</td><td>Relaxation with eyes closed</td><td>Research</td><td>-/45</td><td>(Kinreich et al., 2014)</td></tr><tr><td>iEEG</td><td>Motor cortex</td><td>PSD alpha/beta band (8-30 Hz)</td><td>MI</td><td>Research</td><td>-/17</td><td>(Cury et al., 2020)</td></tr><tr><td>iEEG</td><td>pSTS</td><td>N/A</td><td>Mental imagery of facial expression</td><td>Research</td><td>-/10</td><td>(Simoes et al., 2020)</td></tr><tr><td>iEEG</td><td>RA</td><td>T/A from occipital electrodes</td><td>Relaxation</td><td>Research</td><td>-/20</td><td>(Meir-Hasson et al., 2016)</td></tr><tr><td>iEEG</td><td>RA</td><td>N/A</td><td>Lower the volume of an auditory stimulus</td><td>Research</td><td>-/15-9</td><td>(Keynan et al., 2016b)</td></tr><tr><td>iEEG</td><td>RA</td><td>T/A</td><td>Relaxation</td><td>Research</td><td>-/90-45-45</td><td>(Keynan et al., 2019)</td></tr><tr><td>efMRI</td><td>N/A</td><td>Alpha band (8-12 Hz) over midline parietal cortex (Pz)</td><td>N/A</td><td>Research</td><td>-/34</td><td>(Ros et al., 2013)</td></tr><tr><td>efMRI</td><td>N/A</td><td>Alpha band (8-12 Hz) over midline parietal cortex (Pz)</td><td>N/A</td><td>Rehabilitation</td><td>21/- PTSD</td><td>(Nicholson et al., 2016)</td></tr></tbody></table>

combination

problem of EEG source imaging (LORETA) can be moderated with fMRI spatial constraints (Nunez and Silberstein, 2000; Babiloni et al., 2005; Lei et al., 2011). This spatial information can therefore provide feedback related to the activity of a specific ROI rather than basing training on scalp activity. In 2017,
Noorzadeh and colleagues (Noorzadeh et al., 2017) proposed to adapt a symmetrical approach based on EEG and fMRI for the estimation of the brain sources. They showed that their model provide better spatio-temporal resolution of the estimation of neuronal sources.

Comparing the progress between the two most commonly used EEG-fMRI paradigms for NF, it is interesting to note that there have been no MI-based EEG-fMRI studies in patients where, conversely, in the paradigm of emotion, many studies have been produced (Zotev et al., 2020, 2016, 2018; Nicholson et al., 2016). Yet it would be interesting to study this paradigm for stroke patients, indeed, recent studies have revealed the potential of NF training for stroke rehabilitation (Cervera et al., 2018) (for a review, see e.g. (Wang et al., 2018)). In addition to having the possibility of training the patient on two neuromarkers, MRI makes it possible to verify the effectiveness of NF training using structural magnetic resonance imaging and Diffusion tensor imaging (DTI), which could make it possible to study the integrity of the ipsilateral Corticospinal tract (CST).

The question of the fusion of EEG-fMRI features is also fundamental and must be addressed in future studies. Indeed, when is the NF score, which relies on combinations of complementary analysis as well as recording techniques, is the most revealing of the subject's brain activity? That is why multimodal NF can benefit from a segment of machine learning that is called data fusion, because it subsumes techniques that combine information from multiple signal sources as well as associated databases (Waltz et al., 1990; Hall and Llinas, 1997). Nevertheless, as pointed out by Oviatt (Oviatt, 1999), the main advantage of multimodal interaction is not enhanced efficiency but decreased error rate, flexibility to choose between alternating input modes, and a wider range of users.

# 5 CONCLUSION

In this review, we summarised and discussed the state-of-the-art when combining EEG-fMRI for neurofeedback. We outlined the different studies implying this system such as fMRI informed EEG-NF or simultaneous EEG-fMRI-NF. This survey also stressed the potential of EEG-fMRI for NF studies to improve their design by increasing the pertinence of feedback provided. Further studies are however required to test the use of EEG-fMRI for BCI and NF in order to complete our knowledge of EEG-fMRI fusion for NF.

# 6 ACKNOWLEDGEMENTS

The authors acknowledge the financial support of the Fundacao para a Ciencia e Tecnologia through FCT CEECIND/01073/2018, the LARSyS - FCT Project UIDB/50009/2020, and the NeurAugVR project (PTDC/CCI-COM/31485/2017).

# REFERENCES

* (2015). Feature analysis for correlation studies of simultaneous EEG-fMRI data: A proof of concept for neurofeedback approaches. *Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS* 2015-November, 4065–4068

* Abreu, R., Leal, A., and Figueiredo, P. (2018). Eeg-informed fmri: a review of data analysis methods. *Frontiers in human neuroscience* 12, 29

* Arthurs, O. J. and Boniface, S. (2002). How well do we understand the neural origins of the fmri bold signal? *TRENDS in Neurosciences* 25, 27–31

* Babiloni, F., Cincotti, F., Babiloni, C., Carducci, F., Mattia, D., Astolfi, L., et al. (2005). Estimation of the cortical functional connectivity with the multimodal integration of high-resolution eeg and fmri data by directed transfer function. *Neuroimage* 24, 118–131
Bandettini, P. A., Wong, E. C., Hinks, R. S., Tikofsky, R. S., and Hyde, J. S. (1992). Time course epi of human brain function during task activation. *Magnetic resonance in medicine* 25, 390–397

Berman, B. D., Horovitz, S. G., Venkataraman, G., and Hallett, M. (2012). Self-modulation of primary motor cortex activity with motor and motor imagery tasks using real-time fmri-based neurofeedback. *Neuroimage* 59, 917–925

Biessmann, F., Plis, S., Meinecke, F. C., Eichele, T., and Muller, K.-R. (2011). Analysis of multimodal neuroimaging data. *IEEE reviews in biomedical engineering* 4, 26–58

Biswal, B., Zerrin Yetkin, F., Haughton, V. M., and Hyde, J. S. (1995). Functional connectivity in the motor cortex of resting human brain using echo-planar mri. *Magnetic resonance in medicine* 34, 537–541

Buzsáki, G., Anastassiou, C. A., and Koch, C. (2012). The origin of extracellular fields and currents—eeg, ecog, lfp and spikes. *Nature reviews neuroscience* 13, 407–420

Cervera, M. A., Soekadar, S. R., Ushiba, J., Millán, J. d. R., Liu, M., Birbaumer, N., et al. (2018). Brain-computer interfaces for post-stroke motor rehabilitation: a meta-analysis. *Annals of Clinical and Translational Neurology* 5, 651–663. doi:10.1002/acn3.544

Cheyne, D. O. (2013). Meg studies of sensorimotor rhythms: a review. *Experimental neurology* 245, 27–39

Cury, C., Maurel, P., Gribonval, R., and Barillot, C. (2020). A Sparse EEG-Informed fMRI Model for Hybrid EEG-fMRI Neurofeedback Prediction. *Frontiers in Neuroscience* 13, 1–12. doi:10.3389/fnins.2019.01451

da Silva, F. L. (2013). Eeg and meg: relevance to neuroscience. *Neuron* 80, 1112–1128

Daunizeau, J., Grova, C., Marrelec, G., Mattout, J., Jbabdi, S., Pelegrini-Issac, M., et al. (2007). Symmetrical event-related eeg/fmri information fusion in a variational bayesian framework. *Neuroimage* 36, 69–87

Daunizeau, J., Laufs, H., and Friston, K. J. (2009). Eeg–fmri information fusion: Biophysics and data analysis. In *EEG-fMRI* (Springer). 511–526

Davidson, R. J. (1992). Anterior cerebral asymmetry and the nature of emotion. *Brain and cognition* 20, 125–151

Debener, S., Ullsperger, M., Siegel, M., Fiehler, K., Von Cramon, D. Y., and Engel, A. K. (2005). Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. *Journal of Neuroscience* 25, 11730–11737

DeCharms, R. C., Maeda, F., Glover, G. H., Ludlow, D., Pauly, J. M., Soneji, D., et al. (2005). Control over brain activation and pain learned by using real-time functional mri. *Proceedings of the National Academy of Sciences* 102, 18626–18631

Dong, L., Luo, C., Liu, X., Jiang, S., Li, F., Feng, H., et al. (2018). Neuroscience information toolbox: An open source toolbox for eeg–fmri multimodal fusion analysis. *Frontiers in Neuroinformatics* 12, 56

Fazli, S., Dähne, S., Samek, W., Bießmann, F., and Mueller, K.-R. (2015). Learning from more than one data source: data fusion techniques for sensorimotor rhythm-based brain–computer interfaces. *Proceedings of the IEEE* 103, 891–906

Fede, S. J., Dean, S. F., Manuweera, T., and Momenan, R. (2020). A guide to literature informed decisions in the design of real time fmri neurofeedback studies: A systematic review. *Frontiers in Human Neuroscience*

Friston, K., Frith, C., Liddle, P., and Frackowiak, R. (1993). Functional connectivity: the principal-component analysis of large (pet) data sets. *Journal of Cerebral Blood Flow & Metabolism* 13, 5–14
Gruzelier, J. H. (2014). Eeg-neurofeedback for optimising performance. i: a review of cognitive and affective outcome in healthy participants. Neuroscience & Biobehavioral Reviews 44, 124–141

Hall, D. L. and Llinas, J. (1997). An introduction to multisensor data fusion. Proceedings of the IEEE 85, 6–23

Hartwell, K. J., Hanlon, C. A., Li, X., Borckardt, J. J., Canterbury, M., Prisciandaro, J. J., et al. (2016). Individualized real-time fmri neurofeedback to attenuate craving in nicotine-dependent smokers. Journal of psychiatry & neuroscience: JPN 41, 48

Johnston, S. J., Boehm, S. G., Healy, D., Goebel, R., and Linden, D. E. (2010). Neurofeedback: a promising tool for the self-regulation of emotion networks. Neuroimage 49, 1066–1072

Jorge, J., Van der Zwaag, W., and Figueiredo, P. (2014). Eeg-fmri integration for the study of human brain function. Neuroimage 102, 24–34

Kassab, A., Le Lan, J., Tremblay, J., Vannasing, P., Dehbozorgi, M., Pouliot, P., et al. (2018). Multichannel wearable f nirs-eeg system for long-term clinical monitoring. Human brain mapping 39, 7–23

Keynan, J. N., Cohen, A., Jackont, G., Green, N., Goldway, N., Davidov, A., et al. (2019). Electrical fingerprint of the amygdala guides neurofeedback training for stress resilience. Nature Human Behaviour 3, 63–73. doi:10.1038/s41562-018-0484-3

Keynan, J. N., Meir-Hasson, Y., Gilam, G., Cohen, A., Jackont, G., Kinreich, S., et al. (2016a). Limbic activity modulation guided by functional magnetic resonance imaging—inspired electroencephalography improves implicit emotion regulation. Biological Psychiatry 80, 490–496

Keynan, J. N., Meir-Hasson, Y., Gilam, G., Cohen, A., Jackont, G., Kinreich, S., et al. (2016b). Limbic Activity Modulation Guided by Functional Magnetic Resonance Imaging–Inspired Electroencephalography Improves Implicit Emotion Regulation. Biological Psychiatry 80, 490–496. doi:10.1016/j.biopsych.2015.12.024

Kinreich, S., Podlipsky, I., Jamshy, S., Intrator, N., and Hendler, T. (2014). Neural dynamics necessary and sufficient for transition into pre-sleep induced by EEG NeuroFeedback. NeuroImage 97, 19–28. doi:10.1016/j.neuroimage.2014.04.044

Klovatch-Podlipsky, I., Gazit, T., Fahoum, F., Tsirelson, B., Kipervasser, S., Kremer, U., et al. (2016). Dual array eeg-fmri: an approach for motion artifact suppression in eeg recorded simultaneously with fmri. Neuroimage 142, 674–686

Kohl, S. H., Mehler, D., Luhrs, M., Thibault, R. T., Konrad, K., and Sorger, B. (2020). The potential of functional near-infrared spectroscopy-based neurofeedback—a systematic review and recommendations for best practice. Frontiers in neuroscience 14, 594

LaConte, S. M. (2011). Decoding fmri brain states in real-time. Neuroimage 56, 440–454

Lei, X. (2019). Simultaneous eeg-fmri. In EEG Signal Processing and Feature Extraction (Springer). 377–405

Lei, X., Valdes-Sosa, P. A., and Yao, D. (2012). Eeg/fmri fusion based on independent component analysis: integration of data-driven and model-driven methods. Journal of integrative neuroscience 11, 313–337

Lei, X., Xu, P., Luo, C., Zhao, J., Zhou, D., and Yao, D. (2011). fmri functional networks for eeg source imaging. Human Brain Mapping 32, 1141–1160

Lent, R., Azevedo, F. A., Andrade-Moraes, C. H., and Pinto, A. V. (2012). How many neurons do you have? some dogmas of quantitative neuroscience under revision. European Journal of Neuroscience 35, 1–9

Linden, D. E., Habes, I., Johnston, S. J., Linden, S., Tatineni, R., Subramanian, L., et al. (2012). Real-time self-regulation of emotion networks in patients with depression. PloS one 7, e38115

Logothetis, N. K. (2008). What we can do and what we cannot do with fmri. Nature 453, 869–878
Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., and Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fmri signal. *nature* 412, 150–157

Mano, M., Lécuyer, A., Bannier, E., Perronnet, L., Noorzadeh, S., and Barillot, C. (2017). How to build a hybrid neurofeedback platform combining eeg and fmri. *Frontiers in neuroscience* 11, 140

Mayeli, A., Zotev, V., Refai, H., and Bodurka, J. (2016). Real-time eeg artifact correction during fmri using ica. *Journal of neuroscience methods* 274, 27–37

Mehler, D. M., Williams, A. N., Krause, F., Lührs, M., Wise, R. G., Turner, D. L., et al. (2019). The bold response in primary motor cortex and supplementary motor area during kinesthetic motor imagery based graded fmri neurofeedback. *NeuroImage* 184, 36–44

Meir-Hasson, Y., Keynan, J. N., Kinreich, S., Jackont, G., Cohen, A., Podlipsky-Klovatch, I., et al. (2016). One-class FMRI-inspired EEG model for self-regulation training. *PLoS ONE* 11, 1–17. doi:10.1371/journal.pone.0154968

Meir-Hasson, Y., Kinreich, S., Podlipsky, I., Hendler, T., and Intrator, N. (2014). An eeg finger-print of fmri deep regional activation. *Neuroimage* 102, 128–141

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., Group, P., et al. (2009). Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. *PLoS med* 6, e1000097

Murta, T., Leite, M., Carmichael, D. W., Figueiredo, P., and Lemieux, L. (2015). Electrophysiological correlates of the bold signal for eeg-informed fmri. *Human brain mapping* 36, 391–414

Nicholson, A. A., Ros, T., Frewen, P. A., Densmore, M., Theberge, J., Kluetsch, R. C., et al. (2016). Alpha oscillation neurofeedback modulates amygdala complex connectivity and arousal in posttraumatic stress disorder. *NeuroImage: Clinical* 12, 506–516. doi:10.1016/j.nicl.2016.07.006

Noorzadeh, S., Maurel, P., Oberlin, T., Gribonval, R., and Barillot, C. (2017). Multi-modal eeg and fmri source estimation using sparse constraints. In *International Conference on Medical Image Computing and Computer-Assisted Intervention* (Springer), 442–450

Nunez, P. L. and Silberstein, R. B. (2000). On the relationship of synaptic activity to macroscopic measurements: does co-registration of eeg with fmri make sense? *Brain topography* 13, 79–96

Ostwald, D., Porcaro, C., and Bagshaw, A. P. (2011). Voxel-wise information theoretic eeg-fmri feature integration. *Neuroimage* 55, 1270–1286

Oviatt, S. (1999). Ten myths of multimodal interaction. *Communications of the ACM* 42, 74–81

Perronnet, L. (2017). *Combining EEG and FMRI for Neurofeedback*. Ph.D. thesis

Perronnet, L., Lécuyer, A., Mano, M., Bannier, E., Lotte, F., Clerc, M., et al. (2017). Unimodal versus bimodal eeg-fmri neurofeedback of a motor imagery task. *Frontiers in Human Neuroscience* 11, 193

Perronnet, L., Lécuyer, A., Mano, M., Fleury, M., Lioi, G., Cury, C., et al. (2020). Learning 2-in-1: towards integrated eeg-fmri-neurofeedback. *BioRxiv*, 397729

Pfurtscheller, G. and Neuper, C. (2001). Motor imagery and direct brain-computer communication. *Proceedings of the IEEE* 89, 1123–1134. doi:10.1109/5.939829

Phelps, E. A., O’Connor, K. J., Gatenby, J. C., Gore, J. C., Grillon, C., and Davis, M. (2001). Activation of the left amygdala to a cognitive representation of fear. *Nature neuroscience* 4, 437–441

Purdon, P. L., Millan, H., Fuller, P. L., and Bonmassar, G. (2008). An open-source hardware and software system for acquisition and real-time processing of electrophysiology during high field mri. *Journal of neuroscience methods* 175, 165–186

Ramot, M., Kimmich, S., Gonzalez-Castillo, J., Roopchansingh, V., Popal, H., White, E., et al. (2017). Direct modulation of aberrant brain network connectivity through real-time neurofeedback. *Elife* 6, e28974
Rieke, J. D., Matarasso, A. K., Yusufali, M. M., Ravindran, A., Alcantara, J., White, K. D., et al. (2020). Development of a combined, sequential real-time fmri and fnirs neurofeedback system enhance motor learning after stroke. *Journal of Neuroscience Methods*, 108719

Ros, T., Theberge, J., Frewen, P. A., Kluetsch, R., Densmore, M., Calhoun, V. D., et al. (2013). Mind over chatter: Plastic up-regulation of the fMRI salience network directly after EEG neurofeedback. *NeuroImage* 65, 324–335. doi:10.1016/j.neuroimage.2012.09.046

Ruiz, S., Buyukturkoglu, K., Rana, M., Birbaumer, N., and Sitaram, R. (2014). Real-time fmri brain computer interfaces: self-regulation of single brain regions to networks. *Biological psychology* 95, 4–20

Ruiz, S., Rana, M., Sass, K., Kircher, T., Birbaumer, N., and Sitaram, R. (2011). Brain network connectivity and behaviour enhancement: a fmri-bci study. In *17th Annual Meeting of the Organization for Human Brain Mapping*

Safaie, J., Grebe, R., Moghaddam, H. A., and Wallois, F. (2013). Toward a fully integrated wireless wearable eeg-nirs bimodal acquisition system. *Journal of neural engineering* 10, 056001

Schneider, F., Weiss, U., Kessler, C., Salloum, J., Posse, S., Grodd, W., et al. (1998). Differential amygdala activation in schizophrenia during sadness. *Schizophrenia research* 34, 133–142

Simoes, M., Abreu, R., Direito, B., Sayal, A., Castelhano, J., Carvalho, P., et al. (2020). How much of the BOLD-fMRI signal can be approximated from simultaneous EEG data: Relevance for the transfer and dissemination of neurofeedback interventions. *Journal of Neural Engineering* 17. doi:10.1088/1741-2552/ab9a98

Simoes, M., Direito, B., Lima, J., Castelhano, J., Ferreira, C., Couceiro, R., et al. (2017). Correlated alpha activity with the facial expression processing network in a simultaneous EEG-fMRI experiment. *Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS*, 2562–2565doi:10.1109/EMBC.2017.8037380

Sitaram, R., Ros, T., Stoeckel, L., Haller, S., Scharnowski, F., Lewis-Peacock, J., et al. (2017). Closed-loop brain training: the science of neurofeedback. *Nature Reviews Neuroscience* 18, 86–100. doi:10.1038/nrn.2016.164

Stoeckel, L., Garrison, K., Ghosh, S., Wighton, P., Hanlon, C., Gilman, J., et al. (2014). Optimizing real time fMRI neurofeedback for therapeutic discovery and development. *NeuroImage: Clinical* 5, 245–255. doi:10.1016/J.NICL.2014.07.002

Tatum IV, W. O. (2014). *Handbook of EEG interpretation* (Demos Medical Publishing)

Terrasa, J. L., Barros-Loscertales, A., Montoya, P., and Munoz, M. A. (2020). Self-Regulation of SMR Power Led to an Enhancement of Functional Connectivity of Somatomotor Cortices in Fibromyalgia Patients. *Frontiers in Neuroscience* 14, 1–14. doi:10.3389/fnins.2020.00236

Thibault, R. T., MacPherson, A., Lifshitz, M., Roth, R. R., and Raz, A. (2018). Neurofeedback with fmri: A critical systematic review. *Neuroimage* 172, 786–807

Thibodeau, R., Jorgensen, R. S., and Kim, S. (2006). Depression, anxiety, and resting frontal eeg asymmetry: a meta-analytic review. *Journal of abnormal psychology* 115, 715

Trujillo-Barreto, N., Martinez-Montes, E., Melie-Garcia, L., and Valdes-Sosa, P. (2001). A symmetrical bayesian model for fmri and eeg/meg neuroimage fusion. *Int. J. Bioelectromagn* 3, 1998–2000

Valdes-Sosa, P. A., Sanchez-Bornot, J. M., Sotero, R. C., Iturria-Medina, Y., Aleman-Gomez, Y., Bosch-Bayard, J., et al. (2009). Model driven eeg/fmri fusion of brain oscillations. *Human brain mapping* 30, 2701–2721

van de Ven, V. G., Formisano, E., Prvulovic, D., Roeder, C. H., and Linden, D. E. (2004). Functional connectivity as revealed by spatial independent component analysis of fmri measurements during rest. *Human brain mapping* 22, 165–178
Waltz, E., Llinas, J., et al. (1990). *Multisensor data fusion*, vol. 685 (Artech house Boston)

Wang, T., Mantini, D., and Gillebert, C. R. (2018). The potential of real-time fmri neurofeedback for stroke rehabilitation: A systematic review. *cortex* 107, 148–165

Weiskopf, N. (2012). Real-time fmri and its application to neurofeedback. *Neuroimage* 62, 682–692

Wu, X., Wu, T., Zhan, Z., Yao, L., and Wen, X. (2016). A real-time method to reduce ballistocardiogram artifacts from eeg during fmri based on optimal basis sets (obs). *Computer methods and programs in biomedicine* 127, 114–125

Zich, C., Debener, S., Kranczioch, C., Bleichner, M. G., Gutberlet, I., and De Vos, M. (2015). Real-time EEG feedback during simultaneous EEG-fMRI identifies the cortical signature of motor imagery. *NeuroImage* 114, 438–447. doi:10.1016/j.neuroimage.2015.04.020

Zilverstand, A., Sorger, B., Sarkheil, P., and Goebel, R. (2015). fmri neurofeedback facilitates anxiety regulation in females with spider phobia. *Frontiers in behavioral neuroscience* 9, 148

Zotev, V., Krueger, F., Phillips, R., Alvarez, R. P., Simmons, W. K., Bellgowan, P., et al. (2011). Self-regulation of amygdala activation using real-time fmri neurofeedback. *PloS one* 6, e24522

Zotev, V., Mayeli, A., Misaki, M., and Bodurka, J. (2020). Emotion self-regulation training in major depressive disorder using simultaneous real-time fmri and eeg neurofeedback. *NeuroImage: Clinical* 27, 102331

Zotev, V., Phillips, R., Misaki, M., Wong, C. K., Wurfel, B. E., Krueger, F., et al. (2018). Real-time fmri neurofeedback training of the amygdala activity with simultaneous eeg in veterans with combat-related ptsd. *NeuroImage: Clinical* 19, 106–121

Zotev, V., Phillips, R., Yuan, H., Misaki, M., and Bodurka, J. (2014). Self-regulation of human brain activity using simultaneous real-time fMRI and EEG neurofeedback. *Neuroimage* 85, 985–995

Zotev, V., Yuan, H., Misaki, M., Phillips, R., Young, K. D., Feldner, M. T., et al. (2016). Correlation between amygdala bold activity and frontal eeg asymmetry during real-time fmri neurofeedback training in patients with depression. *NeuroImage: Clinical* 11, 224–238