Author response: Monkey EEG links neuronal color and motion information across species and scales
2019-05-28

Article Figures and data Abstract eLife digest Introduction Results Discussion Materials and methods Data availability References Decision letter Author response Article and author information Metrics Abstract It remains challenging to relate EEG and MEG to underlying circuit processes and comparable experiments on both spatial scales are rare. To close this gap between invasive and non-invasive electrophysiology we developed and recorded human-comparable EEG in macaque monkeys during visual stimulation with colored dynamic random dot patterns. Furthermore, we performed simultaneous microelectrode recordings from 6 areas of macaque cortex and human MEG. Motion direction and color information were accessible in all signals. Tuning of the non-invasive signals was similar to V4 and IT, but not to dorsal and frontal areas. Thus, MEG and EEG were dominated by early visual and ventral stream sources. Source level analysis revealed corresponding information and latency gradients across cortex. We show how information-based methods and monkey EEG can identify analogous properties of visual processing in signals spanning spatial scales from single units to MEG – a valuable framework for relating human and animal studies. https://doi.org/10.7554/eLife.45645.001 eLife digest Neurons carry information in the form of electrical signals, which we can listen to by applying sensors to the scalp: the resulting recordings are called an EEG. Electrical activity within the brain also generates a weak magnetic field above the scalp, which can be measured using a technique known as MEG. Both EEG and MEG only require a few dozen sensors, placed centimeters away from the brain itself, but they can reveal the precise timing and rough location of changes in neural activity. However, the brain consists of billions of neurons interconnected to form complex circuits, and EEG or MEG cannot reveal changes in activity of these networks in fine detail. In animals, and in patients undergoing brain surgery, scientists can use hair-thin microelectrodes to directly record the activity of individual neurons. Yet, it is difficult to know how activity measured inside the brain relates to that measured outside. To find out, Sandhaeger et al. had monkeys and healthy human volunteers perform the same task, where they had to watch a series of colored dots moving across a screen. The brain of the human participants was monitored using MEG; in the monkeys, EEG provided an indirect measure of brain activity, while microelectrodes directly revealed the activity of thousands of individual neurons. All three recordings contained information about movement and color. Moreover, the monkey EEG bridged the gap between direct and indirect recordings. Sandhaeger et al. identified signals in the monkey EEG that corresponded to the microelectrode recordings. They also spotted signals in the human MEG that matched the monkey EEG. Linking non-invasive measures of brain activity with underlying neural circuits could help to better understand the human brain. This approach may also allow clinicians to interpret EEG and MEG recordings in patients with brain disorders more easily. https://doi.org/10.7554/eLife.45645.002 Introduction How do results from human magnetoencephalography (MEG) and electroencephalography (EEG) experiments relate to those obtained from animals in invasive electrophysiology? It is generally well understood how potential changes in large populations of neurons can propagate through tissue types and lead to detectable electric potentials and associated magnetic fields outside the head (Pesaran et al., 2018). Yet, in typical MEG and EEG experiments, we have little clue which specific cellular and circuit mechanisms contribute to the recorded signals (Cohen, 2017). This can be attributed to several factors. First, the reconstruction of cortical sources from non-invasive signals is generally limited and based on assumptions (Darvas et al., 2004). Second, invasive and non-invasive electrophysiology are largely separate research fields. Comparable experiments performed on both levels and in the same species are rare, with few recent exceptions (Bimbi et al., 2018; Godlove et al., 2011; Reinhart et al., 2012; Shin et al., 2017; Snyder et al., 2015; Snyder et al., 2018). Third, studies employing invasive and non-invasive methods in parallel suffer from sparse sampling of recording sites. Massively parallel invasive recordings in multiple brain regions have only recently become viable (Dotson et al., 2017; Jun et al., 2017; Siegel et al., 2015), and EEG recordings in awake behaving animals have so far been limited to relatively few electrodes. This sparsity limits specificity when drawing conclusions from one level to the other. In summary, the mapping between measurement scales is severely underconstrained, both theoretically when trying to infer cortical sources of non-invasively measured activity, and experimentally by the lack of sufficiently comparable data. Thus, key for linking different scales are comparable large-scale recordings on all levels to provide high specificity and eventually trace the origins of large-scale phenomena back to their underlying cellular mechanisms. Importantly, this includes non-invasive recordings in animals. These allow to bridge the gap between invasive animal electrophysiology and non-invasive human experiments by permitting to disentangle similarities and differences due to species membership from those due to measurement technique. An especially suitable candidate for this is monkey EEG, making use of evolutionary proximity and promising to better connect the rich literature in non-human primate neurophysiology with human studies. A powerful tool to link data from different measurement scales is the abstraction from measured activity itself to its information content, as enabled by multivariate decoding methods. Representational similarity analysis (RSA) compares the representational structure of signals (Cichy et al., 2014; Kriegeskorte et al., 2008a). However, as decoding approaches have inherent difficulties to identify the sources of decodable information (Carlson et al., 2018; Liu et al., 2018), it is necessary to employ thoughtful control analyses or experiments (Cichy et al., 2015) to disambiguate different possible mechanisms underlying large-scale information structure. This crucially relies on empirical knowledge about processes on the circuit-scale. To bridge the gap between invasive and non-invasive electrophysiology, in the present study, we developed and employed fully human-comparable high-density monkey EEG. We presented identical color and motion stimuli to both human participants and macaque monkeys and combined large-scale recordings on multiple scales, including invasive electrophysiology from six areas across the macaque brain, monkey EEG and human MEG with multivariate decoding and representational similarity analysis. We found color and motion direction information not only in invasive signals, but also in EEG and MEG. We show how motion and color tuning in human MEG can be traced back to the properties of individual units. Our results establish a proof of principle for using large-scale electrophysiology across species and measurement scales to link non-invasive recordings to circuit-level activity. Results To compare information about color and motion direction in invasive and non-invasive signals, we presented rapid streams of dynamic random dot kinematograms (RDKs) with varying color and motion direction to macaque monkeys and humans (Figure 1A). We measured single-unit activity, analog multi-unit activity and local field potentials from multiple microelectrodes in six areas of two macaques, and MEG in eleven human volunteers. In order to establish a link between these data that differed both in species model and measurement technique, we developed non-invasive, human-comparable macaque EEG (for details, see methods). We used custom-made 65-channel caps with miniaturized EEG electrodes to measure scalp-EEG in two animals. This data, matching the invasive recordings in terms of species and the human MEG in terms of signal scale, allowed to relate circuit-level activity to large-scale measurements in humans. After preprocessing, we treated all data types identically and submitted them to the same multivariate pattern analysis (MVPA) of visual information. We used multi-class LDA (Hastie et al., 2009) and a cross-validation scheme to derive time-resolved confusion matrices. For each combination of two stimulus classes A and B, the confusion matrices indicate the average probability of the LDA to assign a trial of class A to class B. From this, we extracted information time courses, latencies, and tuning properties. Figure 1 Download asset Open asset Experimental paradigm, recording and analyses. (A) We presented a stream of random dot patterns with rapidly changing colors and motion directions. After successfully maintaining fixation for a specified time, a liquid reward or auditory reward cue was delivered. (B) Colors and motion directions were sampled from geometrically identical circular spaces. Colors were uniformly distributed on a circle in L*C*h-space, such that they were equal in luminance and chromaticity, and only varied in hue. (C) We performed simultaneous microelectrode recordings from up to six cortical areas. We used custom 65 channel EEG-caps to record human-comparable EEG in macaque monkeys. MEG was recorded in human participants. (D) We used the same multivariate analysis approach on all signal types: Multi-class LDA applied to multiple recording channels resulted in time-resolved confusion matrices, from which we extracted classifier accuracy time courses and tuning profiles. https://doi.org/10.7554/eLife.45645.003 Color and motion direction information in invasive and noninvasive signals We found that information about both motion direction and color was present in all signal types (Figure 2). In LFP, multi-unit and single-unit data, motion and color information were strongest in areas MT and V4, respectively, in line with their established functional roles. Nonetheless, both features were represented ubiquitously (p<0.05, cluster permutation, for most areas apart from motion in IT LFP). Importantly, monkey EEG (Figure 2B and E) and human MEG (Figure 2C and F) also contained information about motion direction and color (p<0.05 for both features in both species, cluster-permutation). Figure 2 Download asset Open asset Color and motion direction information across areas and measurement scales. All panels show classifier accuracy, quantified as the single trial prediction probability for the correct stimulus. Error bars indicate standard error over recording sessions (in macaques) or participants (in humans), horizontal lines show periods of significant information (cluster permutation with p<0.05, corrected for number of regions). (A) Color and (D) motion information is available in most areas in multi-unit, single-unit and LFP data. Information decreases along the cortical hierarchy. Note that MUA color information has very similar timecourses in PFC and FEF, and thus, FEF is barely visible. (B) Color and (E) motion information is available in monkey EEG. Insets: distribution of information in monkey EEG, estimated using source-level searchlight decoding. Information peaks in occipital areas. (C) Color and (F) motion information is available in human MEG. Insets: distribution of information in human MEG, estimated using source-level searchlight decoding. Information peaks in occipital areas. https://doi.org/10.7554/eLife.45645.004 Our analysis of microelectrode recordings showed decreasing information strength along the cortical hierarchy. To test whether this phenomenon was also detectable non-invasively, we performed source-reconstruction of monkey EEG and human MEG data using detailed physical headmodels (see methods, ‘Source reconstruction and searchlight analysis’). We then repeated the MVPA in a searchlight fashion across the cortex. Indeed, for both monkey EEG and human MEG, this revealed gradients of information with strongest information in early visual areas (Figure 2B,C,E,F; insets). To compare the dynamics of feature information, we estimated information latencies as the time point the decoding performance reached half its maximum (Figure 3). For the invasive recordings, latencies were in accordance with the visual processing hierarchy, with information rising earliest in MT for motion direction (SUA: 78 ms, MUA: 81 ms, LFP: 98 ms), earliest in V4 for color (SUA: 82 ms, MUA: 86 ms, LFP: 91 ms), and last in frontal areas. Generally, color information was available earlier than motion direction information in most areas where latencies could be estimated reliably for SUA (V4: p=0.001; IT: p=0.13; MT: p=0.39, random permutation), MUA (V4: p<0.001; IT: p=0.12; MT: p=0.26, random permutation) and LFP (V4: p=0.006; IT: p=0.75; MT: p=0.37, random permutation), consistent with previous results from the same animals in a different task (Siegel et al., 2015). These results translated to the noninvasive signals: Both for monkey EEG (color: 91 ms, motion: 103 ms, p=0.03, random permutation) and human MEG (color: 70 ms, motion: 97 ms, p<0.001, random permutation), color information rose earlier, and the latencies were comparable with those found invasively. Using the searchlight decoding analysis, we again found gradients consistent with the cortical hierarchy, with lowest latencies in occipital and highest latencies in more frontal regions (Figure 3B,C,E,F; insets), as confirmed by correlating source position and estimated latencies (MEG color: p=10−15, MEG motion direction: p=10−4, monkey EEG color: p=0.017, monkey EEG motion direction: p=10−20). Figure 3 Download asset Open asset Color and motion direction information latencies across areas and measurement scales. All panels show normalized classifier accuracy, and latency estimates as well as confidence intervals (bootstrap, 95%). (A) Color and (D) motion information rises first in early visual areas, and last in frontal areas. (B) Color and (E) motion latencies in monkey EEG are comparable to those in early visual areas. Insets: distribution of latencies in monkey EEG, estimated using source-level searchlight decoding. Information rises later in frontal sources than in occipital sources. Marked positions indicate sources for which time courses are shown. (C) Color and (F) motion latencies in human MEG are comparable to those of early visual areas in the macaque brain. Insets: distribution of latencies in human MEG, estimated using source-level searchlight decoding. Information rises later in frontal sources than in occipital sources. https://doi.org/10.7554/eLife.45645.005 MEG color information cannot be explained by luminance confounds Is it plausible that the contents of sensory representations are accessible to noninvasive electrophysiology? It has been shown that, in general, features represented at the level of cortical columns can propagate to decodable MEG and EEG signals (Cichy et al., 2015). Recently, it was reported that information about the motion direction of random dot stimuli can be extracted from EEG signals (Bae and Luck, 2019). This study is, however, to our knowledge the first direct report of color decoding from MEG or EEG. It is conceivable that luminance confounds introduced by imperfections in the color calibration or individual variation in retinal color processing could explain color decoding. To exclude this possibility, we performed a control experiment in a single human subject, in which we manipulated luminance such that each stimulus was presented in a darker and a brighter version. We then used a cross-classification approach to test whether true color information dominated the artificially introduced luminance effect. To this end, we grouped trials such that, for each color, one luminance level was used for training and the other for evaluating the decoder, effectively creating a mismatch of information between test and training data. The color decoder could now, in principle, pick up three sources of information: true color differences, unknown, confounding luminance differences, and experimentally introduced luminance differences. In isolation, these luminance differences should lead to below-chance accuracy. Therefore, any remaining above-chance effect would either indicate that the luminance confound was even stronger than the control manipulation, or that true color information was present. Indeed, we found that classifier accuracy was still significantly above chance (p<0.05, cluster permutation), and undiminished by the luminance manipulation (Figure 4A). Furthermore, we compared the confusion matrices of classifiers trained and tested on dark or bright stimuli, trained on dark and tested on bright stimuli, or vice versa (Figure 4B). All confusion matrices were highly similar, indicating that the representational structure was comparable for low- and high luminance colors. Taken together, this suggests that in our main experiment, equiluminant and equisaturated color stimuli lead to discriminable MEG signatures, and luminance confounds had only a small, if any, effect. Figure 4 with 1 supplement see all Download asset Open asset Control experiments. (A) Time-resolved classifier accuracy. Accuracy is highest when trained and tested on high-luminance stimuli (L1), and lower when trained and tested on low-luminance stimuli (L2). Training on half of the color space in low luminance, and half of the color space in high luminance, and testing on the remainder (G1 x G2), results in accuracy comparable to the low-luminance stimuli alone. (B) Confusion matrices for low-luminance and high luminance stimuli, as well as classifiers trained on low-luminance and tested on high-luminance stimuli and vice versa (L1 x L2, L2 x L1). (C) Maximum color and motion classifier accuracies for all individual sessions. Color is better classified in monkey EEG, motion is better classified in human MEG and EEG. Simultaneously recorded human MEG and EEG results in overall higher accuracy in MEG, but more motion than color information in both. https://doi.org/10.7554/eLife.45645.006 Our stimuli were generated in L*C*h-space, which is designed based on perceptual uniformity in humans. However, it has been shown that color sensitivities in macaque monkeys are highly similar, but not identical to humans (Gagin et al., 2014; Lindbloom-Brown et al., 2014). To ensure that color decoding in the monkey data was not driven by luminance differences, we performed a psychophysical control experiment in a third macaque monkey. Using a minimum-motion technique and eye movements as readout (Logothetis and Charles, 1990), we found that equiluminant colors generated in L*C*h-space were also close to perceptually equiluminant for this monkey (Figure 4—figure supplement 1). Information contained in human EEG is comparable to MEG While in human MEG data, there was more information about motion direction than about color, monkey EEG data showed the opposite effect (Figure 2B,C,E,F). In principle, this could be due to differences in species, measurement modality (EEG or MEG), or differences in the visual stimulation that were beyond our control due to the separate recording environments. To exclude measurement modality as the relevant factor, we acquired simultaneous MEG and EEG data in one of the human participants and compared the amount of motion direction and color information across MEG and EEG data. All monkey EEG recording sessions contained more information about color, and all human MEG recordings contained more information about motion direction. Notably, the human EEG session was consistent with the MEG results. While information was generally lower for EEG than for simultaneous MEG, EEG showed the same dominance of motion information (Figure 4C). This suggests that the differences of information dominance between human MEG and monkey EEG were not due to the recording modality. Representational similarity analysis Having established the presence of information in all signal types, we next asked how the representational structure of motion direction and color varied across brain areas, species, and measurement scales. To address this, we performed representational similarity analysis (Kriegeskorte et al., 2008a) (RSA) on the LDA confusion matrices averaged over a time window during which visual information was present (50–250 ms). In short, we used RSA to compare patterns of similarity between stimulus classes, as given by the confusion matrices, across areas and signal types. First, we sought to characterize the diversity of representations across the six areas measured invasively (Figure 5A). For color information, we found that representations were highly similar between SUA, MUA and LFP, as well as between all six cortical areas (p<0.05 for most pairs of areas and measures, uncorrected), indicating that a single representational structure was dominant across the brain. In the case of motion direction, areas were split into ventral stream visual areas (IT and V4) and frontal and dorsal visual stream areas (MT, LIP, FEF, PFC). Within each of these two groups, there were again high correlations between areas and measures, but we found no significant similarity between the groups. Figure 5 Download asset Open asset Representational similarity between areas and measurement scales. (A) Similarity between SUA, MUA and LFP color (top) and motion direction representations (bottom) in six areas of the macaque brain, masked at p<0.05 Color representations are highly similar between all motion representations are split between and ventral areas. (B) and (C) Similarity between monkey EEG and human MEG color and motion representations and those in SUA, MUA and LFP in six areas. color representations are similar to all areas, motion representations are similar to IT and V4 representations (p<0.05, random permutation corrected for number of (D) Color and motion representations are similar between human MEG and monkey EEG p<0.001, random permutation How information contained in recorded activity relate to information in large-scale EEG We found that the color in macaque EEG was highly similar to those of SUA, MUA and LFP in all six areas, while the EEG motion direction only the ventral stream areas V4 and IT (Figure p<0.05 for IT SUA and V4 SUA, MUA and LFP, random permutation, corrected for number of Notably, we found no motion direction similarity between MT and EEG (SUA: MUA: LFP: This that, MT contained a large amount of motion direction information, EEG signals were dominated by activity from areas with or motion direction We found similar results when invasive data to human MEG; there were similarities between color representations in all areas and human MEG, as well as between motion direction representations in V4 and IT and human MEG (Figure Furthermore, both color and motion direction representations were highly similar between monkey EEG and human MEG (Figure color: motion: Similarity is explained by tuning properties Color representations were similar across the brain, while motion direction representations were into two only one of which translated to non-invasive signals. To to these we the underlying representations more Figure 6 the color and motion direction confusion matrices for MT and V4 multi-unit activity as well as for monkey EEG and human MEG. All color confusion matrices a pattern decreasing with from the This that neural activity in all areas, signals and both species matched perceptual in color We found a similar of motion direction in Figure 6 Download asset Open asset Color and motion direction Color and motion direction tuning in (A) MT V4 (B) monkey EEG and (C) human MEG. is, for each or signal the prediction probability as a of the between true and stimulus. The dark line the probability of a stimulus the and lines the probability of a stimulus as the opposite in the circular stimulus the confusion indicating prediction for all stimulus A tuning indicating prediction as a of between true and stimulus at the time of accuracy. For color, tuning is with the lowest For motion direction, V4, EEG and MEG, but not MT tuning is with a higher probability than stimulus However, motion direction representations in V4, monkey EEG and human MEG a in similarity on the opposite to the true motion direction, indicating that these signals to to motion in opposite directions. To the dynamics of this we the confusion matrices over stimuli, which results in prediction as a of the between true and stimuli (Figure the in the confusion matrices translated to an probability of a stimulus to be as the one opposite in stimulus all color stimuli were to be classified as the opposite color, there was an probability for motion directions to be identified as the In terms of this to tuning (Figure We quantified the presence of such tuning across areas and measurement scales by the in prediction probability between opposite and and stimuli, normalized by the of prediction (Figure This revealed that motion direction tuning was significantly in V4 and IT as well as monkey EEG and human MEG, but not for any of the more dorsal or frontal areas. was no significant color tuning for any or measurement Figure Download asset Open asset Motion direction tuning across areas and measurement scales representational of color (A) do not show representations of motion (B) do in IT, V4 and non-invasive data, but not in frontal and dorsal areas. of representational similarity between SUA, MUA and LFP in all areas with differences in In case of color there is no in case of motion there is a tuning of individual single units or LFP channels in MT and Color tuning is in both areas, motion tuning is in We used to the of differences to the pattern of similarity between invasively measured areas and signal types (Figure and To this end, we differences in between each combination of SUA, MUA and LFP, and all areas. We then to these differences in for the in representational Importantly, in the case of motion direction, could largely explain the pattern of representational similarity between areas and measures This was not the case for the differences in color which not representational similarity Thus, similar motion direction to V4 and IT similar motion which were also similar to those in monkey EEG and human MEG. Motion direction is present in individual SUA, MUA and LFP channels we asked on which level the motion direction The presence of a effect in MEG, EEG, LFP, multi-unit and data suggests that it was not by in the large-scale of motion direction but by properties of individual if individual single or or LFP were to between opposite motion a multivariate analysis of several channels would be to also this We motion direction tuning to be in those areas which a multivariate effect. To test this, we all tuning