© 2005 Nature Publishing Group http://www.nature.com/natureneuroscience

ARTICLES

Decoding the visual and subjective contentsof the human brain
Yukiyasu Kamitani1 & Frank Tong2,3
The potential for human neuroimaging to read out the detailed contents of a person’s mental state has yet to be fully explored. We
investigated whether the perception of edge orientation, a fundamental visual feature, can be decoded from human brain activity
measured with functional magnetic resonance imaging (fMRI). Using statistical algorithms to classify brain states, we found thatensemble fMRI signals in early visual areas could reliably predict on individual trials which of eight stimulus orientations the
subject was seeing. Moreover, when subjects had to attend to one of two overlapping orthogonal gratings, feature-based attention
strongly biased ensemble activity toward the attended orientation. These results demonstrate that fMRI activity patterns in earlyvisual areas, including primary visual cortex (V1), contain detailed orientation information that can reliably predict subjective
perception. Our approach provides a framework for the readout of fine-tuned representations in the human brain and theirsubjective contents.

Much remains to be learned about how the human brain represents
basic attributes of visual experience. It is commonly assumed thathuman visual perception is based on the neural coding of fundamental
features, such as orientation, color, motion and so forth. Edge orientation is arguably the most fundamental feature analyzed by the visualsystem, providing the basis for defining the contours and shapes ofvisual objects that allow for object segmentation and recognition. Thereis considerable neurophysiological evidence of orientation tuning in
cortical columns and single neurons in both monkey and cat visualcortex1–4. However, non-invasive neuroimaging methods have beenthought to lack the resolution to probe into these feature representationsin the human brain. As a consequence, little is known about the neuralbasis of orientation selectivity in the human visual system or how thesefundamental features are represented during conscious perception.
Here we investigated whether it is possible to read out detailed
information about the visual and subjective contents of the humanbrain using fMRI. Specifically, can activity patterns in the human visualcortex reveal what stimulus orientation a person is viewing or attending
to on a given trial? We present a new approach to measure feature
selectivity from an ensemble of neuroimaging signals, which we call‘ensemble feature selectivity’. Our approach draws on ideas of population coding5,6 and multi-voxel pattern analysis of fMRI data7–9 todemonstrate neural decoding of perceived orientations and a methodof ‘mind-reading’ that is capable of classifying mental states on the basisof measured brain states. Early versions of this work have beenpresented at scientific conferences (Y.K. and F.T., J. Vis. 4, 186a, 2004;
Y.K. and F.T., Soc. Neurosci. Abstr. 370.7, 2004).
A challenge in examining orientation selectivity in the human visualcortex is that putative orientation columns may be too finely spaced toresolve using current fMRI techniques. Orientation-selective columns
in the monkey are only about 300–500 mm in width10, whereas thespatial resolution of human fMRI is limited by many factors. These
include technical limitations of human fMRI, reductions in signal-tonoise proportional to the volume of each voxel, spatial blurring of thepositive blood oxygenation level-dependent (BOLD) hemodynamic
response extending several millimeters beyond the site of neuralactivity11–13, and additional blurring induced by residual head motion.
To bypass these spatial limitations, we developed an alternative
approach of measuring the ensemble orientation information contained in the activity pattern of many voxels. We hypothesized that eachvoxel, sampled from a 3  3  3 mm region of human visual cortex,
may have a weak but true bias in its neural or hemodynamic response todifferent orientations. Such biases could arise from random variationsin the distribution or response gain of orientation columns within eachvoxel. Orientation columns in the monkey typically reveal such spatial
variability, and these variations seem to be stable over time14. Even if
one were to assume that the spatial distribution of orientation columns
is perfectly uniform, variability in the distribution of vasculature would
lead to uneven hemodynamic sampling across orientation columns,
resulting in local biases in orientation preference. We predicted that bypooling together the information from many weakly tuned voxels, theensemble activity pattern of many voxels may show sharp and stableselectivity for orientation (‘ensemble orientation selectivity’). Our
experiments showed that different stimulus orientations give rise todistinct patterns of fMRI activity in early human visual areas, which canbe accurately decoded by linear pattern analysis techniques. Furthermore, these orientation-selective activity patterns allow for reliableneural decoding of the subjective contents of perception.

1ATR Computational Neuroscience Laboratories, 2-2-2 Hikaridai, Keihanna Science City, Kyoto 619-0288, Japan. 2Psychology Department, Princeton University, Green
Hall, Princeton, New Jersey, 08544, USA. 3Present address: Psychology Department, Vanderbilt University, 301 Wilson Hall, 111 21st Avenue South, Nashville, Tennessee
37203, USA. Correspondence should be addressed to Y.K. (kmtn@atr.jp).

Published online 24 April 2005; doi:10.1038/nn1444

NATURE NEUROSCIENCE VOLUME 8 [ NUMBER 5 [ MAY 2005

679

ARTICLES
prediction

a

b

1

k

2

45°
W

© 2005 Nature Publishing Group http://www.nature.com/natureneuroscience

135° detector

45° detector

Max
Linear ensemble
orientation
detectors

fMRI
voxelsVoxel #50

W

135°

Voxel #100

(S1, V1N2 400 voxels)

Figure 1 Orientation decoder and ensemble orientation selectivity. (a) The
orientation decoder predicts stimulus orientation on the basis of fMRI activity
patterns. The cubes depict an input fMRI activity pattern obtained while thesubject viewed gratings of a given orientation (left). The circles are ‘linear
ensemble orientation detectors,’ each of which linearly combines the fMRI
voxel inputs (weighted sum plus bias; bias component not shown). The
weights (W) are determined by a statistical learning algorithm (linear support
vector machine) applied to a training data set, such that the output of eachdetector becomes largest for its ‘preferred orientation’ (yi). The final unit
(rectangle with Max) decides the prediction to be the preferred orientation of
the detector with the highest value. (b) Orientation selectivity of individual
voxels and linear ensemble orientation detectors. The decoder was trainedusing actual fMRI responses to eight orientations (S1, 400 voxels from
V1/V2). Average responses are plotted as a function of orientation for two
representative voxels, and for 451 and 1351 detectors (error bar, s.d.).

RESULTS
Subjects viewed one of eight possible stimulus orientations while
activity was monitored in early visual areas (V1–V4 and MT+) using
standard fMRI procedures (3T MRI scanner, spatial resolution 3  3 
3 mm; Methods). For each 16-s ‘trial’ or stimulus block, a square-wave
annular grating (Fig. 1a) was presented at the specified orientation (0,22.5, y, 157.51), and flashed on and off every 250 ms with arandomized spatial phase to ensure that there was no mutual information between orientation and local pixel intensity. Subjects maintained
steady fixation throughout each fMRI run, during which each of theeight stimulus orientations was presented in randomized order. Subjects viewed a total of 20–24 trials for each orientation.
Orientation decoder and ensemble feature selectivity
We constructed an ‘orientation decoder’ to classify ensemble fMRI
activity on individual trials according to stimulus orientation, based on
the orientation-selective activity pattern in visual cortex that was
obtained from a training data set (Fig. 1a). The input consisted of
the average response amplitude of each fMRI voxel in the visual area(s)of interest, for each 16-s stimulus trial. In the next layer, ‘linear ensemble orientation detectors’ for each of the eight orientations received

Figure 2 Decoding stimulus orientation from ensemble fMRI activity in thevisual cortex. (a) Decoded orientation responses for eight orientations. Polarplots indicate the distribution of predicted orientations for each of eightorientations (S2, 400 voxels from V1/V2, 22 samples per orientation). The
same values are plotted at symmetrical directions as stimulus orientation
repeats every 1801. Solid black lines show the true stimulus orientations.
(b) Decoded orientation responses for all four subjects (400 voxels from
V1/V2, total 160–192 samples for each subject). Results for individual
orientations are pooled relative to the correct orientations, and aligned to thevertical line. (c) Across-session generalization. Decoded orientation responses
were obtained by training a decoder with day 1’s data and testing with day 2’sdata (31 d and 40 d apart for S1 and S2, respectively).

680

weighted inputs from each voxel and calculated the linearly weighted
sum as output. Individual voxel weights for each orientation detector
were determined by using a statistical learning algorithm applied to anindependent training data set15. Voxel weights were optimized so thateach detector’s output became larger for its preferred orientation thanfor other orientations (Methods). The final output prediction wasmade by selecting the most active linear ensemble orientation detector
as representing the orientation most likely to be present.
We first trained the orientation decoder using 400 voxels from areas
V1/V2 for individual subjects (Fig. 1b). Individual voxels showed poor
response selectivity for different orientations (Supplementary Figs. 1
and 2 online). Nonetheless, the output of the linear ensemble orientation detectors, which reflect the weighted sum of many individual voxel
responses, had well-tuned responses centered around the preferredorientation of each detector. Furthermore, the detectors showed agraded response that increased according to the similarity of stimulus
orientation to their preferred orientation. Because the similarity amongorientations was not explicitly specified in the learning procedure, thisgraded response indicates that similar orientations give rise to moresimilar patterns of fMRI activity. These results suggest that theensemble pattern of fMRI activity contains orientation informationthat greatly exceeds the selectivity of individual voxels.
Orientation decoding accuracy across visual areas
We evaluated if fMRI activity patterns in the human visual cortex are
sufficiently reliable to predict what stimulus orientation the subject isviewing on individual trials. We used a cross-validation analysis inwhich the orientation of each fMRI sample was predicted after theorientation decoder was trained with the remaining samples. Therefore, independent samples were used for training and test. Ensemble
fMRI activity in areas V1/V2 led to precise decoding of which of theeight orientations the subject saw on individual stimulus trials(Fig. 2a). Decoded orientation responses peaked sharply at the trueorientation, with errors, which were infrequent, occurring primarily atneighboring orientations and rarely at orthogonal orientations. The
accuracy of these decoded orientation responses was quantified for all
four subjects, by calculating the root mean squared error (RMSE)
between the true and the predicted orientations, which were 17.91,
21.01, 22.21 and 31.21 for subjects S1–S4, respectively (Fig. 2b).
In general, orientation decoding performance progressively
improved with increasing numbers of voxels, as long as voxels wereselected from the retinotopic region corresponding to the stimulateda(S2, V1/V2
400 voxels)

b

S1

S2

c

Training on day 1
Test on day 2

S3

S1

S2

31 d apart

40 d apart

S4(V1/V2 400 voxels)

VOLUME 8 [ NUMBER 5 [ MAY 2005 NATURE NEUROSCIENCEARTICLES
V1d

V2d

V3d

Figure 3 Orientation selectivity across the human visual pathway. Decodedorientation responses are shown for individual visual areas from V1 throughV4 and MT+ (S3, 100 voxels per area). The color map indicates t-values
associated with the responses to the visual field localizer for V1 through V4,
and to the MT+ localizer for MT+ (see Methods). The voxels from bothhemispheres were combined to obtain the results, though only the righthemisphere is shown. All other subjects showed similar results ofprogressively diminishing orientation selectivity in higher areas.

V3a

10

2

(S3, 100
voxels/area)

t

V1v

V2v

V3v

V4v

visual field. Voxels that showed stronger orientation preference wereconfined well within retinotopic boundaries of the annular stimulus(Supplementary Fig. 3), suggesting the retinotopic specificity of theseorientation-selective signals. Consistent with this notion, fMRI activity
patterns from unstimulated regions around the foveal representation in
V1 and V2 led to chance levels of orientation decoding performance.
Although our subjects were well trained at maintaining stable fixation,
it is conceivable that different orientations might elicit small but
systematic eye movements that could alter the global pattern of corticalvisual activity. However, additional control experiments showed thatwhen independent gratings (451 or 1351) were presented simultaneously to each hemifield, visual activity in each hemisphere could
accurately decode the orientation of the contralateral stimulus but
not the ipsilateral stimulus (96.1% versus 54.9% correct using 200
voxels from V1/V2 for each hemisphere; chance, 50%; Supplementary
Fig. 4). The independence of orientation information in the twohemispheres cannot be explained in terms of eye movements or any
other factors that would lead to global effects on cortical activity.
We further investigated the physiological reliability of these orientation signals in the human visual cortex by testing generalization acrossseparate sessions in two subjects. This was done by training theorientation decoder with fMRI activity patterns from one day andusing it to predict perceived orientation with the fMRI data fromanother day (Fig. 2c). The RMSEs for the across-session generalization
were 18.91 (31 d apart) and 21.71 (40 d apart) for subjects S1 and S2,
respectively, almost as small as those for within-session generalization(17.91 and 21.01, respectively). The results indicate that these orientation-selective activity patterns reflect physiologically stable response
properties across the visual cortex.
The ability to extract robust orientation information from ensemble
fMRI activity allowed us to compare orientation selectivity acrossdifferent human visual areas. Orientation selectivity was most pronounced in early areas V1 and V2, and declined in progressively higher
visual areas (Fig. 3). All four subjects showed this same trend of diminishing orientation selectivity across retinotopic visual areas (RMSEs,
mean 7 s.d., for ventral V1, V2, V3 and V4 were 31 7 41, 29 7 41,
40 7 81 and 46 7 41, respectively). This pattern of orientation
selectivity is consistent with monkey data showing poorer orientation
selectivity and weaker columnar organization in higher visual areas10,

NATURE NEUROSCIENCE VOLUME 8 [ NUMBER 5 [ MAY 2005

but has never been shown in the human visual cortex. Unlike areas V1
through V4, human area MT+ showed no evidence of orientation
selectivity (53.2 7 3.71; chance level, 52.81), consistent with the ideathat this region is more sensitive to motion than to stimulus form.
Source of orientation information
Additional analyses confirmed that this orientation information
obtained from the human visual cortex reflects actual orientationdependent responses. Our linear ensemble orientation detectors wereunable to discriminate the orientation of the phase-randomized
stimulus gratings on the basis of pixel intensity values alone because
orientation is a higher-order property that cannot be expressed by alinearly weighted sum of inputs16. However, the decoder composed oflinear detectors could classify these images when they received input
from intervening orientation filters with nonlinearity, analogous to V1
neurons (Fig. 4). The results indicate that ensemble orientation
selectivity does not arise from the retinotopic projection of bitmap
grating images on the cortex but rather from the orientation information inherent in individual voxels, which can then be pooled together.
What is the pattern of orientation preferences among these voxelsthat is responsible for such precise ensemble selectivity? We plotted theorientation preference of individual voxels on flattened cortical representations, coloring voxels according to the orientation detector forwhich each voxel provided the largest weight (Fig. 5). Voxel orientation

(I)

(II)
100
Percentage correct© 2005 Nature Publishing Group http://www.nature.com/natureneuroscience

MT+

50
(I)
(II)
(III)
0

Noise

22.5

45.0

67.5

90.0

Orientation difference (°)(III)

Figure 4 Pairwise decoding performance as a function of orientationdifference (all pairs from eight orientations), for grating images (pixel
intensities), fMRI images (voxel intensities) and transformed grating images.
The gratings (I) were 20  20 pixel black-white images with 2–3 stripes.
The fMRI images (II) were those obtained in the present study (responses
to gratings of eight orientations; 400 voxels from V1/V2). The transformed
images (III) were created by linear orientation filtering (Gabor-like filters forfour orientations) of the grating images followed by thresholding (nonlinearity)
and addition of noise. The orientations of these images were decoded foreach pair of orientations (chance level, 50%). For (I) and (III), the average
performance with five sets of phase-randomized images is plotted (error bar,s.d.). For (II), the average performance of four subjects is shown. The grating
images (I) resulted in poor performance regardless of orientation difference.
In contrast, the fMRI images (II) and the transformed grating images (III)both showed performance that improved with orientation difference, reachingnear perfect levels at 901.

681

ARTICLES
S2

S3

V2 V1

© 2005 Nature Publishing Group http://www.nature.com/natureneuroscience

V2v

V1v

V2v

V1v

Figure 5 Orientation preference map on flattened cortical surface. The color
maps depict the orientation preference of individual voxels on the flattenedsurface of left ventral V1 and V2 for subjects S2 and S3 (scale bar, 1 cm).
Each cell delineated by thick lines is the cross-section of a single voxel(3  3  3 mm) at the gray-white matter boundary. Voxel colors depict theorientation detector for which each voxel provides the largest weight. The
overall color map indicates a template pattern that activates each detector
most effectively. The weights were calculated using 400 voxels from V1/V2,
including all the quadrants. Other subjects also showed scattered but
different patterns of orientation preference. Note that the color map indicates
only weak preference for one orientation over others. Simple averaging of thevoxels with the same orientation preference led to weak orientation tuning(Supplementary Fig. 2), unlike the well-tuned responses of the optimallyweighted linear orientation detectors (Fig. 1b).

preferences revealed a scattered pattern that was variable and idiosyncratic across subjects. Although some local clustering of the same orientation preference was observed, much of this may reflect spatial blurring
resulting from subtle head motion, data reinterpolation required for the
Talaraich transformation and the blurred nature of BOLD hemodynamic responses12,13. There were no significant differences in the proportion of preferred orientations across different visual areas or different
visual field quadrants. Overall, orientation preference maps revealedconsiderable local variability, indicating that global bias effects due toeye movements or other factors cannot account for the high degree oforientation-selective information that resides in these activity patterns.
Additional analyses showed that orientation selectivity remained
equally robust even when the fMRI data underwent normalization
to remove differences in mean activity levels across individual
activity patterns. Also, differences in mean activity level were small(Supplementary Fig. 1), even between canonical and oblique orientations (the oblique effect17). Therefore, gross differences in response
Figure 6 Simulation of one-dimensional array of columns and voxels. (a,b)
Each column was assumed to respond to orientation input according to a
Gaussian-tuning function peaking at its preferred orientation (s.d., 451; noise
was added to the output). The preferred orientation shifted by a constantdegree (a) or by a constant degree plus noise (b). In each trial, a single
orientation was given as input, and the outputs of 100,000 columns (colorband) were sampled by 100 voxels (gray boxes). The actual location of voxel
sampling was randomly jittered on each trial (Gaussian distribution with s.d.
of a quarter voxel size) to take into account residual head motion. The
number of stimulation trials was chosen to match the fMRI experiment. The
sampled voxel data were analyzed using the same decoding procedure. As
can be seen in the polar plots on the right, orientation can be readily decodedfrom the irregular array of columns (b), but not from the regular array (a).
Similar results were obtained with a wide range of simulation parameters.
Note that if voxel sampling is no longer jittered to mimic minor brain motion,orientation can be decoded even from the regular column array. This isbecause the high spatial frequency component can still persist after the
sampling by large voxels. However, given that pulsatile brain motion andminor head motion cannot be fully eliminated or corrected with 3D alignment
procedures, it seems unlikely that such high-frequency information
contributes much to the orientation content in our fMRI data.

682

amplitudes were not a critical source of orientation information for our
decoding analysis. We also tested if a global bias for radial orientations
might account for our results, as some studies have reported evidence ofa bias for orientations radiating outward from the fovea in retinal
ganglion cells and V1 neurons of the monkey18,19. We removed theglobal response modulation along the radial dimension from eachactivity pattern, by dividing voxels into the 16 polar-angle sections
obtained from retinotopic mapping, and then normalizing the mean
response within each set of iso-polar voxels to the same value for every
stimulus trial. After this normalization procedure, orientation selectivity diminished only slightly. The mean RMSEs of four subjects fororiginal and normalized data were 22.7 7 6.21 and 26.7 7 7.11,
respectively (200 voxels from V1/V2). Although global factors, such asradial bias, might account for a small degree of the extracted orientation
information, local variations in orientation preference seem to provide
the majority of the orientation content in these fMRI activity patterns.
The scattered distribution and local variability of orientation preferences across cortex (Fig. 5) are consistent with the idea that randomvariations in the distribution or response strength of individual
orientation columns can lead to small local orientation biases thatremain detectable at voxel-scale resolutions. To evaluate the viability ofthis hypothesis, we performed simulations using one-dimensional
arrays of orientation columns, which were sampled by coarse-scale
voxels and analyzed by our orientation decoder. The array of voxels wasallowed to jitter randomly from trial to trial to mimic the effects ofsmall amounts of brain motion. We compared two types of column
arrays, one with regularly shifting preferred orientations (Fig. 6a) andthe other with small random variations in the shifted orientation(Fig. 6b), as can be observed in columnar structures in animals.
Whereas the regular array showed poor orientation decoding performance, the array with random variation resulted in very similar
performance to what was found from actual fMRI activity patterns in
the human visual cortex (Fig. 2). These results support the hypothesisthat a small amount of random variability in the spatial distribution oforientation columns could lead to small local biases in individual voxelsthat allow for robust decoding of ensemble orientation selectivity.
Mind-reading of attended orientation
Finally, we asked if the ability to characterize brain states corresponding to different stimulus orientations can be extended to the problemof mind-reading, that is, determining a subject’s mental state given

a

b
180°

90°

0°

VOLUME 8 [ NUMBER 5 [ MAY 2005 NATURE NEUROSCIENCE

Figure 7 Mind-reading of attended orientation. (a) Procedure. First, a
decoder was trained using fMRI activity evoked by single gratings todiscriminate 451 versus 1351. Black and white gratings (equal contrast) werecounterbalanced across trials. In the attention experiment, a plaid patterncomposed of two gratings (black-white orientation assignment counterbalanced) was presented. The color of the central fixation spot was changed
to indicate to which orientation (451 or 1351) the subject should attend ineach trial. The fMRI activity patterns obtained in the attention experimentwere classified by the decoder trained with single gratings. (b) Performance.
Gray lines plot decoded orientation responses for the ‘attend to 451’ and‘attend to 1351’ conditions (S3, 800 voxels from V1–V4). Solid black lines
indicate attended orientations. (c) Performance across the human visual
pathway. The percentage of correct decoding is plotted by visual area (chancelevel, 50%; 800 voxels for V1–V4 combined, 200 voxels for V1, V2, V3 andV3a/V4v, and 100 voxels for MT+). Colored lines show the performance offour individual subjects. Black points and lines depict the mean crossvalidation performance obtained with single gratings (training session).

aknowledge of his or her brain state. Specifically, we tested if theactivity patterns evoked by unambiguous single orientations can beused to decode which of two competing orientations is dominant in aperson’s mind under conditions of perceptual ambiguity. We hypothesized that activity in early human visual areas, which was found here to
represent unambiguous stimulus orientations, may subserve a common role in representing the subjective experience of orientation when
two competing orientations are viewed. If so, then when subjects are
instructed to attend to one of two overlapping gratings (that is, plaid),
activity patterns in early visual areas should be biased toward theattended grating.
First, subjects viewed single gratings of 451 or 1351 orientation(Fig. 7a, top; black and white counterbalanced), and the resulting fMRI
activity patterns were used to train the orientation decoder. Then in
separate test runs, subjects viewed a plaid stimulus consisting of both
overlapping gratings (Fig. 7a, bottom). In each 16-s trial, subjects wererequired to attend to one oriented grating or the other by monitoringfor small changes in the width of the bars in the attended grating while
ignoring changes in the unattended grating. The fMRI data obtained
while subjects viewed the two competing orientations were analyzed
using the decoder trained on fMRI activity patterns evoked by thesingle gratings.
Orientation signals in early human visual areas were strongly biased
toward the attended orientation during viewing of the ambiguous
stimulus (Fig. 7b,c). Even though the same overlapping gratings were
presented in the two attentional conditions, fMRI activity patterns in
the visual cortex reliably predicted to which of the two orientations the
subject was attending on a trial-by-trial basis at overall accuracy levels
approaching 80% (P o 0.0005 for 4/4 subjects using 800 voxels from
V1–V4, chi square test). Analyses of individual visual areas showed thatensemble activity was significantly biased toward the attended orientation in all four subjects for area V1 and also V2 (P o 0.05), and in
3/4 subjects for area V3 and areas V3a/V4v combined.
We did an additional control experiment to address whether eye
movements, orthogonal to the attended orientation, might account for
the enhanced responses to the attended grating by inducing retinal
motion. The visual display was split into left and right halves, andactivity from corresponding regions of the contralateral visual cortex
was used to decode the attended orientation in each visual field(Supplementary Fig. 4). Even when the subject was instructed to pay
attention to different orientations in the plaids of the left and rightvisual fields simultaneously, cortical activity led to accurate decoding ofboth attended orientations. Because eye movements would bias only
one orientation in the whole visual field, these results indicate that theattentional bias effects in early visual areas are not due to retinal motion
induced by eye movements.
The robust effects found in V1 and V2 suggest that top-down
voluntary attention acts very early in the processing stream to bias
orientation-selective signals. Although previous studies have reported evidence of feature-based attentional modulation in thevisual cortex20–25, our results provide novel evidence that top-down
attention can bias orientation signals at the earliest stage of corticalprocessing when two competing stimuli are entirely overlapping. These
results suggest that feedback signals to V1 and V2 may have an
important role in voluntary feature-based attentional selection oforientation signals.

NATURE NEUROSCIENCE VOLUME 8 [ NUMBER 5 [ MAY 2005

Trainingb

Attend to 45°

Attend to 135°

45° vs. 135°

(S3, V1–V4
800 voxels)

c 100
Test
Attend to 45° or 135°

Percentage correct© 2005 Nature Publishing Group http://www.nature.com/natureneuroscience

ARTICLES

Single gratings(mean of S1–S4)

50
Plaid/attention
0

V1–V4

V1

V2
V3
Area

S1
S2
S3
S4
V3a
/V4v

MT+

DISCUSSION
We have shown that fMRI activity patterns in the human visual cortex
contain reliable orientation information that allows for detailed prediction of perceptual and mental states. By combining an ensemble of
weakly orientation-selective fMRI voxels, we could accurately differentiate subtle variations in perceived stimulus orientation on a trial-bytrial basis. Moreover, activity patterns evoked by unambiguous stimuli
could reliably predict which of two competing orientations was thefocus of a subject’s attention. These results demonstrate a tightcoupling between brain states and subjective mental states.
Models of human visual perception commonly assume orientation-selective units, similar to those found in animals, to account for avariety of psychophysical data26. However, neurophysiological evidence
to support the existence of such mechanisms in the human brain
has been indirect27–29. Here, we found that early human visualareas were indeed highly orientation selective, and that ensemble
orientation selectivity was most pronounced in areas V1 and V2
and progressively weaker in higher areas, consistent with neurophysiological data in monkeys10. Our results suggest that orientation
selectivity in the human visual system may closely resemble that ofnonhuman primates.
Previous neuroimaging studies have used multi-voxel pattern analysis to reveal broadly distributed object representations extending over
1–5 cm regions of the ventral temporal cortex7. Our approach formeasuring ensemble feature selectivity suggests that multi-voxel analyses may also be effective at extracting feature-tuned information atmuch finer scales of cortical representation, by pooling together weakfeature-selective signals in each voxel, which may arise from variability
in the distribution of cortical feature columns or their vascular supply.
If this is indeed the case, then the approach proposed here could be
used to investigate a variety of feature domains, such as color selectivity,683

© 2005 Nature Publishing Group http://www.nature.com/natureneuroscience

ARTICLES
motion selectivity or feature tuning in other sensory and motor
modalities, and thus may provide a useful bridge between animalstudies and human neuroimaging studies.
We emphasize the importance of using linear approaches to study
ensemble feature selectivity. Our analysis relied on a linear weighting
procedure to measure the orientation-selective information inherent ineach voxel and to pool together the ensemble information from many
voxels. In contrast, nonlinear pattern analysis techniques could extractorientation information even if every individual voxel lacked any
orientation selectivity. Flexible nonlinear approaches would allow for
nonlinear interactions between input voxels; these could be used toconstruct nonlinear oriented filters to decode orientation on the basisof pixel intensity information alone. Therefore, nonlinear methods may
spuriously reflect the feature-tuning properties of the pattern analysis
algorithm rather than the tuning properties of individual units within
the brain. For these reasons, it is important to restrict the flexibility ofpattern analysis methods when measuring ensemble feature selectivity.
Finally, our method of measuring ensemble feature selectivity proved
highly effective at decoding mental states from measured brain states.
Our results on the mind-reading of attended orientation address an
ongoing debate regarding the role of early visual areas and primaryvisual cortex in visual awareness30,31. Voluntary top-down attention
strongly biased the pattern of orientation responses at the earliest stagesof cortical processing in V1 and V2. These results suggest that feedback
projections to these early visual areas may be important in featurebased attentional selection and in maintaining a particular orientationin awareness. More generally, the mind-reading approach presented
here provides a potential framework for extending the study of theneural correlates of subjective experience32. True understanding of theneural basis of subjective experience should allow for reliable predictionof a person’s mental state based solely on measurements of his or her
brain state. By analyzing ensemble activity from human cortex, we wereable to extract information about the person’s subjective mental state
under conditions of ambiguity. Our approach may be extended to
studying the neural basis of many types of mental content, including aperson’s awareness, attentional focus, memory, motor intention andvolitional choice. Further development of such mind-reading
approaches may eventually lead to potential applications for noninvasive brain-machine interface33,34 by providing effective proceduresto translate brain activity into mental contents.
METHODS
Subjects. Four healthy adults with normal or corrected-to-normal visualacuity participated in this study. All subjects gave written informed consent.
The study was approved by the Institutional Review Panel for Human Subjects
at Princeton University.
Experimental design and stimuli. Visual stimuli were rear-projected onto ascreen in the scanner bore using a luminance-calibrated LCD projector driven
by a Macintosh G3 computer.
Each experimental run to measure fMRI responses to gratings consisted of aseries of 16-s stimulus trials (with no intervening rest periods), plus 16-s
fixation-rest periods at the beginning and at the end of each series.
In the eight-orientation experiment, each run had eight trials for eight
different orientations (01, 22.51y157.51). In each trial, a square-wave annulargrating of a given orientation (B100% contrast, 1.5 cycles per degree; 1.51–101
of eccentricity) was flashed at 2 Hz (on/off for 250 ms). The spatial phase of thegrating was randomized in each frame so that the pixel intensity did not carryinformation about orientation. Because of the long temporal interval between
flashed gratings, apparent motion among the gratings was hardly visible35. The
subject passively viewed the stimulus while maintaining fixation on a central
fixation spot. The order of orientation conditions was randomized in each run.
Each subject performed 20–24 runs for a total of 20–24 trials per orientation.

684

In the attention experiment, we interleaved two types of runs, training andtest runs. In each trial of the training runs, either a 451 or 1351 grating (blackor white stripes; 0.75 cycles per degree) was flashed on a gray background at
0.5 Hz (on for 1,750 ms, off for 250 ms) with a randomized spatial phase. The
subject monitored the thickness of the stripes, and reported whether they were
thick or thin (one- or two-pixel difference) by pressing a key within each 2-s
stimulus frame. A single run had 16 trials, and 6 runs were repeated in eachsubject. The order of orientation (451 or 1351) and stripe color (black or white)
was randomized.
In the test runs of the attention experiment, two overlapping gratings of 451
and 1351 (white and black, or black and white; counterbalanced across trials)
were flashed with the same time course as in the training runs. The intersections of the white and black stripes were gray to enhance the perceptual
segregation of the two gratings. During each 16-s trial, the color of the fixation
spot was either red or green (randomized), indicating which grating (451 or
1351) the subject should monitor for changes in thickness. (The colororientation rule was reversed in the middle of the experiment.) A single run
had 16 trials, and 6 runs were repeated in each subject. The overall performanceof the task was 79 7 5% correct with a 50% chance level. Thus, the task was
difficult enough to restrict attention to one of the gratings.
In the same session, subjects viewed a reference stimulus to localize theretinotopic regions corresponding to the stimulated visual field. The ‘visual
field localizer’ composed of high-contrast dynamic random dots was presented
in an annular region for 12-s periods, interleaved with 12-s rest/fixation
periods, while the subject maintained fixation. We used a smaller annularregion for the visual field localizer (21–91 of eccentricity) than for the gratings(1.51–101) to avoid selecting voxels corresponding to the stimulus edges, which
may contain information irrelevant to grating orientation. In separate sessions,standard retinotopic mapping12,36 and MT+ localization procedures37–39 weredone to delineate visual areas on flattened cortical representations.
MRI acquisition. Scanning was performed on a 3.0-Tesla Siemens MAGNETOM Allegra scanner using a standard head coil at the Center for the Study ofBrain, Mind and Behavior, Princeton University. A high-resolution T1weighted anatomical scan was acquired for each participant (FOV 256 
256, 1 mm3 resolution). To measure BOLD contrast, standard gradient-echo
echoplanar imaging parameters were used to acquire 25 slices perpendicular tothe calcarine sulcus to cover the entire occipital lobe (TR, 2,000 ms; TE, 30 ms;
flip angle, 901; slice thickness, 3 mm; in-plane resolution, 3  3 mm). A
custom-made bite bar was used to minimize head motion.
Functional MRI data preprocessing. All fMRI data underwent three-dimensional (3D) motion correction using automated image registration software40,
followed by linear trend removal. No spatial or temporal smoothing wasapplied. The fMRI data were aligned to retinotopic mapping data collected in a
separate session, using Brain Voyager software (Brain Innovation). Automatedalignment procedures were followed by careful visual inspection and manualfine-tuning at each stage of alignment to correct for misalignment error. Rigidbody transformations were done to align fMRI data to the within-session 3D
anatomical scan, and next to align these data to retinotopy data. After ouralignment procedure, any residual misalignment between fMRI scans collectedacross the two sessions appeared very small (less than 1 mm) and were likely ofcomparable order of magnitude to brain motion resulting from respiration,
heart rate and residual head movement while subjects were stabilized with abite bar. After across-session alignment, fMRI data underwent Talairachtransformation and reinterpolation using 3  3  3 mm voxels. Thistransformation allowed us to restrict voxels around the gray-white matterboundary and to delineate individual visual areas on flattened corticalrepresentations. However, these procedures involving motion correction andinterpolation of the raw fMRI data may have resulted in the reduction oforientation information that may be contained in fine-scale activity patterns.
Voxels used for orientation decoding analysis were selected on the corticalsurface of V1 through V4 and MT+. First, voxels near the gray-white matterboundary were identified within each visual area using retinotopic maps
delineated on a flattened cortical surface representation. Then, the voxels were
sorted according to the responses to the visual field localizer (V1–V4) or to theMT+ localizer. We used 200 voxels for each of areas V1–V4 (100 voxels when

VOLUME 8 [ NUMBER 5 [ MAY 2005 NATURE NEUROSCIENCE© 2005 Nature Publishing Group http://www.nature.com/natureneuroscience

ARTICLES
dorsal and ventral parts were separately analyzed) and 100 voxels for MT+ by
selecting the most activated voxels.
The data samples used for orientation decoding analysis were created byshifting the fMRI time series by 4 s to account for the hemodynamic delay, andthen averaging the MRI signal intensity of each voxel for each 16-s stimulus
trial. Response amplitudes of individual voxels were normalized relative to theaverage of the entire time course within each run (excluding the rest periods atthe beginning and the end) to minimize baseline differences across runs. The
resulting activity patterns were labeled according to their corresponding
stimulus orientation and served as input to the orientation decoder analysis.
Decoding analysis. The calculation performed by each linear ensemble
orientation detector with preferred orientation yk can be expressed by a linear
function of voxel inputs x ¼ (x1, x2,y, xd) (‘linear detector function’)gyk ðxÞ ¼

dX

wi xi + w0

i¼1

where wi is the weight of voxel i and w0 is the bias. To achieve this function foreach orientation, we first calculated linear discriminant functions for pairs oforientations using machine learning–pattern recognition algorithms. Linear
support vector machines15 (SVM) were used to obtain the results presented
here, although other algorithms, such as Fisher’s linear discriminant method(combined with principal component analysis) and Perceptrons, could be used
to yield similar results. The discriminant function, gyk yl(x) for the discrimination of orientations yk and yl , is expressed by a weighted sum of voxel inputs
plus bias, and satisfies
gyk yl ðxÞ 4 0 ðx is fMRI activity induced by orientation yk Þ
gyk yl ðxÞ o 0 ðx is fMRI activity induced by orientation yl Þ:Using a training data set, a linear SVM finds optimal weights and bias for thediscriminant function. After the normalization of the weight vectors, thepairwise discriminant functions comparing yk and the other orientations were
simply added to yield the linear detector functionX
gyk ðxÞ ¼
gyk ym ðxÞ:m¼
6 k

This linear detector function becomes larger than zero when the input x (in the
training data set) is an fMRI activity pattern induced by orientation yk. In our
method, orientation is treated as a categorical variable, and no similarity
among orientations is assumed.
To evaluate orientation decoding performance, we performed a version ofcross-validation by testing the fMRI samples in one run using a decoder trainedwith the samples from all other runs. This training-test set was repeated for all
runs (‘leave one run out’ cross-validation). We used this procedure to avoid
using the samples in the same run both for training and test, as they are notindependent because of the normalization of voxel intensity within each run.
Note: Supplementary information is available on the Nature Neuroscience website.

ACKNOWLEDGMENTS
We thank D. Remus and J. Kerlin for technical support, the Princeton Center
for Brain, Mind and Behavior for MRI support and J. Haxby, D. Heeger and
A. Seiffert for comments. This work was supported by grants from Japan
Society for the Promotion of Science and National Institute of Information
and Communications Technology to Y.K., and grants R01-EY14202 andP50-MH62196 from the US National Institutes of Health to F.T.
COMPETING INTERESTS STATEMENT
The authors declare that they have no competing financial interests.
Received 3 January; accepted 29 March 2005
Published online at http://www.nature.com/natureneuroscience/

1. Hubel, D.H. & Wiesel, T.N. Receptive fields, binocular interaction and functionalarchitecture in the cat’s visual cortex. J. Physiol. (Lond.) 160, 106–154 (1962).
2. Hubel, D.H. & Wiesel, T.N. Receptive fields and functional architecture of monkey striate
cortex. J. Physiol. (Lond.) 195, 215–243 (1968).
3. Blasdel, G.G. Orientation selectivity, preference, and continuity in monkey striate cortex.
J. Neurosci. 12, 3139–3161 (1992).

NATURE NEUROSCIENCE VOLUME 8 [ NUMBER 5 [ MAY 2005

4. Bartfeld, E. & Grinvald, A. Relationships between orientation-preference pinwheels,
cytochrome oxidase blobs, and ocular-dominance columns in primate striate cortex.
Proc. Natl. Acad. Sci. USA 89, 11905–11909 (1992).
5. Paradiso, M.A. A theory for the use of visual orientation information which exploits thecolumnar structure of striate cortex. Biol. Cybern. 58, 35–49 (1988).
6. Pouget, A., Dayan, P. & Zemel, R. Information processing with population codes.
Nat. Rev. Neurosci. 1, 125–132 (2000).
7. Haxby, J.V. et al. Distributed and overlapping representations of faces and objects inventral temporal cortex. Science 293, 2425–2430 (2001).
8. Cox, D.D. & Savoy, R.L. Functional magnetic resonance imaging (fMRI) ‘brain reading’:
detecting and classifying distributed patterns of fMRI activity in human visual cortex.
Neuroimage 19, 261–270 (2003).
9. Carlson, T.A., Schrater, P. & He, S. Patterns of activity in the categorical representationsof objects. J. Cogn. Neurosci. 15, 704–717 (2003).
10. Vanduffel, W., Tootell, R.B., Schoups, A.A. & Orban, G.A. The organization of orientation
selectivity throughout macaque visual cortex. Cereb. Cortex 12, 647–662 (2002).
11. Kim, D.S., Duong, T.Q. & Kim, S.G. High-resolution mapping of iso-orientation columns
by fMRI. Nat. Neurosci. 3, 164–169 (2000).
12. Engel, S.A., Glover, G.H. & Wandell, B.A. Retinotopic organization in human visualcortex and the spatial precision of functional MRI. Cereb. Cortex 7, 181–192 (1997).
13. Malonek, D. & Grinvald, A. Interactions between electrical activity and cortical microcirculation revealed by imaging spectroscopy: implications for functional brain mapping.
Science 272, 551–554 (1996).
14. Shtoyerman, E., Arieli, A., Slovin, H., Vanzetta, I. & Grinvald, A. Long-term opticalimaging and spectroscopy reveal mechanisms underlying the intrinsic signal andstability of cortical maps in V1 of behaving monkeys. J. Neurosci. 20, 8111–8121(2000).
15. Vapnik, V.N. Statistical Learning Theory (Wiley, New York, 1998).
16. Minsky, L.M. & Papert, S.A. Perceptrons – Expanded Edition: An Introduction to
Computational Geometry (MIT Press, Boston, 1987).
17. Furmanski, C.S. & Engel, S.A. An oblique effect in human primary visual cortex.
Nat. Neurosci. 3, 535–536 (2000).
18. Bauer, R. & Dow, B.M. Complementary global maps for orientation coding in upperand lower layers of the monkey’s foveal striate cortex. Exp. Brain Res. 76, 503–509(1989).
19. Schall, J.D., Perry, V.H. & Leventhal, A.G. Retinal ganglion cell dendritic fields in oldworld monkeys are oriented radially. Brain Res. 368, 18–23 (1986).
20. Treue, S. & Maunsell, J.H. Attentional modulation of visual motion processing in corticalareas MT and MST. Nature 382, 539–541 (1996).
21. Treue, S. & Martinez Trujillo, J.C. Feature-based attention influences motion processing
gain in macaque visual cortex. Nature 399, 575–579 (1999).
22. Treue, S. Visual attention: the where, what, how and why of saliency. Curr. Opin.
Neurobiol. 13, 428–432 (2003).
23. Roelfsema, P.R., Lamme, V.A. & Spekreijse, H. Object-based attention in the primaryvisual cortex of the macaque monkey. Nature 395, 376–381 (1998).
24. Watanabe, T. et al. Task-dependent influences of attention on the activation of humanprimary visual cortex. Proc. Natl. Acad. Sci. USA 95, 11489–11492 (1998).
25. Saenz, M., Buracas, G.T. & Boynton, G.M. Global effects of feature-based attention inhuman visual cortex. Nat. Neurosci. 5, 631–632 (2002).
26. Wilson, H.R., Levi, D., Maffei, L., Rovamo, J. & DeValois, R. in Visual Perception: The
Neurophysiological Foundations (eds. Spillman, L. & Werner, J.S.) 231–272 (Academic, San Diego, 1990).
27. Tootell, R.B. et al. Functional analysis of primary visual cortex (V1) in humans. Proc.
Natl. Acad. Sci. USA 95, 811–817 (1998).
28. Boynton, G.M. & Finney, E.M. Orientation-specific adaptation in human visual cortex.
J. Neurosci. 23, 8781–8787 (2003).
29. Kamitani, Y. & Shimojo, S. Manifestation of scotomas created by transcranial magnetic
stimulation of human visual cortex. Nat. Neurosci. 2, 767–771 (1999).
30. Rees, G., Kreiman, G. & Koch, C. Neural correlates of consciousness in humans.
Nat. Rev. Neurosci. 3, 261–270 (2002).
31. Tong, F. Primary visual cortex and visual awareness. Nat. Rev. Neurosci. 4, 219–229(2003).
32. Koch, C. The Quest for Consciousness: A Neurobiological Approach (Roberts, Englewood, Colorado, USA, 2004).
33. Donoghue, J.P. Connecting cortex to machines: recent advances in brain interfaces.
Nat. Neurosci. 5 (suppl.): 1085–1088 (2002).
34. Wolpaw, J.R. & McFarland, D.J. Control of a two-dimensional movement signal by anoninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. USA 101,
17849–17854 (2004).
35. Kahneman, D. & Wolman, R.E. Stroboscopic motion: Effects of duration and interval.
Percept. Psychophys. 8, 161–164 (1970).
36. Sereno, M.I. et al. Borders of multiple visual area in humans revealed by functional
magnetic resonance imaging. Science 268, 889–893 (1995).
37. Zeki, S. et al. A direct demonstration of functional specialization in human visual cortex.
J. Neurosci. 11, 641–649 (1991).
38. Watson, J.D. et al. Area V5 of the human brain: evidence from a combined study
using positron emission tomography and magnetic resonance imaging. Cereb. Cortex
3, 79–94 (1993).
39. Tootell, R.B. et al. Functional analysis of human MT and related visual cortical areasusing magnetic resonance imaging. J. Neurosci. 15, 3215–3230 (1995).
40. Woods, R.P., Grafton, S.T., Holmes, C.J., Cherry, S.R. & Mazziotta, J.C. Automatedimage registration: I. General methods and intrasubject, intramodality validation.
J. Comput. Assist. Tomogr. 22, 139–152 (1998).

685