Available online at www.sciencedirect.com

ScienceDirect
Eight open questions in the computational modelingof higher sensory cortex
Daniel LK Yamins and James J DiCarlo
Propelled by advances in biologically inspired computer vision
and artificial intelligence, the past five years have seen
significant progress in using deep neural networks to model
response patterns of neurons in visual cortex. In this paper, we
briefly review this progress and then discuss eight key ‘open
questions’ that we believe will drive research in computationalmodels of sensory systems over the next five years, both invisual cortex and beyond.
Address
Department of Brain and Cognitive Sciences and McGovern Institute forBrain Research, Massachusetts Institute of Technology, Cambridge, MA
02139, USA
Corresponding author: Yamins, Daniel LK (yamins@mit.edu)

Current Opinion in Neurobiology 2016, 37:114–120
This review comes from a themed issue on Neurobiology of behavior
Edited by Alla Karpova and Roozbeh Kiani

http://dx.doi.org/10.1016/j.conb.2016.02.001
0959-4388/# 2016 Elsevier Ltd. All rights reserved.

object category, robust to significant variations present innatural images [8–10]. The featural content of mid-levelvisual areas such as V2, V3, and V4 is less well understood,
but these areas appear to contain intermediate computations between simple edges and complex objects, along apipeline of increasing receptive field sizes [1,11–18].
Many of these observations can be captured mathematically via class of computational architectures known as
Hierarchical Convolutional Neural Networks (HCNNs),a generalization of Hubel and Wiesel’s simple and complex cells that has been developed over the past 30 years[19,20]. HCNN models are composed of several retinotopic layers combined in series. Each layer is very
simple, but together they produce a deep, complex transformation of the input data—in theory, like the transformation produced in the ventral stream. However,mapping a single HCNN model to ventral stream neuraldata has proven extremely challenging [12], in part because subtle parameter changes (e.g. number of layers,
local receptive field sizes, &c) can dramatically affect amodel’s match to neural data [21,22]. Recent work invisual cortex seeks to go beyond this powerful but broadstroke understanding to identify concrete predictivemodels of ventral cortex, and then use these models togain insight inaccessible without large-scale computational precision.

Any scientific development of long-term value opens up
as many new questions as it answers. This is certainly thecase with recent progress in building deep neural networkmodels of visual cortex. In this piece, our goal is to briefly
describe these recent advances, and to outline what we
consider to be the most interesting open problems in
cortical modeling, both in vision and beyond. We focus ison questions that will require both cutting-edge algorithmic developments as well as next-generation neuroscience and cognitive science experiments.

A key aspect of this approach has been performance-based
optimization, in which the parameters of a large multilayer neural networks are chosen to optimize the networks’ performance on a high-level, ecologically valid
visual task [23]. Leveraging computer vision and machine learning techniques, together with large amounts ofreal-world labelled images used as supervised training
data [24,25,26], HCNNs have been produced that
achieve near-human-level performance on challenging
object categorization tasks [27].

Brief review of recent progress

Intriguingly, even though these networks are not directly
optimized to fit neural data, their top hidden layers arenonetheless highly predictive of single-site neural
responses as well population-level representations in IT
cortex both in electrophysiological [23,28], and fMRI data[29,30]. Specifically, model units from the highest hidden
layers of these performance-optimized HCNN can belinearly combined to produce synthetic ‘neurons’ thatpredict the image-by-image response patterns of sites in
IT cortex. Moreover, the population of these synthetic
neurons closely matches the representational dissimilarity

Starting with the seminal ideas of Hubel and Wiesel, workin visual systems neuroscience over the past 60 years has
shown that the ventral visual stream generates invariant
object recognition behavior via a hierarchically-organizedseries of cortical areas that encode object properties withincreasing selectivity and tolerance [1,2–5]. Early visualareas, such as V1 cortex, capture low-level features such asedges and center-surround patterns [6,7]. In contrast,
neural population responses in the highest ventral visualareas, inferior temporal (IT) cortex, can be used to decode
Current Opinion in Neurobiology 2016, 37:114–120

www.sciencedirect.com

Open questions in the modeling of higher sensory cortex Yamins and DiCarlo 115

matrices (RDMs, [31]) of the macaque and human IT
populations. These deep, performance-optimized neuralnetworks have thus yielded the first quantitatively accurate, predictive model of the IT population response.
Moreover, high-throughput computational experiments
evaluating thousands of HCNN models on both task
performance and neural-predictivity metrics, have founda strong correlation between performance of high-levelobject recognition tasks and ability to explain IT corticalspiking data [23]. The predictive power of these models
is driven not just by categorization performance alone, as
ideal observer models with perfect access to object identity do not themselves predict IT neural response patterns nearly as well as the hierarchical neural network
units [23].
Critically, these HCNN models are mappable not only toIT, but also to other levels of the ventral visual stream.
Lower model layer filter weights resemble Gabor wavelets and are effective models of fMRI voxel responses inV1 voxel data [29,30]. Along the same lines, intermediate
HCNN layers are predictive of neural responses in V4
cortex [23]. In other words: combining two general
biological constraints—the behavioral constraint of recognition performance, and the architectural constraint
imposed by the HCNN model class—leads to greatly
improved models of multiple layers of the visual sensory
cascade. An additional benefit of this approach is that eachlayer of the HCNN is a basis set for its corresponding
cortical area, from which large numbers of IT-, V4- or V1like units can be generated. A common assumption invisual neuroscience is that understanding the qualitative
structure of tuning curves in lower cortical areas (e.g.
gabor conjunctions in V2 or curvature in V4 [32]) is anecessary precursor to explaining higher visual cortex.
These recent results show that higher-level constraints
can yield quantitative models even when bottom-up
primitives have not yet been identified.
The mapping between neural networks and cortical neural responses is still far from perfect. However, theserecent results are encouraging, and they advance theunderstanding of the ventral stream in at least two new
ways. First, the predictive accuracy of these models
suggests that the principles of cortical processing may
be best described at the level of architectural statistics(rather than precise wiring patterns), learning rules (rather
than descriptors of tuning curves), and ethological task
goals (rather than information transmission). Second,
because models derived from this approach are bothaccurately predictive and generative, they act as hypothesis generators that can be richly interrogated to explore
key open questions and enable the rational design ofneuroscience experiments to answer those questions.
Below we list eight exciting open questions that are
now approachable from this new vantage point.
www.sciencedirect.com

Why is IT cortex heterogenous at large spatial
scales?
IT cortex is not a single monolithic computational massin which output features are randomly intermixed acrossthe cortical surface, but instead is likely to containmultiple retinotopic areas, with posterior IT, central
IT, and anterior IT areas performing potentially different computations [1]. It is also now known that specialized face, place, body, and color-preferring regions at
the multiple-millimeter scale are found in each of theseIT areas [33–36]. Are these the only regions? If so, why
these and not others? How do the regions arise in thefirst place? Understanding this heterogeneity withcomputational models has two components: first, identifying whether and how the observed distributions ofunit selectivities arise, independently of their spatial
clustering; and second, explaining the observed spatial
clustering.
Existing HCNN models could likely be used to generate detailed predictions about the unit distributions. A
basic question is: to what extent are the existence ofapparently specialized populations of units (e.g. faceselective units) strongly dependent on the semantic
content of the training data of the neural networks?
Will standard neural network model approaches yieldobserved unit populations if trained on datasets with amix of semantic content close to that experienced byhumans during development (e.g. a large fraction of
faces)? How sensitive are unit selectivity distributions
to this semantic content?
The second question, about spatial clustering, will require a more substantial extension of the HCNN framework, since those models make no specific predictions
about how their units are to be mapped to the twodimensional cortical sheet. It is possible that using asimple self-organizing map approach [37] to cluster inspace units with similar feature tunings would explain alarge fraction of the spatial structure in IT. However,there is some evidence that clustering may not be along
purely geometric or featural lines—for example, bodypreferring patches arise near face-preferring patches eventhough there is no obvious geometric similarity between
these two categories [38]. If the known regions do not
emerge in these types of models, it will be important tounderstand what additional principles are required tobuild them. If they do, it will also be of interest to search
for new model-predicted regions that could subsequently
be confirmed or falsified using primate fMRI and electrophysiolgy experiments.

Which visual properties are explicitly encoded
in intermediate ventral stream areas?
Intermediate visual areas such as V2 and V4 have
proven especially hard to understand because, unlike
V1 and IT, they are removed both from low-level image
Current Opinion in Neurobiology 2016, 37:114–120

116 Neurobiology of behavior

properties and from higher-level human semantic intuition [32,39,40]. Because intermediate layers of computational models are predictive of these cortical areas,
performing high-throughput ‘virtual electrophysiology
to characterize the model’s internal structure should
yield insight into tuning curves in corresponding corticalareas. One of the key advantages of an image-computable model is that it can be analyzed in great detail at
low cost. It will be of great interest to perform highthroughput ‘virtual electrophysiology’ on models todetermine what their intermediate units represent. Recent progress in visualization techniques have allowed
for structures in intermediate layers to be explored.
Techniques that seek to optimize input images either
for matching the statistics of existing images or for theactivation of a unit (or units) within model higher layers
have been seen to produce impressive results in texture
generation, image style matching, and model exploration [41,42,43]. Versions of these techniques could
have useful application in neuroscience for efficiently
separating multiple candidate models, by optimizing forstimuli that produce the largest difference between two
such models, even if, and perhaps most especially if,
those alternative models do not differ in simple conceptual ideas. Such techniques (inspired by [9,44]) could be
extremely helpful in reducing the huge stimulus space
to a set that would be small enough to measure neural
responses using existing experimental techniques.

Can models predict the perceptual
consequences of direct neural perturbations?
Computational models enable the efficient exploration of
the effects of perturbing unit activations in a highly
selective manner. In the short term, such an approach
could be used to make testable predictions of the behavioral changes (e.g. in facial expression identification ability) that arise from inactivating/stimulating specificsubpopulations of units defined by cell type or functionality (e.g. high face-selectivity). Over the next 2–5 years,
it should be possible to combine computational models
with cutting-edge optical techniques in non-human primates [45,46] to help design highly targeted real-timeneural perturbation studies.

purposes [8,47,48]. Evaluating this hypothesis more thoroughly is important, involving a variety of questions,
including: To what extent is the hypothesis true? Are downstream
visual readouts really linear as [8] might suggest, or forsome tasks do they need to be (at least somewhat)
nonlinear? How important are temporal coding mechanisms in visual task readout?
 Building on [47], can we identify one or several
examples of these linear readouts in action, connecting
IT to some specific downstream area? What types
neurons, in which cortical layers, are involved in thereadout process?
 How are linear readouts learned, and over whattimescales do they form? For them to play the role
they are often assigned in the current models, suchreadouts would have to form comparatively quickly,
with small amounts of on-line training data. Can theprocess of fast learning be captured empirically? What
specific rules of learning are used? Are the classifier
weight learning rules implicated more like those using
in machine learning classifier algorithms, such as
Support Vector Machines (SVMs), or some simpler
procedure like linear discriminants [49]? If there reallyare linear classifier learning algorithms running neurally, do their weightings on input neurons get regularized
as they might in machine learning algorithms [50]—that is, discouraged from having strong connections
from too many upstream neurons, beyond theconstraints imposed by biophysics alone? If so, what
type of sparsity priors are imposed by the regularization
used in the real brain?
 What are the default readouts that run automatically in a
typically inattentive or passive state? How is taskswitching accomplished, and how does this relate toattentional state? Presumably the task attention of theanimal is deeply connected to switching between the‘active’ readout at any given time. How is thisaccomplished mechanistically? How would we even
identify the neural signature of ‘active’ readout?

Is ‘linear readout’ a real model of downstream
neural decoding?

How do feedforward discriminative networks
connect to top-down inference in generativemodels?

In the above work, it is often implicitly assumed thatbehavior is generated from neural representations bylinear readout, for example, neurons in other downstream
brain areas (e.g. PFC, motor cortex, parietal cortex) using
the information in the high-level representational areas(e.g. IT) by forming linear combinations of neurons fromthose areas. In other words, linear classifiers are notmerely thought of as ‘information measurement’ devices,
but also as a concrete hypothesis for how multiple downstream brain areas could utilize the robust, explicit information in a high-level representation for multiple task

The HCNN models of vision mentioned above are largely discriminative, in that they model the flow of information from bottom-to-top: information comes in to thesystem along sensory input arrays (e.g. the retina) andexits via outputs that indicate an attribute of the data
relative to an (often discrete) discrimination task (e.g.
recognizing a given category of object in the image).
However, the generative point of view is also relevant
and a potentially critical next step in sensory models.
In this view, there is an explicit process through which
high-level labels about the world (e.g. object identities,

Current Opinion in Neurobiology 2016, 37:114–120

www.sciencedirect.com

Open questions in the modeling of higher sensory cortex Yamins and DiCarlo 117

relation locations and relationships) can be turned into
predictions about the low-level (pixel) description consistent with those high-level labels. Generative models
are important because they naturally support a number ofkey cognitive phenomena, such as inference, explaining
away, and imagination—and would be a natural source of‘data’ for self-supervised learning procedures. They alsooffer the potential of explaining the well-known, but
poorly understood role of cortico-cortical feedback connections in the brain. Figuring out how to combine
bottom-up discriminative models with more top-down
generative models, and connect them both to the neuroscience of visual perception in ambiguous or complex
images (like the ‘Dallenbach Cow’ [51]) would be asignificant step forward.

How is visual learning actually implemented in
the brain?
While recent work has begun to uncover how images are
encoded in adult IT cortex, very little is known about how
the IT representation arises in the first place. To what
extent does visual learning during development shapehigh-level vision? Computational models—which, atheart, are really the product of learning rules in action—can help us think about these key questions in a
new way.
Three important questions associated with this are:
 What aspects of the visual system are evolved, vs.
developed, vs. learned? How do the learning rules
active during post-natal development differ from those
active in expertise learning in adulthood? Mapping
learning trajectories in models to developmental data
will help make predictions about these importantquestions.
 What types of data are learning rules receiving? A
significant limitation of many of the most effective
existing learning rules for computational models—such as error backpropagation for training categorization in deep neural networks—is that they requirelarge amounts of high-level semantically labeledtraining data. Over the next few years, an importantarea of research will be the discovery of semi- or unsupervised neural network learning algorithms thatblend features of existing machine-learning techniques with constraints from neural data. An intriguing
possibility is that heavily supervised category-label
training could be replaced by optimization forproperties (e.g. position, size, and pose) that can be
more easily estimated from motion heuristics innatural video.
 How are learning rules implemented at the circuit
level? Existing results suggest a number of costfunction objectives that may be involved in perceptual
learning, including stimulus reconstruction goals,
www.sciencedirect.com

efficient-coding constraints [52], and top-down supervised task prediction errors [53]. These rule need to be
implemented as a (presumably recurrent) part of theneural network that can be then used for tasks during‘runtime’. By combining high-resolution neural techniques such as calcium imaging and axonal tracing, it
may be possible to watch learning in action and use
these data to constrain an understand of how thealgorithmic rules underlying this learning are implemented in neural circuits.

Can sensory systems beyond vision benefit
from performance-driven neural-network
approaches?
Recent work in vision suggests a more general hypothesis
about how to model sensory cortex: selecting biologicallyplausible neural networks for high performance on an
ecologically-relevant sensory task will yield a detailed
model of the actual cortical areas that underlie that task.
Since this idea has some traction in the ventral visualstream, it is natural to ask whether it also yields insight inother sensory domains.
Some initial recent work has found that HCNNs trained
to solve challenging high-variation word recognition tasks
are predictive of voxel patterns in auditory cortex [54].
These models are also able to differentiate auditory areas,
with lower model layers more predictive of inferior colliculus, intermediate layers more predictive of primary
auditory cortex, and higher layers of speech and musicselective areas identified in recent imaging studies [55].
These results open up a variety of computational audition
questions, including: Detailed characterization of non-primary auditory
cortex. As with higher visual areas, models can beused to make detailed testable predictions about
poorly-understood auditory cortex subregions, especially in relation to speech and natural soundrepresentations [55,56]. This problem could beapproached experimentally using both human fMRI/
eCOG and non-human primate electrophysiology
techniques.
 What auditory tasks best explain cortical differentiation? Given the evolutionary history of audition, nonspeech tasks (e.g. environmental sound differentiation)
might be as important for driving auditory cortex
structure as speech. This question could be exploredby training networks on a variety of ecological auditory
tasks.
 How do audition-optimized architectures compare to
those optimized for vision? Are there deep but hidden
structural similarities between visual and auditory
cortex that arise from underlying similarities inauditory and visual data? This fascinating question
could be attacked both from a purely algorithmic point
Current Opinion in Neurobiology 2016, 37:114–120

118 Neurobiology of behaviorof view, as well as by comparing auditory neural data toventral-stream data.
It will also be of interest to see if similar ideas can used inother sensory domains. One promising idea would be tomodel neural responses in rat barrel cortex through hierarchical networks optimized for somatosensory tasks. It isalso an intriguing question whether the performancedriven modeling approach will have utility for understanding evolutionarily older sensory systems, such as
olfaction, where key challenges will include framing
behavioral tasks and architecture classes in the first place.

How should working memory be included?
Beyond Sense-At-A-Glance
Largely feedforward models like HCNNs cannot provide afull account of the dynamics of brain systems that store
extensible state – most notably, any brain area that involves
working memory, since the ‘dynamics’ of a feedforward
network will always converge to the same state independent of input. However, in studying behavior downstreamof the sensory system, such as that arising from visual
working memory, having a computational stand-in for
the visual system to use as input to models of memoryareas would be of great use. Moreover, there is a growing
body of literature connecting recurrent neural networks toneural phenomenon in attention, decision making andmotor program generation [57,58,59]. Results from reinforcement learning [60] have shown how powerful suchtechniques may be in the realm of AI. Mapping these toideas in the neuroscience of the interface between ventral
visual cortex and (e.g.) parietal cortex or the hippocampus
will be of great interest [61,62]. Models that combine richsensory input systems, as modeled by deep neural networks, with these recurrent networks, could provide afruitful avenue for exploring more sophisticated cognitive
behaviors beyond simple categorization or binary decision
making, breaking out of the pure ‘representation’ framework in which sensory models are often cast. This isespecially interesting when there could be a complex loop
between behavioral output and input stimulus, for example, when modeling exploration of an agent over long timescales in a complex sensory environment [63].

Conclusion
Recent computational advances have transformed ourability to accurately model the neural responses of sensory systems, even at high levels of the sensory hierarchy
that were previously deeply mysterious. However, there
is much more exciting work to be done, and, if the recent
past is a guide to the future, the next successes will
require the continued convergence of techniques andworkers in neuroscience, machine learning, computerscience, and psychophysics.

Conflict of interest statement
None declared.
Current Opinion in Neurobiology 2016, 37:114–120

References and recommended reading
Papers of particular interest, published within the period of review,
have been highlighted as:
 of special interest
 of outstanding interest
1. DiCarlo JJ, Zoccolan D, Rust NC: How does the brain solvevisual object recognition? Neuron 2012, 73:415-434.
A summary of core ideas about object recognition in the ventral stream.
2.

Malach R, Levy I, Hasson U: The topography of high-orderhuman object areas. Trends Cogn Sci 2002, 6:176-184.

3.

Felleman D, Van Essen D: Distributed hierarchical processing in
the primate cerebral cortex. Cereb Cortex 1991, 1:1-47.

4.

Rust NC, DiCarlo JJ: Selectivity and tolerance (‘‘invariance’’)both increase as visual information propagates from corticalarea v4 to it. J Neurosci 2010, 30:12978-12995.

5.

Connor CE, Brincat SL, Pasupathy A: Transformation of shapeinformation in the ventral pathway. Curr Opin Neurobiol 2007,
17:140-147.

6. Carandini M et al.: Do we know what the early visual system
does? J Neurosci 2005, 25:10577-10597.
A summary of core ideas about understanding early visual areas.
7.

Movshon JA, Thompson ID, Tolhurst DJ: Spatial summation in
the receptive fields of simple cells in the cat’s striate cortex. J
Physiol 1978, 283:53-77.

8.

Majaj NJ, Hong H, Solomon EA, DiCarlo JJ: Simple learned
weighted sums of inferior temporal neuronal firing rates
accurately predict human core object recognition
performance. J Neurosci 2015, 35:13402-13418.

9.

Yamane Y, Carlson ET, Bowman KC, Wang Z, Connor CE: A
neural code for three-dimensional object shape in macaqueinferotemporal cortex. Nat Neurosci 2008.

10. Hung CP, Kreiman G, Poggio T, Dicarlo JJ: Fast readout of object
identity from macaque inferior temporal cortex. Science 2005,
310:863-866.
11. Freeman J, Simoncelli E: Metamers of the ventral stream. Nat
Neurosci 2011, 14:1195-1201.
12. DiCarlo JJ, Cox DD: Untangling invariant object recognition.
Trends Cogn Sci 2007, 11:333-341.
13. Schmolesky MT et al.: Signal timing across the macaque visualsystem. J Neurophysiol 1998, 79:3272-3278.
14. Lennie P, Movshon JA: Coding of color and form in the
geniculostriate visual pathway (invited review). J Opt Soc Am A
Opt Image Sci Vis 2005, 22:2013-2033.
15. Schiller P: Effect of lesion in visual cortical area v4 on therecognition of transformed objects. Nature 1995, 376:342-344.
16. Gallant J, Connor C, Rakshit S, Lewis J, Van Essen D: Neural
responses to polar, hyperbolic, and Cartesian gratings inarea v4 of the macaque monkey. J Neurophysiol 1996,
76:2718-2739.
17. Brincat SL, Connor CE: Underlying principles of visual shapeselectivity in posterior inferotemporal cortex. Nat Neurosci
2004, 7:880-886.
18. Yau JM, Pasupathy A, Brincat SL, Connor CE: Curvatureprocessing dynamics in macaque area v4. Cereb Cortex 2012.
bhs004.
19. Fukushima K: Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected byshift in position. Biol Cybern 1980.
The original paper introducing convolutional neural networks, with biological inspiration.
20. LeCun Y, Bengio Y: Convolutional networks for images, speech,and time series. The handbook of brain theory and neural networks.
1995:255-258.
A good summary of the first wave of serious application of HCNNs in
artificial intelligence and signal processing problems.
www.sciencedirect.com

Open questions in the modeling of higher sensory cortex Yamins and DiCarlo 119

21. Pinto N, Cox DD, Dicarlo JJ: Why is real-world visual object
recognition hard? PLoS Comput Biol 2008.
22. Yamins D, Hong H, Cadieu C, Dicarlo J: Hierarchical modular
optimization of convolutional networks achieves
representations similar to macaque it and human ventral
stream. Advances in Neural Information Processing Systems.
2013.
23. Yamins DL et al.: Performance-optimized hierarchical models
predict neural responses in higher visual cortex. Proc Natl Acad
Sci U S A 2014, 111:8619-8624.
A recent paper illustrating how deep neural networks, optimized for visual
recognition tasks, provide effective models of neural responses in intermediate and higher macaque visual cortex.
24. Krizhevsky A, Sutskever I, Hinton G: ImageNet classification
 with deep convolutional neural networks. Advances in Neural
Information Processing Systems. 2012.
A seminal paper in the recent Deep Learning revolution, showing the useof large datasets and powerful GPUs in training HCNNs.
25. Bergstra J, Yamins D, Cox D: Making a science of model search:
hyperparameter optimization in hundreds of dimensions forvision architectures. In Proceedings of the 30th International
Conference on Machine Learning. 2013:115-123.
26. Deng J, Li K, Do M, Su H, Fei-Fei L: Construction and analysis ofa large scale image ontology. Vision Sciences Society. 2009.
The paper introducing ImageNet, the core dataset used for much ofrecent supervised training of category tasks in deep neural networks.
27. Simonyan K, Zisserman A: Very deep convolutional networks forlarge-scale image recognition. 2014arXiv:1409.1556.
28. Cadieu CF et al.: Deep neural networks rival the representationof primate it cortex for core visual object recognition. PLOS
Comput Biol 2014, 10:e1003963.
29. Khaligh-Razavi SM, Kriegeskorte N: Deep supervised, but notunsupervised, models may explain it cortical representation.
PLOS Comput Biol 2014.
A paper illustrating how deep neural networks explain human visualcortex, including the similarities between lower layers of the HCNNs
and early visual cortex.
30. Güçlü U, van Gerven MA: Deep neural networks reveal a
gradient in the complexity of neural representations acrossthe ventral stream. J Neurosci 2015, 35:10005-10014.
31. Kriegeskorte N et al.: Matching categorical object
representations in inferior temporal cortex of man andmonkey. Neuron 2008, 60:1126-1141.

40. Nishimoto S, Gallant JL: A three-dimensional spatiotemporal
receptive field model explains responses of area MT neurons
to naturalistic movies. J Neurosci 2011, 31:14551-14564.
41. Gatys LA, Ecker AS, Bethge M: A neural algorithm of artistic style.

2015arXiv:1508.06576.
A paper showing how the concept of ‘artistic style’ can be capture andmixed the concept of with ‘image content’ using HCNNs.
42. Gatys LA, Ecker AS, Bethge M: Texture synthesis and thecontrolled generation of natural stimuli using convolutional
neural networks. Advances in Neural Information Processing
Systems. . 2015arXiv:1505.07376.
43. Mordvintsev A, Tyka M, Olah C: Inceptionism: Going deeper into
neural networks, google research blog. 2015.
44. Paninski L, Pillow J, Lewi J: Statistical models for neural
encoding, decoding, and optimal stimulus design. Prog Brain
Res 2007, 165:493-507.
45. Afraz SR, Kiani R, Esteky H: Microstimulation of inferotemporal
cortex influences face categorization. Nature 2006, 442:692-695.
46. Afraz A, Boyden ES, DiCarlo JJ: Optogenetic andpharmacological suppression of spatial clusters of faceneurons reveal their causal role in face gender discrimination.
Proc Natl Acad Sci U S A 2015, 112:6730-6735.
47. Freedman DJ, Riesenhuber M, Poggio T, Miller EK: Categoricalrepresentation of visual stimuli in the primate prefrontal
cortex. Science 2001, 291:312-316.
48. Pagan M, Urban LS, Wohl MP, Rust NC: Signals in
inferotemporal and perirhinal cortex suggest an untangling ofvisual target information. Nat Neurosci 2013, 16:1132-1139.
49. Misaki M, Kim Y, Bandettini PA, Kriegeskorte N: Comparison ofmultivariate classifiers and response normalizations forpattern-information FMRI. Neuroimage 2010, 53:103-118.
50. Evgeniou T, Pontil M, Poggio T: Regularization networks andsupport vector machines. Adv Comput Math 2000, 13:1-50.
51. Dallenbach KM: A puzzle-picture with a new principle ofconcealment. Am J Psychol 1951.
52. Barlow HB: Possible principles underlying the transformations of
sensory messages. 1961.
A classic in the psychophysical study of visual property representations,especially in terms of thinking about how multiple complementary properties (identity and position, for instance) can be represented together.

32. Sharpee TO, Kouh M, Reyholds JH: Trade-off between curvaturetuning and position invariance in visual area v4. Proc Natl Acad
Sci U S A 2012, 110:11618-11623.

53. Bottou L: Large-scale machine learning with stochastic

gradient descent. In Proceedings of COMPSTAT’2010. Springer;
2010:: 177-186.
This paper describes stochastic gradient descent, a core recent technique for efficiently training deep neural networks.

33. Downing PE, Chan A, Peelen M, Dodds C, Kanwisher N:
Domain specificity in visual cortex. Cereb Cortex 2006,
16:1453-1461.

54. Kell A, Yamins D, Norman-Haignere S, McDermott J: Functionalorganization of auditory cortex revealed by neural networks
optimized for auditory tasks. Society for Neuroscience. 2015.

34. Freiwald WA, Tsao DY: Functional compartmentalization andviewpoint generalization within the macaque face-processing
system. Science 2010, 330:845-851.

55. Norman-Haignere S, Kanwisher N, McDermott JH: Cortical pitch
regions in humans respond primarily to resolved harmonics
and are located in specific tonotopic regions of anterior
auditory cortex. J Neurosci 2013, 33:19451-19469.

35. Vaziri S, Carlson ET, Wang Z, Connor CE: A channel for 3d
environmental shape in anterior inferotemporal cortex. Neuron
2014, 84:55-62 http://www.sciencedirect.com/science/article/pii/
S0896627314007442.
36. Lafer-Sousa R, Conway BR: Parallel, multi-stage processing ofcolors, faces and shapes in macaque inferior temporal cortex.
Nat Neurosci 2013, 16:1870-1878 http://dx.doi.org/10.1038/
nn.3555.
37. Kohonen T: The self-organizing map. Neurocomputing 1998,
21:1-6.
38. Schwarzlose RF, Baker CI, Kanwisher N: Separate face andbody selectivity on the fusiform gyrus. J Neurosci 2005,
25:11055-11059.
39. Naselaris T, Kay KN, Nishimoto S, Gallant JL: Encoding anddecoding in FMRI. Neuroimage 2011, 56:400-410.
www.sciencedirect.com

56. Leaver AM, Rauschecker JP: Cortical representation of naturalcomplex sounds: effects of acoustic features and auditory
object category. J Neurosci 2010, 30:7604-7612.
57. Chikkerur S, Serre T, Tan C, Poggio T: What and where: aBayesian inference theory of attention. Vision Res 2010,
50:2233-2247.
58. Mante V, Sussillo D, Shenoy KV, Newsome WT: Contextdependent computation by recurrent dynamics in prefrontal
cortex. Nature 2013, 503:78-84.
A paper showing how simple recurrent neural networks (RNNs) optimized
to solve a context-dependent motor task can explain neural response
properties in prefrontal cortex.
59. Sussillo D, Churchland MM, Kaufman MT, Shenoy KV: A neuralnetwork that finds a naturalistic solution for the production ofmuscle activity. Nat Neurosci 2015, 18:1025-1033.
Current Opinion in Neurobiology 2016, 37:114–120

120 Neurobiology of behavior

60. Mnih V et al.: Human-level control through deep reinforcementlearning. Nature 2015, 518:529-533.
This paper shows how reinforcement learning algorithms can be applied
to train HCNNs to play simple video games.
61. Harvey CD, Coen P, Tank DW: Choice-specific sequences inparietal cortex during a virtual-navigation decision task.
Nature 2012, 484:62-68.

Current Opinion in Neurobiology 2016, 37:114–120

62. Hulbert J, Norman K: Neural differentiation tracks improved
recall of competing memories following interleaved study andretrieval practice. Cereb Cortex 2014:bhu284.
63. Stadie BC, Levine S, Abbeel P: Incentivizing exploration in
reinforcement learning with deep predictive models.
2015arXiv:1507.00814.

www.sciencedirect.com