Mind the gap: comparing multiple models ofscene representation in brain and behavior

Iris Groen, PhD

New York University

Deep nets and object recognition

Krizhevsky,
Sutskever& Hinton, 2012

Yamins et al, 2014

Khaligh-Razavi& Kriegeskorte, 2014

beach hutsand
beach chair“beach”

Does deep learning explain
scene perception?

Outline• Scene vs. object perception• fMRI study 1: objects-in-context• fMRI study 2: comparing multiple models• How to move forward?

Scene vs. object perception
A scene is a semantically coherent (and often namable) view of areal-world environment comprising background elements andmultiple discrete objects arranged in a spatially licensed manner.

Henderson and Hollingworth (1999)

Scene perception can be usefully contrasted to object perception:
whereas objects are spatially compact entities that one acts upon,
scenes are spatially distributed entities that one acts within.

Epstein (2005)

Scene recognition without objects
Object information degraded

Object information absent

Oliva, 2005

Scene-selective brain regions
scenesobjects>
lateralmedialM e d ia l P la c e A r e a ( M P A )anterior
anteriorposterior

P a r a h ip p o c a m p a l P la c e A r e a ( P P A )O c c ip ita l P la c e A r e a ( O P A )See e.g. Epstein (2014) for a review

Understanding scene perception
Do DNNs predict humanbehavioral scene perception?

Scene
perception
behavior

Computationalmodels ofscene content
Do DNNs predict humanbrain responses to scenes?

Neuralresponses toscenes

OPA

Outline• Scene vs. object perception• fMRI study 1: objects-in-context• fMRI study 2: comparing multiple models• How to move forward?

DNNs and object recognition

Yamins et al, 2014

Khaligh-Razavi& Kriegeskorte, 2014

Study 1: objects-in-context + scenes
48 naturalistic image categories(3 exemplars per category, 2 sets)
accessories
adultsairplanes
appliances
bags
bathrooms
beaches
beds
bikes
birds
boats
body parts
bugs
butterflies
cars
churches
cityscapes
clothesdeserts
dishesdolls
factories
farm animals
fireflowers
foodforests
gyms
houseskids
kitchens
living rooms
lizards/snakes
masks
motorcycles
mountainsolder adults
pets
pools
roads
signs
spiders
sportssuburbs
tools
toys
trainswild animals

Free arrangement behavioral task
“Group these images according to their similarity”

Mur et al., 2013

Free arrangement behavioral task
All pairwisecomparisonsBehavioral dissimilarity
OutdoorscenesTransport
Man-made
HumansNaturalIndoor
scenes/Householditems
AnimalsNaturalobjects& scenes

King &, Groen &, Steel, Kravitz & Baker (in revision)

AlexNet
layer 8DNN?

Ventral temporal cortex?

Correlation distance

Euclidean distance

Behavior

Scene-selective cortex?
PPAOPAKing &, Groen &, Steel, Kravitz & Baker (in revision)

Multi-voxel pattern analysis

1-correlation

AlexNet layer 8DNNPPAOPA

Correlation distance

Scene-selective cortex

Correlation distance

Ventral temporal cortex

Correlation distance

Euclidean distance

Behavior

King &, Groen &, Steel, Kravitz & Baker (in revision)Behavior - DNNvTC - DNN

PPA - DNN

King &, Groen &, Steel, Kravitz & Baker (in revision)Understanding scene perception
Do DNNs predict humanbehavioral scene perception?
Yes - layer 8 best

Computationalmodels ofscene content
Do DNNs predict humanbrain responses to scenes?
Yes - layer 5 best

Sceneperception
behavior

OPAMismatch!

Neuralresponses toscenes

Why the mismatch?
• Nature of stimuli (mix of objects-in-context and scenes)?
• Multi-arrangement task?
• Alternative models for behavior?

Large-scale scene perception• Mechanical Turk
• SUN database• > 300 scene categories
SAME or DIFFERENT CATEGORY?

> 2000 participants> 5 million trials
Greene et al., 2016 (JEP:General)Comparing behavior to multiple models
Correlation with scene categorization

Greene et al., 2016 (JEP:General)Three best models of scene perceptionScenes

Convolutional Neural Network(AlexNet; 7th layer)

Actionlabels

Greene et al., 2016 (JEP:General)

Study 2• Goal: determine how well the top three models predict fMRI
responses and behavioral multi-arrangement task!! Inherent correlations make it difficult to disambiguate
models, even within large sets of naturalistic images(Lescroart, Stansbury & Gallant, 2015; Malcolm, Groen & Baker, 2016)

à Minimize model correlations through stimulus selection:
Iterative sampling from SUN database until we obtainmaximally different predictions
à Use variance partitioning to identify unique contribution ofeach modelStimuli forby fMRI
function
access_road

airplane_cabin

apse

badminton_court

bamboo_forestbar

batting_cagebindery

bus_depot

butte

control_tower

dolmenescalator

hedgerow

lido_deck

naval_basepierpilothouseplayroom

pump_room

putting_green

skyscraper

stadium

stilt_house

tea_garden

underwater_pool

volcanovolleyball_court

woodlandyouth_hostel

Stimuli by DNN
bindery

pilothousevolleyball_court

playroom

airplane_cabin

buttedolmenbar

youth_hostel

volcanolido_deckapse

pump_room

badminton_court

control_tower

skyscraper

stadium

bus_depot

tea_garden
woodlandputting_green
bamboo_forest

hedgerowaccess_road

batting_cage

naval_base

stilt_house

underwater_pool

pierescalator

Stimuli by Objects
airplane_cabinbarlido_deck
pilothouse

control_towerpier

youth_hostel

bindery

volcanounderwater_pool

volleyball_court

stilt_house

pump_room

apse

access_road
tea_garden
dolmenescalator

hedgerow
skyscraper

butte
woodlandbatting_cage

bus_depot

bamboo_forest

badminton_court

playroom

stadium

putting_green

naval_base

Model predictions
O b je c t m o d e lD N N m odel

F u n c tio n m o d e ldistance

category 1… 30max

mincategory 1…30

fM R IPPA

B e h a v io r

O PA

Model predictions
O b je c t m o d e l

D N N m odel

F u n c tio n m o d e ldistance

category 1… 30max

mincategory 1…30

PPA

fM R IOPA

B e h a v io rModel correlations with behavior
Correlations

Variance partitioning

Groen, Greene, Baldassano, Fei-Fei, Beck & Baker, 2018, eLife

Model correlations with brain
Correlations

Variance
partitioning

Groen, Greene, Baldassano, Fei-Fei, Beck & Baker, 2018, eLife

DNN correlations in scene cortex
Object-trained(ReferenceNet)

Scene-trained(Places 205)

Groen, Greene, Baldassano, Fei-Fei, Beck & Baker, 2018, eLife

DNN correlations in scene cortex

Groen, Greene, Baldassano, Fei-Fei, Beck & Baker, 2018, eLife

Where are the functions?
Unique correlations with function model outside scene cortex

OPA

Object-trained(ReferenceNet)

OPA
LOTC: Action perception(Lignau & Downing, 2015)

Scene-trained(Places 205)

Groen, Greene, Baldassano, Fei-Fei, Beck & Baker, 2018Understanding scene perception
Do DNNs predict humanbehavioral scene perception?
Yes, but hand-labeled functions
explain additional variance

Computationalmodels ofscene content
Do DNNs predict humanbrain responses to scenes?
Yes, but not in all regions, andnot much gain with higher layers

Scene
perception
behavior

OPA

Not one region, but
distributed mapping?

Neuralresponses toscenes

Outline• Scene vs. object perception• fMRI study 1: objects-in-context• fMRI study 2: comparing multiple models• How to move forward?

How can we close the loop?
1. DNNs trained on other tasks than object or scene
recognition

Malcolm, Groen& Baker (2016) TICS

2. Other types of DNNs (e.g. RNNs), conceptual models
3. Use simpler and more explicit computational models

Neurophysiologically plausiblemodel of scene statistics
Population response
to local contrast

Parvo

Magno

Frequency

C o n tr a s t e n e r g yσ

S p a tia l fr a g m e n ta tio nμContrast response magnitude
Scholte et al., (2009);
Groen et al., (2012a/b, 2013, 2017)

Model predicts behavioral
categorization
Man-made/natural categorization task

• 1600 scenes• Including ‘ambiguous’

Groen et al., (2013) J Neurosci

Model predicts EEG
responses…

Groen et al., (2013) J Neurosci

.. and decoding of behavior
from EEG
D e c o d in g a c c u r a c y

C o r r e la tio n w ith
d is c r im in a tin g c o m p o n e n t

Groen et al., (2013) J Neurosci

“Closed loop”
Scene
perception
behavior
Computationalmodels ofscene content
Neuralresponses toscenes

OPA

Theories of scene processing
G lo b a l v s . lo c a lC o a r s e - to - fin e

Wolfe et al., 2011

Hochstein & Ahissar 2002

Dynamic feedback for object recognition

Groen et al., 2018, PLoS Comp Biol

Does deep learning explain
scene perception?
Not yet.

Why not?
• Scene perception entails more than object recognition• High-level information not captured by object-trained DNNs
(e.g. functions), is necessary to fully account for scene
perception behavior• Low-level information captured by image statistics, or early
DNN layers, may be important for global scene representation• D N N s c a p a c it y t o p r e d ic t ( f M R I ) r e s p o n s e s in v T C d o e s n o ta u t o m a t ic a lly e x t e n d t o p r e d ic t in g h u m a n s c e n e p e r c e p t io nThank you!
Jon Winawer
Noah Benson
Dora Hermes
Stephanie
Montenegro
Jing-Yang Zhou
C h r is B a k e r
Brett Bankson
Martin Hebart
D w ig h t K r a v itz
M a r c ie K in gEdward Silson
V ic to r L a m m e
S te v e n S c h o lte
S e n n a y G heb reabSara Jahfari
Noor Seijdel

Chris Baldassano (Columbia U)

NYU

N IH

Morgan Barense (U Toronto)
Diane Beck (U Illionois)
Tessa Dekker (UCL)
Michelle Greene (Bates College)
Assaf Harel (Wright University)
Fei-Fei Li (Stanford University)
Natalia Petridou & BAIR team (UMC Utrecht)
Kandan Ramakrishnan (MIT)U vA

www.irisgroen.com