Neural Representations of Faces are Tuned to Eye Movements

Lisa Stacchi, Meike Ramon, Junpeng Lao, & Roberto Caldara*

Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of
Fribourg, Fribourg, Switzerland* Corresponding author:
Roberto Caldara
Department of Psychology
University of Fribourg
Faucigny 21700 Fribourg
Switzerland
Email: roberto.caldara@unifr.ch
Tel: +41 (26) 300 76 36

Running title: Neural Representations of Faces are Tuned to Eye Movements
Word length: 4227

1

ABSTRACT

2

Eye movements provide a functional signature of how human vision is achieved. Many recent

3studies have reported idiosyncratic visual sampling strategies during face recognition. Whether

4

these inter-individual differences are mirrored by idiosyncratic neural responses has not been

5

investigated yet. Here, we tracked observers’ eye movements during face recognition;

6

additionally, we obtained an objective index of neural face discrimination through EEG that7

was recorded while subjects fixated different facial information.

8

Across all observers, we found that those facial features that were fixated longer during face9

recognition elicited stronger neural face discrimination responses. This relationship occurred

10

independently of inter-individual differences in fixation biases. Our data show that eye

11movements play a functional role during face processing by providing the neural system with

12

information that is diagnostic to a specific observer. The effective processing of face identity

13

involves idiosyncratic, rather than universal representations.

14
15
16
17

Keywords: eye movements, individual differences, face discrimination, EEG, fast periodic

18

visual stimulation.

1

1

INTRODUCTION2

The visual system continuously processes perceptual inputs to adapt to the world by selectively

3

moving the eyes towards diagnostic information. As a consequence, eye movements do not

4

unfold randomly, and during face processing humans deploy specific gaze strategies. Since

5

Yarbus’s seminal report (Yarbus, 1967), a multitude of studies have reported a distinct

6

triangular fixation pattern encompassing the eye and mouth region during face recognition

7(Henderson, Williams, & Falk, 2005). For many years, this T-shaped fixation pattern was

8

considered to be universal and shared across observers, suggesting that the presence of a face9

triggers a unique biologically-determined information extraction pattern.

10

However, over the last decade, a growing body of work has challenged this view by

11

revealing cross-cultural (e.g., Blais, Jack, Scheepers, Fiset, & Caldara, 2008; Miellet, Vizioli,

12He, Zhou, & Caldara, 2013), idiosyncratic (Mehoudar, Arizpe, Baker, & Yovel, 2014) and13

even within-observer (Miellet, Caldara, & Schyns, 2011) differences during face recognition.

14

Contrary to the average T-shaped fixation pattern displayed by Westerners, Easterners deploy

15

a global sampling strategy by directing the majority of their fixations toward the center of the16

face, while reaching comparable efficiency in face recognition (for a review see Caldara, 2017).

17

Even among Western observers there is a conspicuous degree of variability in the sampling

18

strategies adopted to achieve face identification (Miellet et al., 2011). In addition, in line with19

early observations based on individual participants (Walker-Smith, Gale, & Findlay, 1977),

20

recent studies demonstrate that observers deploy unique sampling strategies (Kanan, Bseiso,

21

Ray, Hsiao, & Cottrell, 2015; Arizpe, Walsh, Yovel, & Baker, 2017), which are stable over22

time (Mehoudar et al., 2014), and relevant to behavioral performance (Peterson & Eckstein,232013). Individuals’ sampling strategies deviate considerably from the well-established T-

24shaped pattern reported for Westerners observers, which is merely the result of the group25

averaging of the idiosyncratic visual sampling strategies of individual observers (Mehoudar et26al., 2014).

27

Despite the growing literature on the existence of idiosyncratic sampling strategies,

28

their functional role and the underlying neural mechanisms remain poorly understood. Some

29

studies have investigated the impact of the fixated facial information input on neural responses,

30

by recording the electroencephalographic (EEG) signals while observers fixated different facial

31

information (i.e., viewing positions; VPs). This body of work has focused on the N170 ERP

32(Event Related Potential) component (Bentin, Allison, Puce, Perez, & McCarthy, 1996), the21earliest face sensitive neural marker characterized by an occipito-temporal negative deflection

2of the EEG signal 170ms after stimulus onset. Collectively, the results of these studies

3

demonstrated that VPs differentially modulate the N170, with fixation on the eye region

4

electing larger amplitudes (de Lissa et al., 2014, Itier, Latinus, & Taylor, 2006; Nemrodov,

5

Anderson, Preston, & Itier, 2014; Rousselet, Ince, van Rijsbergen, & Schyns, 2014). This

6

observation would suggest a possible universal neural preference toward this facial

7

information. However, these studies have mainly involved grand-average analyses and did not8control for individual fixation preferences. As a consequence, while this analytical approach9

allows enhancing commonalities across observers, it confounds crucial individual differences,10leaving unaddressed the question of whether idiosyncratic fixation biases concur with

11

idiosyncratic neural responses.

12

Fast-periodic visual stimulation (FPVS) has been increasingly used to examine

13

different aspects of face processing, including e.g. face detection, discrimination, and14

categorization (Ales, Farzin, Rossion, & Norcia, 2012; Norcia, Appelbaum, Ales, Cottereau,

15& Rossion, 2015; Rossion, Torfs, Jacques, & Liu-Shuang, 2015; Liu-Shuang, Norcia, &16

Rossion, 2014). Compared to traditional ERPs, the FPVS response has many advantages: it is17

far less susceptible to noise artefacts, and its remarkably high signal-to-noise ratio increases

18

the likelihood of detecting subtle differences between experimental manipulations of interest

19(Norcia et al., 2015). Such signal properties make the FPVS paradigm paired with EEG

20recordings ideal to investigate the potential relationship between VP dependency of neural21

responses and idiosyncratic visual sampling strategies.

22

To this aim, we tracked the eye movements of observers performing an old/new face23

recognition task (Blais et al., 2008) and extracted their fixation patterns. Within the same24

testing session, we then recorded their neural face discrimination responses by means of a

25

FPVS paradigm, while observers fixated faces on one out of ten viewing positions covering all

26

inner facial features (Fig. 2A). We then applied a robust statistical data-driven approach to27

relate the idiosyncratic sampling strategies and the electrophysiological responses across all

28

electrodes independently, without any a-priori assumptions regarding the topography of a29

potential effect. To account for visual sampling idiosyncrasies, this computation was

30

performed at the individual level. Our data show a strong positive relationship between

31

idiosyncratic sampling strategies and neural face discrimination responses as recorded at32

different viewing positions, for all the observers. In particular, independently of the sampling3

1

strategy, the longer a viewing position was fixated under natural viewing conditions, the more2

likely this VP was to elicit the strongest neural face discrimination response when its fixation3

was enforced.

4

METHODS

5

Participants

6

The sample size opted for was motivated by studies using the same FPVS paradigm to index

7

neural face discrimination that were published up to data acquisition (Dzhelyova & Rossion,8

2014a, 2014b; Liu-Shuang et al., 2014; Liu-Shuang, Torfs, & Rossion, 2016; sample size

9

range: 8-12). In Dzhelyova and Rossion’s (2014b) study using a within subject design, the10observed minimal effect size resulting from a repeated ANOVA was .2 (partial-eta). As the11

effect size estimation is often overly optimistic in the literature, we planned our experiment12

based on an effect size of .1 and an estimated sample size of 15 participant which results in a

13

power of .95 to detect an effect. Based on prior experience and the requirement of high quality

14

data from independent methods, we chose to test a total number of 20 participants. Our cohort

15

comprised 20 Western Caucasian observers (11 females, two left-handed, mean age: 25±3

16years) with normal or corrected-to-normal vision and no history of psychiatric or neurological17

disorders. Three observers were excluded due to poor quality of the eye movement data. All

18

participants provided written informed consent and received financial compensation for

19

participation; all procedures were approved by the local ethics committee.

20

Procedures

21

Eye-tracking

22

Stimuli and procedure

23

Stimuli consisted of 112 grey-scaled pictures portraying 56 Western Caucasians (i.e. WC) and24

56 East Asians (i.e. EA) respectively obtained from the KDEF (Lundqvist, Flykt, & Öhman,

251998) and the AFID (Bang, Kim, & Choi, 2001). Faces were presented at a viewing distance26of 75 cm and subtended 12.56° (height from chin to hairline) x 9.72° (width) of visual angle

27

on a VIEWPIxx/3D monitor (1920 x 1080 pixel resolution, 120 Hz refresh rate).

28

Participants performed an Old-New face recognition task (Blais et al., 2008), with two

4

1

blocks, each comprising a learning and a recognition phase of either WC and EA faces. In each2learning phase participants were presented with 14 identities (7 female) with a neutral, happy

3or disgust expression; the recognition phase involved the presentation of these encoded4

identities alongside of 14 new ones, with a change of facial expression for the learned identities

5

in order to prevent for face image matching strategies instead of genuine face recognition.

6

Participants were required to indicate via button press whether a stimulus had been previously

7seen or not. During the learning phase, the faces were presented for five seconds; during the8recognition phase presentation was terminated upon participants’ responses. The eye

9movements were recorded during both the learning and recognition phases.

10

Data acquisition and processing

11

The oculomotor behavior was recorded for each participant using an EyeLink 1000 Desktop

12

Mount with a temporal resolution of 1000 Hz. The raw data are available in the public domain

13(Stacchi, Ramon, Lao, & Caldara, 2018). Data were registered by using the Psychophysics

14(Brainard, 1997) and the EyeLink (Cornelissen, Peters, & Palmer, 2002) Toolbox running in a

15

MatlabR2013b environment. Calibrations and validations were performed at the beginning of16

the experiment using a nine-point fixation procedure. Additionally, before each trial a fixation

17

cross appeared in the center of the screen and participants were instructed to fixate on it until

18

a new stimulus appeared to ensure eye movements were correctly tracked. A new calibration19

was performed if the eye drift exceeded 1° of visual angle.

20

After removing eye blinks and saccades using the algorithm developed by Nystrom et21

al. (Nyström & Holmqvist, 2010), observers’ eye movement data from the learning phases of22

the Old-New task were processed to create individual fixation maps. Previous studies have

23

shown that with this paradigm there are no differences in the sampling strategies used to24samples WC or EA faces (Blais et al., 2008; Caldara, 2017). Therefore, in order to increase the25signal-to-noise ratio, fixation maps were extracted from all face presentations. Individuals’

26

fixation intensities (based on the cumulative fixation duration) per observers were derived

27

using these fixation maps and pre-defined circular regions of interest (ROIs; see Fig. 1). The

28

ROIs covered 1.8° of visual angle and were centered on the ten viewing positions fixated during

29

the FPVS experiment.

5

1
2
3
4
5

Fig. 1. Illustration of the regions of interest (ROI) surrounding the 10 viewing positions. Observers’ fixation maps
were overlaid onto a ROI mask to compute the fixation intensity per ROI. The ROI were covering 1.8° of visual
angle and were centered on 9 equidistant viewing positions (red circles) and on an additional VP corresponding
to the center of the stimulus (black circle).

6

EEG

7

Stimuli and procedure

8

We used full-front, color images of 50 identities (25 female) from the same set described

9

previously (Liu-Shuang et al., 2014). All faces conveyed a neutral expression, were cropped to10

exclude external facial features, and were presented against a grey background. Each original

11

stimulus subtended 11.02° (height) x 8.81° (width) of visual angle at a viewing distance of 70

12

cm.

13

Face stimuli were presented at a constant frequency of 6 Hz, with intervening oddball14

identities every 7th base (0.857 Hz) (Fig. 2A). The experiment comprised 20 trials: ten

15

conditions (the viewing positions participants were required to fixate on; Fig. 2B), with two

16

62s trials per condition (trials differed with respect to the gender of the face stimuli). To prevent

17

eye-movements, participants were instructed to maintain fixation on a central cross. The

18

position of face stimuli was manipulated to vary, across trials, the fixated viewing position,

19

hence the facial information. Faces were presented through sinusoidal contrast modulation (see20

Fig. 2A). Additionally, two seconds of gradual fade in and fade out were added at the beginning21

and end of each trial. To maintain subjects’ attention, the fixation cross briefly (200ms)

22changed color (red to blue) randomly between seven and eight times within each trial;

23

participants were instructed to report the color change by button press. Subjects were also

24monitored trough a camera placed in front of them communicating the experimenter computer.

25

Finally, to avoid pixel-wise overlap, stimulus size varied randomly between 80% and 120% of

26

the original size (visual angle ranged between 8.82-13.22° (height) and 7.05-10.57° (width)).

6

1
2
3
4
5
6

Fig. 2. FPVS paradigm and viewing positions. (A) Faces were presented at a frequency rate of 6Hz throughsinusoidal contrast modulation. Base stimuli consisted of images of the same facial identity; interleaved oddballstimuli conveying different identities were presented every 7th base stimulus. (B) Illustration of the 10 viewing
positions (VPs) fixated by participants. (C) Examples of two trials displaying fixation on the left eye (VP1, toprow), or mouth (VP8, bottom row).

7

Data acquisition and processing

8

Electrophysiological responses were recorded with Biosemi Active-Two amplifier system

9(Biosemi, Amsterdam, Netherlands) with 128 Ag/AgCl active electrodes and a sampling rate10

of 1024Hz. Additional electrodes placed at the outer canthi and below both eyes registered eye

11movements and blinks; electrode impedance was maintained between ±25kΩ. The recorded

12

EEG was analyzed using Letswave 5 (http://nocions.webnode.com/letswave; (Mouraux &13

Iannetti, 2008)). The raw data are available in the public domain (Stacchi et al., 2018).

14

Preprocessing consisted in high- and low-pass filtering the signal (with a 0.1Hz and 100Hz

15

Butterworth band-pass filter (4th order). Data were subsequently downsampled to 256Hz and16

segmented according to condition resulting in 20 66-second epochs, which included two

17

seconds before and after stimulation. Independent component analysis was performed on each18

participant’s data to remove contamination due to eye-movements.

19

Noisy electrodes were interpolated using the three nearest spatially neighboring

20

channels; this process was applied to no more than 5% of all scalp electrodes. Segments were21

then re-referenced to a common average reference and cropped to an integer number of oddball22

cycles, excluding two seconds after stimulus onset and two seconds before stimulus offset
7

1(~58-second epochs; 14932 bins). Epochs were then averaged separately for each subject per2

condition.

3

Frequency domain

4

Fast Fourier Transform (FFT) was applied to the averaged segments and amplitude was

5extracted. The data were baseline corrected by subtracting from each frequency’s amplitude

6

the average of its surrounding 20 bins excluding the two neighboring ones. Finally, for each7

subject and condition, the summed baseline-corrected amplitude of the oddball frequency and8its significant harmonics provided the index of neural face discrimination. Following previous

9

procedures (Dzhelyova, Jacques, & Rossion, 2016), harmonics were considered significant

10

until the mean z-score across all conditions was no longer above 1.64 (p<.05). Based on this11

criterion we considered the first 11 harmonics excluding the 7th harmonic, which is confounded

12

with the base stimulation frequency rate.

13

Analysis

14

Using the iMAP4 toolbox (Lao, Miellet, Pernet, Sokhn, & Caldara, 2017) we computed a linear

15

regression to explore the relationship between fixation bias (the z-scored fixation duration) and16

neural face discrimination (i.e., the FPVS response amplitude) within observer. To this aim we

17

performed a linear mixed-effects model with random effect for intercept and Fixation duration

18

grouped by subject. To avoid a-priori assumptions regarding topography of the effect, we

19

regressed the two variable at all scalp electrodes independently.

20

FPVS_amplitude ~ 1 + Fixation_duration + (1 + Fixation_duration | Subjects)

(1)

21

This computation will determine whether, VP-dependent fixation duration are

22

associated with the amplitude of the neural face discrimination response elicited by each VP,

23for each subject independently. Importantly, because the analysis is performed at the individual

24level, there is no a priori expectation on how VPs are ranked. We opted for this approach in25

light of individual differences in fixation patterns reported previously (Mehoudar et al., 2014;26

Arizpe et al., 2016; Kanan et al., 2015), and similar idiosyncrasies assumed to exist for neural27

face discrimination responses across VPs. Therefore, the model used here allows each subject

28

to have his/her specific VP-pattern and a relationship emerges if the fixation pattern is29

predictive of the neural response pattern of the same subject. Finally, as the current work does8

1

only focus on individual subjects, we did not perform any analysis involving average fixation2

maps and average EEG responses. The processed data used to compute this analysis are

3

available in the public domain (Stacchi et al., 2018)

4

RESULTS

5

Behavior

6

As expected, subjects’ performance in the Old-New task, as indexed by d’, was significantly

7

better for Western Caucasian (M=1.62, SD=.64) than East Asian faces (M=0.97, SD=.60),

8t(16)=5.72, p<.01. Subjects’ performance was nearly at ceiling for the FPVS orthogonal task

9consisting in detecting the color-change of the fixation-cross (M=.95, SD=.11). Due to10

technical issues, one subject’s behavioral response at this orthogonal task was not recorded.

11

However, as for all subjects, behavior was monitored trough a webcam.

12

Eye-movements and FPVS response

13

Description of fixation and neural biases at the group and individual level

14

The average fixation maps (computed for descriptive purposes and) shown in Figure 3A

15

demonstrate that as a group observers preferentially sampled facial information encompassing

16

the eyes, nasion, nose and mouth. However, because the focus of this work was to investigate

17

the relationship between fixation patterns and neural responses at the individual level, group18

data were not subject to any further analysis.

19

At the individual level, the majority of individual observers’ fixation maps did not20

perfectly conform to the grand average fixation pattern (Figure 3A-B – see also supplementary

21

Figure S1), clearly demonstrating the existence of idiosyncratic visual sampling strategies (see22

Figure S1 for all individual observers’ fixation maps). Mirroring these results, the grand23

average neural face discrimination response amplitudes varied as a function of VPs, with the24

greatest amplitudes for the central position (Figure 3C). However, the neural response25

amplitudes also markedly differed across individuals (Figure 3D).

9

1
2
3
4

Fig. 3. Fixation maps and oddball responses. A and C show the grand-average fixation map and FPVS responses
respectively, while B and D show the two measures for the same subjects. For illustration, only four subjects arereported.

5

Regression analysis: Assessing the relationship between fixation and neural biases at the6

individual level

7

The data-driven regression between individuals’ fixation durations and FPVS responses

8

across VPs computed independently on all electrodes revealed a positive relationship at right9

occipito-temporal (OT) and central-parietal (CP) clusters (see Figure 4A).

10

The occipito-temporal cluster includes 12 significant electrodes with the strongest

11effect at A28 (F(1,169)=30.02, ß=.22 [.14 .30], p=1.54e-07) and the smallest at A13

12(F(1,169)=17.18 ß=.20 [.10 .29], p=5.37e-05) (Table S1). The average across the electrodes in13

the OT significant cluster shows that individual subjects exhibited variable intercepts but

14

similar slopes (see Fig. 4B). Despite inter-individual variations in the neural face15

discrimination response amplitude and fixation durations, the relationship was present within

16

all the observers (see Fig. 5).

10

1

A small effect was also found on the central-parietal cluster comprising seven2electrodes, with D14 showing the strongest effect (F(1,169)=26.12 ß=.13 [.08 .17], p=9.64e-

307) and C1 exhibiting the smallest effect (F(1,169)=17.78 ß=.11 [.06 .17], p=4.05e-05) (Fig.

4

4A; Table S1).

5
6
7
8
9
10
11
12
13

Fig. 4. The relationship between fixation duration and neural face discrimination responses across VPs observed
across all subjects considered individually. (A) Regression F-values (left) and beta maps (right). Original maps
were overlaid with a mask and only electrodes exhibiting a significant effect (p< 7.81e-05) are shown. (B) On theleft, the scatterplot illustrates individual subjects’ (light grey lines) as wells as the group (black line) effectaveraged across the significant occipito-temporal cluster of electrodes. Crucially, the ranking of fixation andneural bias for VPs differed across subjects. On the right, data of two subjects illustrate that the relationshipbetween fixation and neural bias emerges irrespectively of the fixation pattern exhibited (i.e., left-eye for S01 andmouth for S16). Note that here individual subjects’ correlations are displayed (black line).

14

11

1
2
3
4
5
6
7
8

Fig. 5. The relationship between individual subjects’ fixation and neural bias across VPs. For each observer (S1S17), the VP-dependent fixation duration (x-axis) is plotted against their neural face discrimination response
amplitude (y-axis) along with their individual correlation (black line); VPs are color- and shape-coded as indicatedin the legend. The subjects are ordered as a function of their correlation strength, from high (.227) to low (.219).
Although observers exhibited idiosyncratic VP-dependent fixation durations, all showed the general pattern, that
facial features fixated longer (i.e., VPs) elicited larger neural responses. Note that here the neural facediscrimination response magnitude is displayed at the electrode showing the largest effect (i.e., A28).

9
10
11
12

Can specific fixation biases account for the observed relationship?

13

show stronger correlations between fixation and neural biases, we first ranked observers’

14

fixation maps based on the magnitude of their individual relationship. As shown in Figure 6A,

15

subjects showing similar fixation patterns could exhibit relationships of slightly different

16

magnitude (e.g., left eye: S01 and S04), while observers exhibiting different fixation maps

17

could show comparable correlation strengths (e.g., S10 and S13). Additionally, we computed18

the distance of each observer’s fixation map from the average fixation pattern. In this case,19

each map is treated as a vector and the measure of interest is the cosine distance between each20

observers’ map and the average one. This produces a value ranging between 0 and 1 for each21

subject. The higher the distance the more dissimilar that given subject’s pattern is from the22average. Finally, we performed a Spearman correlation between this distance and the strength

To assess whether subjects exhibiting a particular fixation bias (e.g., for the eyes) would12

1of the relationship between fixation and neural bias, which resulted to be non-significant (r=-2.18, p=.49) (Figure 6B).

3

4
5
6
7
8
9
10

Fig. 6. Fixation maps and strength of the fixation-neural-bias relationship. (A) Observers’ fixation maps sorted asa function of the slope of observers’ relationship between fixation bias and neural face discrimination response
amplitude. The slope is reported for the electrode showing the strongest effect (i.e., A28). (B) The scatterplot
illustrates the lack of correlation between: the cosine distance of individuals’ fixation maps from the average
fixation map (y-axis) and strength of the relationship between fixation and neural bias (x-axis). Data show there
was not a particular fixation bias more likely to correlate with the neural bias.

11

DISCUSSION

12This study investigated the relationship between idiosyncratic visual sampling strategies for13

faces and the magnitude of neural face discrimination responses during fixation on different

14

types of facial information. Our data show that visual information sampling is distinct across15

observers, and these differences are positively correlated with idiosyncratic neural responses

16

predominantly at occipito-temporal electrodes. Specifically, the Viewing Positions (VPs) that17elicited stronger neural face discrimination responses coincided with the VPs that were more

18

fixated under free-viewing conditions. Altogether, our data show that face processing involves

19

idiosyncratic coupling of distinct information sampling strategies and unique neural responses

20

to the preferentially sampled facial information.

21

For many years, the accepted notion in vision research was that face processing elicits

22

a unique and universal cascade of perceptual and cognitive events to process facial identity,

23

with particular importance ascribed to information conveyed by the eye region. For instance,24

eye movement studies have revealed a bias towards sampling of the eye region (Blais et al.,252008), the diagnosticity of which has been further documented by psychophysical approaches
13

1(e.g., Bubbles) (Gosselin & Schyns, 2001). Electrophysiological studies have also reported

2

increased N170 magnitude during fixation on the eyes, compared to other information (de Lissa

3

et al., 2014; Nemrodov et al., 2014). Collectively, these independent findings were taken to

4

support the existence of a fixation and neural preference for the eye-region that is shared across5

all observers.

6

However, this idea has recently been challenged. For example, findings from eye

7

movement studies emphasize idiosyncrasies in sampling preferences that are highly distinct

8from the group-average T-shaped pattern (Arizpe et al., 2017; Mehoudar et al., 2014), or by

9

the existence of cultural differences (Blais et al., 2008; Caldara, 2017). These individual

10

differences are not systematically associated with performance, as “mouth lookers” (i.e.,

11observers showing preferential fixation on the mouth) could perform similarly to “eyes12lookers”. Equally, two “eyes lookers” could exhibit very different performance (Peterson &13

Eckstein, 2013). Nonetheless, each observer’s adopted sampling strategy is optimal in the sense14

that performance is maximal when fixation is enforced on preferably sampled information, and15

decreases during fixation of other information (Peterson & Eckstein, 2013). These results

16

suggest that individual differences do not reflect random inter-subject variation, but rather

17

subtend functional idiosyncrasies in face processing.

18

Our results replicate and extend these previous findings, by showing that idiosyncratic

19

visual sampling strategies strikingly mirror individuals’ patterns of neural face discrimination20

responses across VPs. Specifically, the facial regions preferentially sampled during natural

21

viewing were those eliciting stronger neural face discrimination responses when fixated. This22

pattern was present in all observers, with even some of them (n=4) showing a perfect match

23

between the most fixated facial feature and the one eliciting the strongest neural response at24

the electrode showing the strongest statistical relationship.

25

The strong and striking relationship between information sampling and neural

26

idiosyncrasies suggests a functionally relevant process. Eye movements feed the neural face27

system with the diagnostic information in order to optimize information processing. The eyes

28

constantly move to center elements of interest in the fovea, where visual acuity is greatest. This29

critical functional role, coupled with the relationship reported here between idiosyncratic

30

sampling strategies and the neural face discrimination response pattern thus leads to two main31

considerations. First, our data show that face identity processing involves a fine-tuned interplay

32

between oculomotor mechanisms and face-sensitive neural network. Second, the diagnosticity
14

1

associated different types of facial information varies across observers. For a long time,2

researchers have debated on the nature of face representations, mainly opposing the idea of3

faces being represented as indivisible wholes (holistic or configural), as opposed to a collection

4of multiple, distinctively perceivable features (featural). This ongoing debate cannot be settled

5

based on our finding of visual and neural idiosyncrasies. These idiosyncrasies do, however,6

refute the concept of a single face representation format shared across observers.

7

Our observations raise further important methodological and theoretical questions. The

8

first concerns the traditional approach of standardizing the visual input to allow comparability

9

across observers. This inherently creates a bias as not all observers are comparably tuned to10this conventional VP. Additional open questions concern for instance (a) the extent to which11

the relationship between the visual sampling strategies and neural response patterns is task-

12

specific, and (b) the direction of this relationship. Future studies are also necessary to establish

13

whether similar effects can be found for scene, object and word processing. Finally, our14

approach may offer a promising novel route in clinical settings, if disorders comprising face

15

processing impairments (i.e., prosopagnosia, autism, schizophrenia, etc.) involved an abnormal

16

relationship between fixation patterns and neural responses to faces.

17

CONCLUSION

18

When processing faces, observers deploy idiosyncratic sampling strategies by preferentially19

moving their eyes towards specific features. To assess the neural correlates underlying these20

idiosyncrasies, we recorded eye movements and neural face discrimination responses by means

21

of FPVS. Our data show that idiosyncratic sampling strategies are mirrored by neural22

preferences, as facial information that is sampled longer during free viewing elicits stronger

23

neural face discrimination responses when fixation is enforced on this feature. These findings

24

show that eye movements and neural responses are finely tuned towards the facial information

25

that is optimal for individual observers. The biologically relevant feat of processing facial

26

identity is achieved by the use of idiosyncratic visual and neural representations.

27

15

1

REFERENCES

2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

Ales, J. M., Farzin, F., Rossion, B., & Norcia, A. M. (2012). An objective method formeasuring face detection thresholds using the sweep steady-state visual evoked response.
Journal of Vision, 12(10), 18–18. https://doi.org/10.1167/12.10.18
Arizpe, J., Walsh, V., Yovel, G., & Baker, C. I. (2017). The categories, frequencies, andstability of idiosyncratic eye-movement patterns to faces. Vision Research, 141, 191–203.
https://doi.org/10.1016/j.visres.2016.10.013
Bang, S., Kim, D., & Choi, S. (2001). Asian Face Image Database. In Lab IM. Postech, Korea.
Bentin, S., Allison, T., Puce, A., Perez, E., & McCarthy, G. (1996). ElectrophysiologicalStudies of Face Perception in Humans. Journal of Cognitive Neuroscience, 8(6), 551–
565. https://doi.org/10.1162/jocn.1996.8.6.551
Blais, C., Jack, R. E., Scheepers, C., Fiset, D., & Caldara, R. (2008). Culture Shapes How We
Look at Faces. PLoS ONE, 3(8), e3022. https://doi.org/10.1371/journal.pone.0003022
Brainard, D. H. (1997). The Psychophysics Toolbox. Spatial Vision, (10), 433–436.
Caldara, R. (2017). Culture Reveals a Flexible System for Face Processing. Current Directionsin Psychological Science, 26(3), 249–255. https://doi.org/10.1177/0963721417710036
Cornelissen, F. W., Peters, E. M., & Palmer, J. (2002). The Eyelink Toolbox: Eye tracking with
MATLAB and the Psychophysics Toolbox. Behavior Research Methods, Instruments,
and Computers.
de Lissa, P., McArthur, G., Hawelka, S., Palermo, R., Mahajan, Y., & Hutzler, F. (2014).
Fixation location on upright and inverted faces modulates the N170. Neuropsychologia,
57, 1–11. https://doi.org/10.1016/j.neuropsychologia.2014.02.006
Dzhelyova, M., Jacques, C., & Rossion, B. (2016). At a Single Glance: Fast Periodic Visual
Stimulation Uncovers the Spatio-Temporal Dynamics of Brief Facial Expression Changes
in
the
Human
Brain.
CerebralCortex,27(8),
4106–4123.
https://doi.org/10.1093/cercor/bhw223
Dzhelyova, M., & Rossion, B. (2014a). Supra-additive contribution of shape and surface
information to individual face discrimination as revealed by fast periodic visual
stimulation. Journal of Vision, 14, 15–15. https://doi.org/10.1167/14.14.15
Dzhelyova, M., & Rossion, B. (2014b). The effect of parametric stimulus size variation on
individual face discrimination indexed by fast periodic visual stimulation. BMC
Neuroscience, 15(1), 87. https://doi.org/10.1186/1471-2202-15-87
Gosselin, F., & Schyns, P. G. (2001). Bubbles: a technique to reveal the use of information inrecognition tasks. Vision Research, 41(17), 2261–2271. https://doi.org/10.1016/S00426989(01)00097-9
Henderson, J. M., Williams, C. C., & Falk, R. J. (2005). Eye movements are functional during
face learning. Memory & Cognition, 33(1), 98–106.
Itier, R. J., Latinus, M., & Taylor, M. J. (2006). Face, eye and object early processing: What istheface
specificity?
NeuroImage,29(2),
667–676.
https://doi.org/10.1016/j.neuroimage.2005.07.041
Kanan, C., Bseiso, D. N. F., Ray, N. A., Hsiao, J. H., & Cottrell, G. W. (2015). Humans haveidiosyncratic and task-specific scanpaths for judging faces. Vision Research, 108, 67–76.
https://doi.org/10.1016/j.visres.2015.01.013
Lao, J., Miellet, S., Pernet, C., Sokhn, N., & Caldara, R. (2017). iMap4: An open source toolbox
for the statistical fixation mapping of eye movement data with linear mixed modeling.
Behavior Research Methods, 49(2), 559–575.

16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

Liu-Shuang, J., Norcia, A. M., & Rossion, B. (2014). An objective index of individual facediscrimination in the right occipito-temporal cortex by means of fast periodic oddballstimulation.
Neuropsychologia,
52,
57–72.
https://doi.org/10.1016/j.neuropsychologia.2013.10.022
Liu-Shuang, J., Torfs, K., & Rossion, B. (2016). An objective electrophysiological marker offace individualisation impairment in acquired prosopagnosia with fast periodic visual
stimulation.
Neuropsychologia,
83,
100–113.
https://doi.org/10.1016/j.neuropsychologia.2015.08.023
Lundqvist, D., Flykt, A., & Öhman, A. (1998). The Karolinska directed emotional faces.
Stockholm, Sweden: Karolinska Institute. Stockholm. Sweden: Karolinska Institute.:
Stockholm. Sweden: Karolinska Institute.
Mehoudar, E., Arizpe, J., Baker, C. I., & Yovel, G. (2014). Faces in the eye of the beholder:
Unique and stable eye scanning patterns of individual observers. Journal of Vision, 14(7),6. https://doi.org/10.1167/14.7.6
Miellet, S., Caldara, R., & Schyns, P. G. (2011). Local Jekyll and Global Hyde. Psychological
Science, 22(12), 1518–1526. https://doi.org/10.1177/0956797611424290
Miellet, S., Vizioli, L., He, L., Zhou, X., & Caldara, R. (2013). Mapping Face Recognition
Information Use across Cultures. Frontiers in Psychology, 4, 34.
https://doi.org/10.3389/fpsyg.2013.00034
Mouraux, A., & Iannetti, G. D. (2008). Across-trial averaging of event-related EEG responses
andbeyond.
Magnetic
Resonance
Imaging,26(7),
1041–1054.
https://doi.org/10.1016/j.mri.2008.01.011
Nemrodov, D., Anderson, T., Preston, F. F., & Itier, R. J. (2014). Early sensitivity for eyes
within faces: A new neuronal account of holistic and featural processing. NeuroImage,97, 81–94. https://doi.org/10.1016/j.neuroimage.2014.04.042
Norcia, A. M., Appelbaum, L. G., Ales, J. M., Cottereau, B. R., & Rossion, B. (2015). The
steady-state visual evoked potential in vision research: A review. Journal of Vision, 15(6),4. https://doi.org/10.1167/15.6.4
Nyström, M., & Holmqvist, K. (2010). An adaptive algorithm for fixation, saccade, andglissade detection in eyetracking data. Behavior Research Methods, 42(1), 188–204.
https://doi.org/10.3758/BRM.42.1.188
Peterson, M. F., & Eckstein, M. P. (2013). Individual Differences in Eye Movements During
Face Identification Reflect Observer-Specific Optimal Points of Fixation. Psychological
Science, 24(7), 1216–1225. https://doi.org/10.1177/0956797612471684
Rossion, B., Torfs, K., Jacques, C., & Liu-Shuang, J. (2015). Fast periodic presentation ofnatural images reveals a robust face-selective electrophysiological response in the humanbrain. Journal of Vision, 15(1), 18–18. https://doi.org/10.1167/15.1.18
Rousselet, G. A., Ince, R. A. A., van Rijsbergen, N. J., & Schyns, P. G. (2014). Eye coding
mechanisms in early human face event-related potentials. Journal of Vision, 14(13), 7–7.
https://doi.org/10.1167/14.13.7
Stacchi, L., Ramon, M., Lao, J., & Caldara, R. (2018). Data from: Neural representations of
faces
are
tuned
to
eye
movements.
Dryad
DigitalRepository.
https://datadryad.org/review?doi=doi:10.5061/dryad.89c5k58
Walker-Smith, G. J., Gale, A. G., & Findlay, J. M. (1977). Eye Movement Strategies Involved
in Face Perception. Perception, 6(3), 313–326. https://doi.org/10.1068/p060313
Yarbus, A. L. (1967). Eye Movements During Perception of Complex Objects. In Eye
Movements and Vision (pp. 171–211). Boston, MA: Springer US.

17

SUPPLEMENTAL INFORMATION

Fig. S1. Fixation maps and neural face discrimination responses of all subjects.

1

Table S1.
F, beta and p-values for electrodes showing a significant effect.
Fßp-value

17.19
21.54
21.78
18.08
26.88
29.91
19.73
30.02
29.91
28.26
22.00
23.55

0.20
0.20
0.18
0.18
0.22
0.26
0.20
0.22
0.28
0.29
0.25
0.24

5.37e-05
6.96e-06
6.22e-06
3.50e-05
6.16e-07
1.62e-07
1.61e-05
1.54e-07
2.51e-07
3.32e-07
5.62e-06
2.76e-06(A) OT
A13
A14
A23
A24
A25
A26
A27
A28
B7
B8
B9
B10
(B) CP
B1
19.27
0.12
1.99e-05
C1
17.78
0.11
4.05e-05
D1
20.35
0.13
1.21e-05
D2
23.04
0.13
3.49e-06
D14
26.12
0.13
8.64e-07
D15
21.93
0.12
5.79e-06
D16
17.80
0.10
3.99e-05
Clusters: OT - occipito-temporal; CP - centro-parietal

2