Eye movements during free viewing to maximize scene understanding

Domain: neuroscience
Published: 2025-12-21
Editorial outcome: major_revision

Abstract
What humans look at and do when freely viewing a scene is not well understood. We measure observer eye movements under different instructions while participants view a customized set of image pairs containing small visual alterations that greatly change scene interpretation (Winograd images). We show that free-viewing fixations resemble those of observers describing scenes but differ from those of observers counting or searching for objects. Fixations are more often directed toward people and objects whose removal most alters scene interpretation, rather than toward the most salient or meaningfully judged regions (meaning maps), or objects perceived to be grasped or gazed at. Small image changes that modify scene understanding (Winograd images), but not salience or meaning maps, alter fixation patterns. By instructing observers to describe scenes while fixating on objects either relevant or irrelevant to scene understanding, we demonstrate that free-viewing eye movements are functionally important for accurate scene comprehension. Thus, an important human default task of free viewing eye movements is to comprehend scenes.

Introduction
Eye movements are a critical part of human vision. Ever since Buswell (1935) and Yarbus (1967) conducted their classic eye-tracking studies, researchers have tried to understand what drives and influences eye movements. One of the prominent theories proposed two decades ago is that people make eye movements to salient regions. Saliency was originally defined (Itti et al.) in terms of a combination of low-level features such as color, intensity, and orientation at any given location relative to its immediate surroundings, with many subsequent elaborations of the model. Since then, studies have shown that when humans perform specific tasks such as searching for an object or person, executing a motor action, navigating in an environment, or identifying faces, they do not look at the most salient object/region. Instead, humans look at locations that contain objects or visual features relevant to the task, and/or that allow maximizing task accuracy. When instructed to artistically evaluate or describe scenes using keywords, observers fixate on objects more than salient regions.

Humans are not always engaged in specific tasks. They might look through a bus window, sit on a park bench, wait for a restaurant table, and explore a scene with no particular task. This is often called free viewing. Fixations during free viewing have been commonly used to support the theory that people direct eye movements to salient regions. More recently, studies have shown that even during free viewing, people do not direct their eyes to salient regions but rather to objects or meaningful regions measured by observers’ subjective judgments of the meaningfulness of segmented local patches of scenes (meaning maps; subsequently referred to as locally meaningful regions).

Here, we introduce a new theory. We hypothesize that, during free viewing, humans aim to understand a scene. To rapidly do so, they direct their eyes to regions critical to scene understanding rather than looking at those salient or segmented regions, locally judged to be meaningful. An important distinction between a locally meaningful region and one critical to scene understanding is that the latter considers how the region, people, or object contributes to understanding the whole scene. For example, Fig. 1a illustrates the concept with an image described by observers as: Someone replacing batteries of a TV remote control. The highly contrasting blue paper is the most salient object, and the sunglasses are the most locally meaningful object. However, the inverted remote with its battery cap open is the most critical object to the scene’s understanding. How would one measure the contribution of an object to the scene’s understanding? We propose a new method by having observers describe the scene and assess the impact of removing one object at a time on the scene descriptions. Removing the remote control from the image in Fig. 1a significantly alters the description of the scene, whereas eliminating the sunglasses or the blue paper does not (see Fig. 1a, right side). Thus, our theory hypothesizes that the remote control, critical to scene understanding, would attract more human fixations than the blue paper or the sunglasses. Furthermore, our second hypothesis is that fixating the object critical to scene understanding has a functional role in extracting the information required to understand it. The theory would predict that directing the high-resolution fovea to the remote control would be important to understand the scene (accurately describe it), while fixating on the blue paper would not. Although previous work has related eye movements to scene comprehension when observers are instructed to describe scenes, these studies have not examined how free-viewing fixations relate to the contributions of each object to scene understanding, evaluated other tasks or models, or experimentally investigated the causal functional importance of fixations on scene understanding.

Our initial methodology and analysis focused on predicting human fixations on different objects. In the latter part of the paper, we generalize the analysis to assess whether the theoretical framework can predict the well-documented frequent fixations on people (bodies and faces; for example, deleting the hand with the batteries in Fig. 1a would maximally alter the description of the scene). We also assess how social cues (gaze and grasp cues) fit the theoretical framework.

One of the main challenges in evaluating the hypothesis that eye movements are directed to people or objects that maximize scene understanding is that in most image datasets the person or object critical to scene understanding is often also the most salient object and the locally most meaningful region. This is a consequence of photographers’ practice of typically positioning the people or objects critical to the scene’s understanding as salient and central in the photograph. To overcome this challenge, we developed a new dataset that decouples salient regions from regions that contribute to scene understanding. We created pairs of real scenes that were minimally altered visually but resulted in large changes in the scene’s understanding (Fig. 1b; left) while maintaining the saliency maps of the scenes and the local meaningfulness of each region. These small visual changes across image pairs could be manipulations of an object’s position, its substitution by another object, or an actor’s posture. We refer to these image pairs as Winograd Images (WI), inspired by the Winograd schema developed for sentences for which minor modifications of a word in a sentence lead to a large change in the sentence’s meaning. To determine which objects were critical to the scene’s understanding, we assessed how deleting each object impacted the accuracy of scene descriptions relative to the intact images.

To test our first hypothesis, we had different groups of observers view our Winograd pair image dataset following different condition instructions (experiment 1): search for a specific object (object search; OS), describe the scene (scene description; SD), count the number of objects (counting objects; CO), or critically, free viewing (no instructions; FV). If observers execute eye movements during free viewing to understand the scene, we would expect various specific results. First, fixation patterns during free viewing should be similar to those obtained from observers during the scene description condition, which explicitly requires the observer to understand a scene to describe it. Fixation locations during free viewing should be less similar to the fixations of observers executing other specific tasks that do not require understanding the scene, such as searching for an object or counting objects. Second, Winograd image pairs, which alter scene understanding, should result in varying fixation patterns (across Winograd images) for the scene description and free viewing conditions, but less for the object search and counting conditions. Third, fixations during the free viewing and scene description conditions should be directed to objects critical to scene understanding. In contrast, fixations during the object search would be directed to the object searched (target), and fixations during object counting should be equally spread among objects. As benchmark comparisons, we also evaluated the ability of a saliency model (Graph-Based Visual Saliency (GBVS)), a human eye movement-trained deep neural network (DeepGaze), and measurements of the most meaningful local regions (meaning maps) to predict the fixations across Winograd pairs. We hypothesize that the objects critical to scene understanding in our data would be fixated more frequently than the most salient or locally meaningful region (as measured by meaning maps) for the free viewing and scene description conditions.

To test the functional role of fixations during free viewing in scene understanding, we conducted a second experiment. Observers described a scene while maintaining fixation at an object critical to scene understanding or another object irrelevant to scene comprehension. Suppose that fixations play a functional role in rapidly extracting the information necessary to understand a scene. In that case, scene descriptions should be more accurate when observers maintain fixation on objects relevant to scene understanding than when they fixate on objects irrelevant to comprehending the scene.

Results
Free viewing fixations are most similar to scene description
We created a total of 18 Winograd pairs. Fifty participants took part in each of the four conditions (a between-subjects design, with a total of 200 participants). Within each condition, participants were assigned randomly and equally to view one set of Winograd pairs (18 image trials). Figure 1c details the procedural flow of the four conditions.

Our primary analyses removed fixations/model predictions on human faces or bodies in the scenes. This pre-processing enabled us to focus on fixations on objects that were critical versus those irrelevant to scene understanding, rather than on human figures, which were critical across all Winograd image pairs. Figure 2a shows an example of the fixation heatmaps (see methods for details) generated using 25 participants for each of the four conditions. The gray regions indicate fixations on people that were not considered in our primary analysis (but see further below for an analysis including fixations on people). Figure 2b shows the correlation of fixation heatmaps across conditions (for 25 observers) computed for each of the 36 images (18 pairs) in our dataset. The fixation heatmaps for free viewing were significantly more similar to the fixation heatmaps from the scene description condition (FV-SD, r = 0.54) than to those from the object search (FV-OS, r = 0.32, bootstrap resampling of observers and images for all analyses, p < 0.001 vs. FV-SD, Cohen’s d = 1.28) or counting objects (FV-CO, r = 0.42, p < 0.001 vs. FV-SD, Cohen’s d = 0.75, see Figure Supplementary 1a). To compare the across-condition fixation heatmap correlations to an upper bound, we estimated a within-condition (FV-FV) correlation of fixation heatmaps, each using distinct subgroups of n = 12 observers (Fig. 2c, see methods). As with the n = 25 analysis, we found higher FV-SD fixation heat map correlations (r = 0.4) than those for FV-OS (r = 0.23, p < 0.001, Cohen’s d = 1.19) and FV-CO (r = 0.29; p < 0.001, Cohen’s d = 0.9). Notably, the FV-SD fixation heatmap correlation was the closest to the within-condition (FV-FV) inter-observer (see dotted lines in Fig. 2c) heatmap correlations (Δr = rFV-FV - rFV-SD = 0.06) relative to the other condition comparisons (the FV-OS vs. FV-FV correlations: Δr = 0.26; and the FV-CO vs. FV-FV correlations: Δr = 0.17).

Changes in scene understanding with small visual manipulations influence observer fixations
We compared the measured human fixation patterns across the Winograd image pairs for different conditions with other existing fixation prediction models: a purely low-level saliency model (GBVS), a neural network model trained on human fixations and images to predict fixations on novel images (DeepGaze II), meaning maps that quantify the meaningfulness of local regions of scenes, and our newly proposed scene understanding maps that quantify the contribution of each object to the meaning of the entire scene. Figure 3a, b provides a flowchart of all the models we have used in this comparison.

To measure the contribution of an object to scene understanding, we developed a technique that measures the impact of an object present in the scene on the description provided by participants. Figure 3a illustrates the procedure with an example. The original descriptions for the intact (i.e., no object removed) images were compared with the descriptions of the images after the object removal to quantify the object’s impact on scene understanding. If the object is irrelevant to scene understanding, the scene descriptions with the object present or removed would be similar. In contrast, if the object is critical to scene understanding, removing it would greatly impact the scene descriptions. The similarity of the scene descriptions can be measured by relying on a separate group of participants who view sentence pairs (without the image) and rate their similarity. Quantifying the impact of removing each object on the scene description allows us to create a map that visualizes the objects’ contribution to scene understanding (scene understanding map, SUM). The SUM can be used to predict fixation distributions and the most fixated object (see methods and Fig. 4a). Similar SUM results were obtained using similarity ratings of descriptions based on automated Large Language Models (LLMs) based metrics (cosine similarity computed with the LLM embeddings of the sentences) rather than human ratings. The human-LLM rating agreement (rGemini-Human = 0.7, rGPT4-Human = 0.72) was comparable to the average human-human (r = 0.75) agreement in the similarity ratings of these descriptions (see Figure Supplementary 1b, c).

We hypothesized that altering the scene understanding through small visual manipulations would change the human fixations in both the free viewing and scene description conditions. Thus, we predicted that the correlation of human fixation distributions across Winograd image pairs would be smaller than that for the saliency, local meaningful regions, and DeepGaze prediction maps. Similar to the human fixation distributions, our hypothesis predicts that the correlation across the scene understanding maps of Winograd image pairs would also be low. In addition, human fixation heat maps would be more similar (higher correlation) across the Winograd image pairs for the object search and counting objects conditions compared to the free viewing and scene description conditions.

Figure 4a shows examples of the predicted fixation heatmaps of different models and measured fixation heatmaps for humans for the four conditions for a Winograd image pair. The correlation between the human fixation heatmaps of Winograd images for the free viewing condition (Fig. 4b) was significantly lower than for the object search (p < 0.001, Cohen’s d = 1.43) and counting objects (p = 0.018, Cohen’s d = 0.93) conditions. The correlation of Winograd image fixation heat maps for the scene description condition was significantly lower than for the free viewing condition (p = 0.034, Cohen’s d = 0.53). The correlations of fixation prediction heatmaps between Winograd images for saliency, meaning maps, and DeepGaze were significantly higher (p < 0.001 vs. free-viewing for all comparisons, Cohen’s d = 1.86, 1.68, 1.25, respectively) than the correlations observed for free-viewing human fixation heatmaps between Winograd pairs. In contrast, the correlation of the fixation predictions of Winograd image pairs for the SUM was the lowest and not different from that observed for human fixations for the free viewing and scene description conditions (p = 0.19, Cohen’s d = 0.54 for free viewing; p = 0.51, Cohen’s d = 0.12 for scene description condition).

These variations in Winograd pair fixation heat map correlations across conditions cannot be accounted for by differences in inter-observer variability (rinter-observer = 0.45 for free viewing; rinter-observer = 0.47 for scene descriptions; rinter-observer = 0.49 for object search; rinter-observer = 0.38 for counting objects condition, see Figure Supplementary 2a, dotted lines). In addition, as a control comparison, the correlations observed for a random pairing of images (shaded area in Fig. 4b) do not result in the ordering observed in Winograd image correlations across conditions.

Even though the SUM is most helpful in predicting the most frequently fixated object, it also attains comparable accuracy at predicting all human fixations to the meaning maps and DeepGaze and is better than GBVS (shuffled AUROC analysis; see methods for implementation details and see Figure Supplementary 2b for the results).

Observers look at regions/objects critical to scene understanding during free viewing
To assess where human observers fixate most frequently during free viewing, we categorized objects based on their contribution to the scene understanding map (see Fig. 3a). We categorized objects as the most relevant to scene understanding (SU-relevant) if erasing them from the scene resulted in the largest change in the participants’ scene description. Similarly, we categorized objects as irrelevant to scene understanding (SU-irrelevant) if erasing them from the scene did not result in large changes in the participants’ scene description (see methods; this analysis also excluded people in scenes, see further below for analysis including people). To control for low-level visual properties of objects and object types, we designed our stimuli so that the same objects were SU-relevant for one Winograd image (example: projector in Fig. 5a; left, clothes in Fig. 5a; right) and SU-irrelevant to the complementary Winograd image (example: clothes in Fig. 5a; left, projector in Fig. 5a; right). Across the Winograd images, SU-relevant and SU-irrelevant categories were the same set of objects, providing a strong control.

We measured the frequency of human fixations on SU-relevant and SU-irrelevant objects for each condition. We also quantified the frequency of human fixations on the top predictions of the GBVS saliency model, DeepGaze, and the most locally meaningful region based on meaning maps. Figure 5b shows the fixation frequency for each object category averaged across all images and participants. Results are shown for the four conditions. Observers fixated on objects more frequently when they were critical to scene understanding than when they were irrelevant for both the free viewing (SU-relevant vs. SU-irrelevant objects, p < 0.001, Cohen’s d = 1.41, adjusted p-values (q-value) reported for False Discovery Rate (FDR) at α = 0.05 for 28 comparisons; 20 comparisons in Fig. 5b and four comparisons in Fig. 6d and four comparisons in Fig. 8c) and scene description conditions (p < 0.001, Cohen’s d = 1.70). Furthermore, in the free viewing and scene description conditions, the SU-relevant objects were fixated significantly more often than any of the top GBVS salient (p < 0.001 for both conditions, Cohen’s d = 1.41, 1.46, respectively), DeepGaze (p = 0.005, Cohen’s d = 0.90 and p < 0.001, Cohen’s d = 0.98 respectively), or locally meaningful objects (p < 0.001 for both conditions, Cohen’s d = 1.46, 1.51, respectively). In the search condition, the searched target was fixated the most frequently (p < 0.001 for all comparisons, Cohen’s d > 1.50 for all comparisons). Still, a smaller but significant difference was observed in fixation frequency between SU object categories (higher frequency to SU-relevant vs. SU-irrelevant objects) for the object search (p = 0.048, Cohen’s d = 0.150) and the counting conditions (p = 0.029, Cohen’s d = 0.790).

To assess how observer fixation preferences develop through time, we analyzed the cumulative fixation frequencies to SU-relevant and SU-irrelevant objects for all conditions as a function of fixation number after image onset. For the free viewing and scene description conditions, the cumulative fixation frequency difference across SU-relevant and SU-irrelevant objects (ΔCFR,I) became significant after the 3rd fixation (p < 0.001, Cohen’s d = 1.24, 1.29, respectively, Figure Supplementary 3a, adjusted p-values (q-value) reported for FDR at α = 0.05 for all 40 comparisons in Figure Supplementary 3a). No statistically significant difference was observed for the search and counting objects conditions up to the 10th fixation. Figure 5c compares ΔCFR,I as a function of fixation number for all conditions. The ΔCFR,I in the scene description significantly deviated from that in free-viewing at the 6th fixation (p = 0.043, Cohen’s d = 0.43, adjusted p-values (q-value) reported for FDR at α = 0.05 for all 30 comparisons in Fig. 5c), while the free-viewing deviated significantly from the object counting condition at the 5th fixation (p = 0.049, Cohen’s d = 0.93). No significant difference in ΔCFR,I was observed between object counting and object search conditions. An analysis that used time-weighted fixations rather than fixation frequency resulted in similar findings (Figure Supplementary 3b, c).

The role of gaze, head, and hand position in directing eye movements
Many of the images in our experiment present one or more individuals directing their gaze, head, or hands to objects that are typically relevant to the scene understanding. Thus, a possible explanation for our results is that observers’ fixations on the SU-relevant locations and objects are a byproduct of the well-documented finding that observers often follow with their eyes the gaze or anticipated action of others.

To show that the SU-relevant fixations are not solely a byproduct of following gaze, head, hand direction, or body posture but rather eye movements to objects critical to scene understanding, we conducted an experiment to assess which objects observers perceived to be grasped or gazed at for each scene. Figure 6a (left) details the experiment procedure. A separate group of participants (n = 25) viewed the Winograd images with unlimited time and were asked to click the box they perceived to be grasped or gazed at by the person in the scene. Figure 6a (right) shows examples of scenes where a SU-relevant object was perceived by the majority of participants to be grasped or gazed at (top two images), and scenes where it was not (bottom two images). We found 20 images out of the 36 images for which the SU-relevant objects were not considered the object to be grasped or gazed at by the human in the image. We will refer to these scenes as the No Gaze image subset.

Analysis of the No Gaze image subset showed a similar pattern of results to our main findings (Figs. 2c, 4b, 5b). Observer fixation heatmap correlations across conditions (25 participants per image, 20 images) for the No gaze image subset were similar to the pattern of correlations for the entire image set (p = 0.03 for SD-FV vs CO-FV; p < 0.001 for SD-FV vs OS-FV) (Fig. 6b).

The observer fixation heat map correlation across Winograd images (Winograd correlation, from seven pairs in the 20 images) for the free viewing condition was no different than the scene description condition (p = 0.22, Cohen’s d = 0.56) but significantly lower than that observed for the object search (p = 0.006, Cohen’s d = 1.34) and counting objects (p = 0.037, Cohen’s d = 1.03) conditions (Fig. 6c). The Winograd correlation for the existing fixation prediction models (GBVS, meaning maps, and Deepgaze, Fig. 6c) was significantly higher than human correlations (p < 0.001 vs. free-viewing for all comparisons, Cohen’s d = 1.91, 1.70, 1.23, respectively), while the correlation for the scene understanding map (SUM) was the lowest and not different from that observed in human fixation heat maps for the free viewing and scene description conditions (p > 0.9 in both conditions, Cohen’s d = (0.20, 0.18) respectively).

In addition, similar to what we observe in the entire dataset, for the No Gaze subset, observers also execute more frequent fixations on SU-relevant objects in the free viewing and scene description conditions (p < 0.001 for both conditions, Cohen’s d = 1.26, 1.62, respectively) compared to the SU-irrelevant objects. However, unlike the entire dataset, the No Gaze subset does not show a significant difference between the fixation frequencies to the SU-relevant and SU-irrelevant object categories for the object search (p = 0.282, Cohen’s d = 0.14) and counting objects (p = 0.170, Cohen’s d = 0.61) conditions, suggesting that for those two conditions, fixations on SU-relevant objects were driven mainly by gaze cues (images for which gaze cues point to the SU-relevant objects and which were not part of the No Gaze Subset). Finally, fixation frequencies for other object categories and the cumulative fixation frequency analysis show similar results for the No Gaze subset as in the entire image dataset (see Figure Supplementary 4a, b).

To further assess whether observer eye movements are more guided by objects critical to scene understanding (SU-relevant objects) vs. objects perceived to be grasped or gazed at, we directly compared the frequency of fixations on the two categories. For the entire image dataset and the No Gaze image subset, SU-relevant objects were fixated more frequently than objects perceived to be grasped or gazed at (Fig. 6d, adjusted p-values (q-value) reported for FDR at α = 0.05 for all 24 comparisons in the No Gaze image subset, 16 comparisons in Figure Supplementary 4a) in free-viewing (p < 0.001 for both image sets, Cohen’s d = 1.19, 1.33, respectively), scene description (p < 0.001 for both image sets, Cohen’s d = 0.92, 1.20, respectively) and counting objects conditions (p = 0.009, Cohen’s d = 0.86 for the entire image set and p = 0.007, Cohen’s d = 0.99 for the No Gaze subset), but not for the object search condition (p > 0.7 for both image sets, Cohen’s d = 0.15, 0.20, respectively).

Together, all the analyses suggest that during free viewing or while describing scenes, observers most frequently direct their eyes not just to objects inferred to be grasped or gazed at but to objects critical to understanding the scene.

Causal influence on scene understanding of fixating critical objects
More frequent fixations on SU-relevant objects during free viewing and scene description conditions might imply that processing those objects with the high-resolution fovea is functionally important to maximize the understanding of the scene. We separately analyzed the scene descriptions condition trials for which observers correctly described the scenes and those for which they incorrectly described them (using a threshold based on cosine similarity of LLM embeddings of descriptions, see the methods section for details). Across all images, the difference in fixation frequency to SU-relevant vs. SU-irrelevant objects was significantly greater for trials that resulted in correct descriptions than for trials with incorrect descriptions (p = 0.004, Cohen’s d = 0.53; see Figure Supplementary 5a for examples of descriptions near the threshold used for classifying a description to be correct/incorrect and see Figure Supplementary 5b, c for the overall results and variation of adopted threshold to classify descriptions as correct/incorrect). This association between eye movements and scene descriptions has been shown in previous studies. However, the association does not necessarily imply a causal influence of fixation on SU-relevant objects on accurate scene understanding. An alternative explanation is that observers process the entire scene and extract its meaning before eye movements. In this alternative explanation, observer fixations on SU-relevant objects are a subsequent process of looking at objects critical to the already extracted scene understanding. In this interpretation, fixations on the SU-relevant objects are not required to understand the scenes accurately.

We conducted a separate experiment to evaluate the causal influence of fixations on SU-relevant objects on scene understanding. Two new groups of 15 observers maintained fixation either at the SU-relevant or the SU-irrelevant objects during a 500 ms presentation of the same test images used in experiment 1. The observers then described what was happening in the scene. The collected descriptions were compared to the gold standard description (descriptions from observers with unlimited time and eye movements) using sentence similarity measures with the embeddings of Large Language Models (LLMs; see methods for validation of Gemini similarity scores with human ratings). Figure 7a, b shows the procedural workflow of the experiment and an example description from observers fixating on the SU-relevant or SU-irrelevant object for a sample image. Figure 7c, d shows that observers’ descriptions were more similar to the gold standard description of the scene when they fixated on SU-relevant objects compared to when they fixated on SU-irrelevant objects (p < 0.001, Cohen’s d = 1.18). For reference, Fig. 7d shows an upper bound of similarity of descriptions provided by the average score across different gold standard descriptions (different observers describing the scene while exploring the image with unlimited time). A lower bound on similarity scores was estimated by permuting scene descriptions across images and comparing their similarity to the gold standard from other unmatched images (see methods. Also, see Figure Supplementary 5d for similar results obtained using embeddings from the GPT4 LLM model rather than Gemini).

Together, the results suggest that fixating on SU-relevant objects during free viewing causally leads to more accurate descriptions of the scenes than when fixating on SU-irrelevant objects.

Fixations on people and their role in the understanding of scenes
All presented analyses involved fixations on objects in the scene and excluded the fixations on people (hands, faces, bodies) to isolate the influence on fixations of SU-relevant objects, which varied across Winograd images (while the people in the scene were present in both). However, the larger literature documenting the importance of fixations on people’s heads and hands motivated us to understand how such fixations fit within the theoretical framework of eye movements to maximize scene understanding.

We first re-estimated the SUMs by collecting descriptions of the scenes that included the deletion of people or people’s hands in the scenes (in addition to the deletion of objects). Figure 8a presents a sample SUM showing that the person’s hands in the scenes were the most critical (altered the scene description the most). We found that the people were the most critical in 77.78% (28 images) of the images in the dataset, while the SU-relevant object was in the remaining.

We then repeated all analyses from previous sections, but included the fixations on people in the scenes. As with the initial analyses excluding fixations on people, we found higher correlations of observer fixation heatmaps of the free viewing with those of the scene description condition (FV-SD, r = 0.66) than with the fixation heatmaps of counting objects and object search conditions (Fig. 8b; p < 0.001 in both comparisons, Cohen’s d = 1.50, 1.70, respectively). The presence of fixations on people increased the correlations across fixation heatmaps of the Winograd image pairs for all conditions and images. Although the results remain the same as the analysis excluding fixations on people (see Figure Supplementary 6a), the fixation heat map correlation across the Winograd image pairs for the counting objects condition was no longer significantly different from that for the free viewing condition (p = 0.26, Cohen’s d = 0.45).

We also re-analyzed the eye movement data for the four conditions to assess the fixation frequency on people in the scenes (their hands, bodies, or faces; see methods). Observers fixated significantly more frequently on people for the scene description condition (Fig. 8c; p < 0.001, Cohen’s d = 1.45). Fixations on people in free viewing were significantly more frequent than those observed in the object counting and search conditions (p < 0.001 for both comparisons, Cohen’s d = 1.59, 1.65, respectively).

We calculated the cumulative fixation frequencies to understand the temporal dynamics of fixation selection on different scene components (people, SU-relevant objects, perceived to be grasped or gazed at objects, and SU-irrelevant objects). We used the No-Gaze image subset for which the perceived grasped/gazed-at objects were distinct from the SU-relevant objects (see Figure Supplementary 6b for similar results for the set of images in our dataset where the grasped/gazed at objects were the same as the SU-relevant objects). Figure 8d shows that starting with the 1st fixation, observers most frequently fixated on people (or the highest value of the scene understanding map estimated with the inclusion of people, see SU-relevant including people in Fig. 8d, adjusted p-values (q-value) reported for FDR at α = 0.05 for all 30 comparisons) for free viewing and scene description. For these conditions, early fixations (1st and 2nd) were also more frequently directed to objects perceived to be grasped or gazed at significantly above the SU-relevant object category (free viewing: p = 0.011, Cohen’s d = 1.3 for the 1st fixation; p = 0.016, Cohen’s d = 1.13 for the second fixation; scene description: p = 0.011, Cohen’s d = 1.26 for the first fixation, p = 0.003, Cohen’s d = 1.23 for the second fixation). Subsequently, the cumulative fixations on SU-relevant objects were significantly higher than the perceived grasped/gazed-at from the 6th fixation onward for free-viewing (p = 0.027 at the 6th fixation, Cohen’s d = 0.95) and the 7th fixation onward for scene description (p = 0.039 at the 7th fixation, Cohen’s d = 0.97) condition. We observed no statistically significant differences in cumulative fixations across scene components for the object counting and search conditions.

Similar to the AUROC analysis without people, AUROC analysis including fixations on people also resulted in a significant difference between SUM and GBVS saliency maps (p < 0.001, Cohen’s d = 1.87) and no statistically significant difference for DeepGaze, meaning maps, and SUMs, though SUMs were lower than meaning maps and DeepGaze (p > 0.09 for both comparisons; see Figure Supplementary 6c).

Methods
Winograd image pairs (WI)
The experiment stimuli include 20 pairs of Winograd images photographed within the University of California, Santa Barbara premises. The images include indoor, outdoor, and table scenes. Each pair was ensured to have almost the same set of objects and positions. The small visual changes across image pairs could be manipulations of an object’s position, its substitution by another object, or an actor’s posture. The pairs were split into two sets: Winograd Set 1 and Winograd Set 2. The changes greatly alter how the scene was described while aiming to preserve the lower-level saliency and meaning maps. However, some changes would be introduced in the saliency and/or meaning maps by replacing an object or a variation of its location. Each image pair was carefully curated to ensure that at least five random people described the pair differently and were consistent for a given image. Two pairs of Winograd images were removed from the dataset because their gold standard descriptions were inconsistent and vague. Out of 180 descriptions (from the other 36 images * 5 descriptions each) in our entire dataset, 149 were consistent with their corresponding image (at least three descriptions were consistent for each image). All of these images feature a person (or parts of a body) and tend to depict some future action or behavior that forms the scene’s understanding.

Participant information and informed consent
For all eye tracking experiments, the participants (260 participants in total) were undergraduate and graduate students of UCSB Psychological Brain Sciences. Participants did not know a priori about the hypothesis or the details of the experiment. Although the main study was conducted with participants ranging from 18 to 30 years, we expect the main findings to generalize to other ages. We collected self-reported ethnicity and gender data, but due to a technical issue, we have the data for only 100 out of 260 observers. We had consent to collect the gender and ethnicity data. We also collected Amazon Mechanical Turk data from US workers (276 participants in total) for our online studies, but we did not collect ethnicity or gender data from those participants. Race, gender, and ethnicity were not considered in the study design. Our main hypothesis is that eyes are directed to regions/objects that maximize scene understanding and are presumed to apply to people of all genders and ethnicities.

Participants provided written consent before participating in all experiments. This was approved by the Office of Research and Human Subjects, University of California, Santa Barbara. The authors affirm that all participants who took part as actors in the Winograd scenes provided informed consent for the publication of all images in the dataset.

Eye tracking and experimental setup
Eye movements were recorded using an EyeLink 1000+ desktop-mount eye tracker (spatial resolution: 0.01°) with a sampling rate of 500 Hz. Participants sat 75 cm away from a 19-inch monitor, so that the screens subtended a visual angle of 26.6° × 21.8° at 1,280 × 1024 pixels. The height of a stimulus image was approximately 12.7° of visual angle. Head movements were minimized using a chin and forehead rest. A velocity threshold of 22°/sec and an acceleration threshold of 4000°/sec2 were used for the detection of saccades. Eye movements were recorded from the left eye. The experiment was controlled with SR Research Experiment Builder software.

Eye movement experiment with four conditions
In the first experiment, participants (between subject design) viewed the images in four conditions: free viewing, scene description, object search, and counting objects. Each condition involved 50 participants who were asked to complete 18 trials in which they viewed one set of Winograd images from all pairs (25 participants per image). Each image was shown for two seconds and was followed by condition-specific instructions. Below, we describe the instructions given to participants for the four conditions.

Free viewing. Participants were instructed to view the scene naturally. No explicit tasks or instructions were given to them.

Scene description. Participants were instructed to describe the presented scenes. After they viewed the scene, they typed their description with no time limit.

Object search. Participants were instructed to search for an object within the scene. Before the start of the trial, they were presented with the name of the object they needed to search for. After the trial, they reported back whether the object was on the left or right side of the image. By design, the image contained the object searched at the same location across the Winograd image pair.

Counting objects. Participants were instructed to count the objects on each image’s left and right sides. After the trial, they reported which side of the image had more objects (see Fig. 1b for the experiment’s procedural flow diagram.

Fixation prediction models
The study used four fixation prediction models. The models’ fixation prediction heatmaps were compared against the fixation heatmaps collectively made by participants in each of our eye-tracking experiments.

Graph based visual saliency (GBVS). A bottom-up saliency model proposed by Harel et al. constructs a computational graph using Markov chains on top of the extracted image features to generate a heat map of possible fixation locations. Our study uses an implementation provided by Kümmerer for the GBVS model. We chose the GBVS model because it is one of the top saliency models relying purely on low-level image features to compute its saliency map, true to the original definition of saliency.

DeepGaze. DeepGaze is a neural network model trained on image features extracted from VGG-19 Convolutional Neural Network (CNN) along with human fixations on those images while free viewing. The model produces a heatmap of possible fixation locations given an image. This study uses an implementation provided by Kümmerer.

Meaning maps. Meaning maps is a crowd-sourced model developed by Henderson et al., which uses the subjective ratings provided by people on how meaningful they find local circular patches of an image. Each image was divided into overlapping circular patches, which were then randomized across all images. Different individuals then rated small portions of these patches to determine their meaningfulness. The procedure ensure each patch of images had three raters, and the overlap between the patches ensured that any region of the image had ratings from 27 raters. To facilitate the creation of meaning maps for all the images in our study, 48 Amazon Mechanical Turk participants rated approximately 300 circular patches, each measuring 3 dva in size. The final result was a map of meaningful locations in a scene as predictors of locations people might fixate on while viewing the scenes. The procedure proposed by Henderson et al. also included creating and rating 7 dva patches to compose a 7 dva meaning map and averaging it with the 3 dva meaning map. We collected 7 dva meaning maps but did not use them in our presented meaning maps. Since in our study image height was 12.7°, the 7 dva patch resulted in every patch having multiple objects that could be recognized by the raters, and making most of the image regions meaningful. Thus, the 7 dva meaning maps were uniformly meaningful, and averaging them with the 3 dva maps did not change the resulting meaning maps. Thus, our presented results show the 3 dva maps, but including the 7 dva maps did not alter the results. Refer to Fig. 3b for a procedural flowchart of all these models.

Scene understanding map
The scene understanding map visualizes an object’s contributions to understanding a scene by quantifying the change measured in participants’ descriptions of the scene after the object was removed. Creating a scene understanding map requires several steps described below: 1) Creating images with individual objects removed; 2) Collecting descriptions of images with objects removed; 3) Establishing a gold standard description for the intact image; 4) Assessing the similarity of the scene descriptions with an object removed to the scene description of the intact image; 5) Generating the scene understanding map. Below, we describe each of these steps.

Creating images with individual objects removed. Each image of the 18 Winograd pairs (36 images) was digitally manipulated to create versions with one object removed at a time. The resulting total number of images was 330. Each image had a total of 5 to 10 objects removed. Figure 3a shows an example of an image with some of its digitally manipulated versions. The photo editor app on the Samsung Galaxy S21 (version 3.4.2.43) was used to remove objects from these scenes.

Descriptions of images with objects removed. The dataset was split across the Winograd set, so participants viewed only one of the images of each Winograd pair. To ensure that participants do not see more than one version of each image, each set was divided into 11 unequal groups, each containing exactly one copy (original or manipulated) of each image. This implied that all participants provided descriptions for at most 18 images. One hundred and ten Amazon Mechanical Turk participants took part in this study. We collected five descriptions for each image (or its digitally manipulated copies) in the dataset. Before the online experiment, five images were used as a pre-test to ensure that observers followed the task instructions. Observers who did not accurately describe the pre-test images were not allowed to proceed to the main test with the 18 images.

Gold standard descriptions. To identify the impact of an object on scene understanding, we compared the descriptions of the original image to those collected for the manipulated image that had that specific object removed. We collected five descriptions for each of the original Winograd images. The gold standard description was defined as the best description among the five for each image. To establish which description was best for each image, a new group of fifty Amazon Mechanical Turk participants (twenty-five per Winograd Image set) selected the description that best described each of the original 18 Winograd pairs. Participants were randomly assigned to one of the Winograd pair sets (18 images). In each trial, the participant was shown an image with 7 descriptions (5 descriptions given to the image + two random descriptions from other images). Participants had to select the description that best described the presented scene. The two random descriptions were used to identify participants who did not follow the instructions. All data from participants who selected random descriptions were eliminated from the study. Importantly, the description with the highest vote (as the best description) was chosen for each image as the gold standard description.

Description similarity ratings to determine the contribution of objects to scene description. Eighteen Amazon Mechanical Turk participants were asked to rate the similarity of the descriptions using a scale from 1 to 10, with 10 indicating highly similar and 1 indicating very low similarity. In each trial, the participant saw the gold standard description for an image, followed by descriptions corresponding to each object being removed from the scene. Since there were five such description sets for each Winograd image, each participant had to finish 180 trials (36 images * 5 sets of descriptions). Refer to Fig. 3a for the procedural flowchart of this process.

Heatmap generation for scene understanding map. To generate a heatmap that visualizes the contribution of each object present in the scene to the scene description, we inverted the scale to have higher scores for dissimilar descriptions. Then we computed the median rating score corresponding to the description when each object was removed. The median score was assigned to all the pixels within the object’s corresponding bounding box (defined in methods; refer to Fig. 3a). To highlight relative contributions within each image, we normalized the ratings by subtracting the lowest-rated object’s rating from all the other objects’ ratings and then by dividing the map by the highest score for each image (scale-inverted normalized ratings). We followed Stoll et al. to generate the preferred fixation location in the images by using scale-inverted, normalized ratings corresponding to the removal of each object from the scene. Using the bounding boxes for our object, we modeled the ratings as 2D Gaussians centered at each object, with its amplitude corresponding to scale-inverted normalized rating and the Gaussian horizontal and vertical standard deviation derived as a fraction of the size of the bounding box (fraction = 0.29 along the x-direction and fraction = 0.34 along the y-direction). Figure 3a shows an example of a scene understanding map.

Defining SU-relevant and SU-irrelevant objects. Objects in each scene were categorized based on their scene understanding map score (SU-relevant and SU-irrelevant). Objects that belong to the SU-relevant category had the highest impact (most critical to scene understanding among objects in that image) on participants’ scene descriptions when erased. If erasing them did not impact the participants’ scene description, they would belong to the SU-irrelevant category. For each Winograd pair, the same set of objects belonged to the SU-relevant category in one image and to the SU-irrelevant category in the other image of the pair (refer to Fig. 3a). Our analyses focused on objects that were SU-relevant for one Winograd image and SU-irrelevant for the other. We did not focus on objects that had a low impact on the participants’ scene descriptions for both images of a Winograd pair.

Measuring objects to be grasped and/or gazed at in the dataset
Fifty Amazon Mechanical Turk participants were divided equally between the Winograd sets and were shown the images with objects in the scene covered with black boxes. This was done to isolate the perceived grasped or gazed at object from any contextual information that could influence participants’ judgments. Participants were asked to click on the box they perceived to be grasped or gazed at by the person in the scene. The box with the maximum number of selections was considered the object perceived to be grasped or gazed at for each image. Figure 6a shows the experimental procedure and some examples. Images, where the SU-relevant object differed from the object perceived to be grasped/gazed at, constitute the No Gaze image subset.

Forced fixation scene description experiment
Sixty participants were split into four equal groups, and each group saw 18 trials. Two groups were assigned to images from Winograd set 1, while the others were assigned to set 2. A between-subjects design across the Winograd pairs ensured that observers did not use their knowledge of one image to interpret its corresponding pair. Within each set, the two groups were asked to fixate on nine SU-relevant and nine SU-irrelevant locations. Each group saw a unique combination of image and fixation location. The images were presented for 500 ms. All trials where eye movements (saccades larger than one dva) were detected within the presentation interval were discarded. On average, the discarded trials accounted for thirteen percent of all trials, resulting in thirteen descriptions per image and fixation location. Participants described each scene after it was presented. Because of the atypical nature of the task, observers were instructed to provide their best guess of what was happening in the scene.

Repeating analyses including fixations/predictions on people in the scenes
We repeated the procedure to generate scene understanding maps (SUMs) but incorporated the influence of people in the scenes. Similarly, we also included the fixation predictions of the other models on people. We generated the measured fixation heat maps, including fixations on people. With these updated fixation/prediction heatmaps, were computed the correlations of fixation heat maps across conditions and within the Winograd pairs for each condition. We also added fixations on people and the prediction of SU-relevant with the people category in the cumulative fixation distribution plots. Finally, we performed the AUROC analysis for all fixation prediction models, including fixations/predictions on people.

Dependent variables
Heatmap generation for measured fixations. Fixations on an image (a pixel white dot on a blank canvas) were convolved with a Gaussian kernel of standard deviation 0.5° visual angle. OpenCV 2’s GaussianBlur function was used for this purpose. These convolved fixation points were added onto a blank map using their x and y location to generate the heatmap.

Selecting top predictions from fixation models. Fixation prediction models can sometimes predict contrastive regions that are not objects of importance. These regions were identified by finding the maximum locations determined by the convolution between the map and a 2 dva uniform and normalized circular patch. If the model’s prediction landed on an object, the object’s bounding box was used to represent its prediction. If not, a bounding box was generated with the average dimensions of all the bounding boxes in the image.

AUROC analysis for the fixation prediction models. The shuffled AUC (sAUC) technique was used for ROC analysis to evaluate the performance of prediction models in predicting human fixation heat maps. For each image and observer combination, the prediction maps were thresholded at different levels, and the true positives (TPs) at each level were calculated by counting the number of fixations that fell in the areas above the thresholded level. The false positives (FPs) at each threshold level were calculated by sampling fixations (the same number of fixations as the positive set) from the same observer in other images (the negative set) in the dataset, and counting the number of these fixations that fell in areas above the threshold level. The sAUC was then calculated by finding the area under the ROC curve (TPs vs FPs) for each observer-image combination. The sAUCs were averaged across all observers and images to get a final value for each map. The error bars for the sAUCs were obtained from bootstrapping the images and participants for 1000 trials.

Frequency of fixations/fixation time to object categories. Each scene had a maximum of 10 bounding boxes corresponding to different objects or regions. These bounding boxes were assigned specific identifiers or categories based on their relevance to scene understanding, as measured by the scene understanding map (SU-relevant and SU-irrelevant), the top predictions of the fixation prediction models (e.g., most salient), and condition-specific identifiers (search target from the object search condition). We counted the frequency of fixations or added the fixation times for these object/region categories across all participants. If an object category contains more than one object for a given image, we averaged fixations/fixation times for these objects.

Inter-observer correlation of fixation maps. For each condition, the 25 participants were divided into two random groups of 12 participants each. The correlation of the heatmaps for each image was computed between these two groups. This process was repeated for 1000 trials. The average correlation across all images and 1000 trials was used as the inter-observer correlation for each condition. The correlations across Winograd images were computed using 25 participants in each group. In contrast, within-observer correlation computations have half the number of participants in the analysis. We computed the Winograd correlation with half the number of participants to make a valid comparison with the inter-observer correlation. We used 1000 combinations of 12 participants (with non-repeating participants) out of the 25 and calculated the mean fixation heatmap correlation for all images for each condition. We performed a similar analysis for across-condition correlations (Fig. 2c), where we take 1000 combinations of 12 participants (with non-repeating participants) for each condition across all images to compute the mean correlations.

Computing Similarity using Large Language Models (LLMs). To compute the similarity of experiment participants’ descriptions to the gold standard (defined in the methods above), the embeddings of the large language model (Gemini) were utilized. The model API provides access to convert text into learned feature embeddings. A cosine similarity score was used to measure the similarity between these embeddings. To ensure consistency, we implemented the same analyses using the embeddings of another large language model, GPT4. We used cosine similarity to assess the agreement between the LLM similarity metric and the human ratings obtained for all comparisons in the object erasure experiment.

Computing the lower and upper bound for similarity scores. Upper bound: Each Winograd image had five gold standard descriptions (defined in the methods above). A pairwise similarity comparison of all five gold standard descriptions was computed using the embedding similarity measure. The average similarity score across all pairwise comparisons across all images constitutes the upper bound for the similarity scores.

Lower bound: The gold standard description of each image was compared to the descriptions obtained from the force fixation scene description experiment for another random image in the dataset using LLMs. The lower bound was computed by averaging the similarity scores for 1000 such trials.

Fixations executed by observers who incorrectly described the scene. We calculated the fixation frequency distribution for object categories for images with correct and those with incorrect descriptions. The classification of the descriptions as correct or incorrect was based on human judgment similarity ratings relative to the gold standard description, and similarity ratings based on the embeddings of a Large Language Model, Gemini.

For the LLM similarity measure, the participants whose description rating fell below one standard deviation (SD) from the mean rating were classified as participants who did not correctly describe the scene, and the rest were grouped as those who correctly described the scene. We also investigated how the analysis varied with different thresholds for categorizing descriptions: SD cutoffs ranging from 0 to 2.5 below the mean similarity rating for each image.

We compared the LLM similarity measures to human classification of correct and incorrect descriptions. The first author and three other research assistants (RAs) from the lab judged the correctness (binary classification: correct or incorrect) of these descriptions (based on the gold standard descriptions). A majority decision across the four raters was used to assign the final correct/incorrect classification. Ties were resolved through discussion to reach a consensus agreement.

Statistical analyses
Data analysis tools. Python (3.6 or above) and PsychoPy (2020 or above) are used to setup experiments, and other Python libraries are used to handle data (json lines, json_lines, pandas). Amazon Mechanical Turk was utilized to conduct online studies. Python libraries like numpy, pandas, and scipy were used for analyzing and testing statistical significance, and matplotlib, cv2, and seaborn were used for plotting and visualizing data and images.

Bootstrap resampling. Error bars for all our analyses were obtained using bootstrap resampling of participants, images, descriptions, and ratings. Below, we describe the general procedure. We created 100,000 resamples (with replacement) of images and participants for all the fixation frequency analyses. We used 10,000 observer/image resamples for the fixation heatmap correlation analyses because of the computational cost of calculating heatmap correlations. In the analysis that quantified inter-observer fixation heatmap correlations, we picked 10 random samples of 12 participants from the 25. For each of the 10 samples, we used only 1000 bootstrap resamples (n = 12) due to computational constraints. The Cohen’s d reported was computed either across the image or the participant-level distributions of the data, depending on the analysis.

False discovery rate. All analyses involving moderate to large sets of comparisons had their significance levels corrected for the False Discovery Rate (FDR, α = 0.05) using the Benjamini-Hochberg method, and the adjusted p-values were reported (q-value). For the cumulative fixation distributions, the comparison across SU-relevant and SU-irrelevant for each successive fixation depended on previous comparisons. Therefore, we added a simple, conservative modification to the FDR procedure, as detailed by the Benjamini-Yekutieli procedure. All significances were determined using a one-tailed significance level.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "Review for ‘The Curious Mind: Eye Movements to Maximize Scene Understanding’\nSummary and strengths\nThis paper addresses a fundamental question about human gaze behaviour: What are people doing when they are ‚free\nviewing‘, that is, looking at a scene without a specific task or instruction? This question is of particular importance as many (if\nnot most) key publications and databases on human gaze rely on free-viewing data. Previous models assume that people\nprimarily fixate on locally salient or meaningful regions or on objects which share semantic similarity with many other objects\nin the scene (‘concept maps’). The authors propose that instead, people try to understand the scene. This proposal is similar\nto Loschky’s SPECT theory about event comprehension but applies it to the free viewing of static images. Specifically, the\nauthors propose that people fixate those parts of a scene most which matter most for its global understanding.\nThe paper tests this hypothesis with a clever experimental design, which disentangles local salience and meaning from\nimportance for contextual, global understanding. Stimuli come in pairs of images, in which a given object is once crucial and\nonce irrelevant for scene understanding, because of its relevance for a depicted or inferred action (see Fig 1b of the paper as\nan example). As predicted under the authors’ hypothesis, an object is far more potent in attracting human gaze when the\ncontext renders it crucial for the global understanding of the scene. Crucially, this effect is not adequately predicted by any of\nthe previous models but can be captured with ‘scene understanding maps’. These maps are produced by iteratively\nremoving objects from a scene and quantifying the changes this removal causes in scene descriptions (either via human\nratings or LLM embeddings). The authors further show that this effect cannot be reduced to the objects being perceived to be\ngazed at or grasped by actors in the images. Finally, the authors run a separate forced fixation experiment with brief\npresentation times, showing that there is a causal connection between scene understanding and fixating the crucial object.\nFixating on the irrelevant object instead leads to erroneous descriptions of the scene.\nIn our view, there is much to like about this paper. It presents a simple but powerful hypothesis for an unsolved crucial\nquestion and presents convincing evidence across a range of predictions and experiments, enabled by a clever design.\nGiven the broad relevance of free viewing data for studies on gaze behavior in psychology, neuroscience, computer vision\nand clinical fields we think it is a highly important contribution for the broad readership of Nature Communications.\nNevertheless, we think a few improvements are possible, which we suggest below.\nSuggestions\n- How important is scene understanding for typical free-viewing behaviour? The paper convincingly shows that scene\nunderstanding is a strong driver for the type of images used in this study. The authors discuss that this may not generalise\nwell to ‘empty’ scenes not implying any past, present or future actions, which we appreciate. However, it also seems unclear\nhow strong the effect is for other images depicting people and events, which are not designed to imply a crucial role for a\nsingle object. Scene understanding maps may capture fixation behaviour which previous models fail to do, for other images\nof scenes and events as well, but we don’t yet know this. This seems a relevant and exciting question, esp. because scene\nunderstanding maps are potentially image computable with the help of multimodal transformers. We think this warrants\nfurther discussion. To be clear, we do not consider this a problem for the main conclusion of the paper. It’s a virtue of the\nstudy that the authors deliberately designed their stimuli to be maximally sensitive to the hypothesized effect and\ndisentangle it from other forms of saliency. However, it’s a relevant question for future research how important this effect is\nrelative to other baseline models for everyday scenes, not specifically chosen for this purpose.\n- A related and important point is the nature of images and differences between image pairs. We applaud the authors for\nmaking the stimuli available but encourage them to show more examples in the paper and describe them in more depth. The\ncurrent description of ‘small’ changes between images seems a bit misleading. Changes go beyond an object’s position\n_or_ actor’s posture. Typically, there is a whole array of changes to imply a different narrative, including changes which will\naffect the low-level salience and local meaning of the object (e.g. the laundry being placed more centrally or the projector\nbeing switched on in the example given). This of course is controlled for by the baseline models but seems worth pointing\nout. Also, some of the scenes seem hard to understand. The analysis focussing on misunderstood scenes used an image-\nwise deviation from the gold standard description of 1 SD as cut-off. We suspect this may mean very different levels of\n(mis)understanding for different images. It would be good to provide a feeling for this by showing some example descriptions\nand discuss the nature of the scenes in this light.\n- The discussion points out that models like DeepGaze could in theory pick up on contextual clues but may fail to do so\nbecause of the images they are trained on. Another important reason seems that these models are trained to predict fixation\ndensity maps, rather than peaks in those maps. The main analysis of the paper uses metrics that these models simply\nweren’t optimised for, and Fig S2 shows that they perform at the same level as scene understanding maps when considering\nAUCs for the whole map. This doesn’t take away from the clear evidence for the importance of scene understanding for free-\nviewing over and above what these previous models can explain. But the lacking performance of these previous models\ncould be discussed with more nuance. A related question is how the models compare in terms of AUC when not leaving out\nthe fixations attracted by people?\n- The abstract states that the ‘human brain’s default task during free viewing is to understand scenes‘. This may be a bit\npremature, given the study only tested WEIRD adults. There is a plethora of studies showing individual, cultural and\ndevelopmental differences in free-viewing. These differences may mean that not everybody’s gaze is driven by scene\nunderstanding. Alternatively, differences in gaze may reflect differences in scene understanding. The latter seems supported\nby recent findings regarding the connection between individual gaze and individual scene descriptions (Kollenda, Reher &\nde Haas, 2024 ). Either way, this point seems worth discussing and phrasing with a bit more caution.\nMinor:\n- The discussion states the prediction of future events as an important function of scene understanding. A recent paper by\nRoth et al. (2024 PsychScience) shows that gaze is attracted by perceived potential for change and seems relevant in this\ncontext.\n- The example image in Fig 1 does not seem to be part of the stim set uploaded to figshare. Was it not used in the\nexperiment?\n- Typo on page 14: ‘p ! 0.01’\n- Fig 4a: The arrangement of the maps is potentially confusing, because the reader is tempted to group them in columns,\nwhile there is no one-to-one relationship between models and experimental conditions. Maybe consider a line between the\nupper (predicted) and lower (measured) rows of maps?", "reviewer_2": "Summary: The authors investigate the fundamental goal of human eye movements during free viewing. Prior research has\nsuggested that fixations are guided by low-level saliency or local meaning. Here a new “theory” is tested: free viewing seeks\nto maximize scene understanding by directing gaze toward objects critical to scene interpretation (rather than just salient or\nlocally meaningful regions). The authors introduce Winograd image pairs, where small visual changes change the scene’s\nmeaning without changing its saliency or local meaning maps. Eye-tracking experiments across four viewing conditions\n(free viewing, scene description, object search, and counting objects) reveal that free-viewing fixations closely resemble\nthose during scene description (and are significantly different from object search or counting). Furthermore, observers fixate\nmost frequently on objects that, when removed, alter the scene’s overall interpretation—even more than objects identified as\nthe most salient or locally meaningful. A forced fixation experiment confirms that fixating on scene-critical objects enhances\ncomprehension, providing causal evidence that eye movements actively support scene understanding. The study also\ncompares multiple fixation models (saliency, DeepGaze, meaning maps, and the newly developed Scene Understanding\nMaps (SUMs)), showing that SUMs outperform previous models in predicting human fixations.\nOverall Evaluation: This paper presents an interesting contribution, but the fixation data are over-interpreted. It is\nunsurprising that scene interpretation guides eye movements, and I would hesitate to call this a theory, given that multiple\nfixations are required to bring high-resolution information to the brain—especially when relevant details are spatially\ndistributed. Moreover, in small scenes (as shown in the figures), where no fixation is needed to capture the full visual angle,\nthe free-viewing framework collapses, reducing the process to covert attention. That said, the use of Winograd image pairs is\na clever approach, effectively disentangling competing explanations and supporting the role of scene interpretation in\nfixation behavior.\nMajor concern: A major limitation of the study in relation to the proposed theory is that it removes fixations on people, even\nthough humans are often central to scene interpretation. Faces, gaze direction, and hand actions provide crucial contextual\ninformation about events, interactions, and object use, making them integral to understanding a scene’s meaning. By\nexcluding these fixations, the study risks underestimating the role of social elements in natural vision and may artificially\nsimplify how scene understanding operates. While this approach isolates object-driven fixations from social biases, it also\ncreates a paradox: if free-viewing supports scene interpretation, ignoring people—the most semantically rich elements in\nmany scenes—limits the generalizability of the findings. A more nuanced analysis could compare fixations in “people-rich”\nversus “object-rich” scenes or distinguish between fixations on people for social attention versus functional scene\nunderstanding. Additionally, understanding when in a sequence of fixations people are looked at would clarify how the\nvisual system prioritizes scene information, offering a stronger theory of scene understanding.\nRecommendation: As it stands, this paper would be better suited for a specialist journal such as Journal of Vision or Vision\nResearch rather than a broad-readership publication like Nature Communications. However, if the authors were to revise\ntheir theory of scene interpretation to explicitly incorporate social biases in fixations, the paper would become significantly\nmore compelling and more aligned with their own fixation data. This would likely require revisiting their analyses and\npotentially conducting a control experiment, but I strongly encourage the authors to pursue this approach. Rather than\nomitting what may be some of the most insightful findings, embedding social fixations into their theoretical framework would\nenhance both the impact and the generalizability of their conclusions.", "reviewer_4": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature\nCommunications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career\nResearchers who co-review manuscripts."}, "author_response": "Dear Reviewers,\nWe thank the reviewers for the thoughtful comments, which have helped us revise our paper,\nimprove its clarity, add new results, and increase its potential impact.\nIn general, there was enthusiasm about the paper's potential contributions, but concerns were\nraised. The most serious concern was the exclusion of fixations on people from the analysis\n(R2) and the lack of assessment of how eye movements to people and social cues fit within the\nframework of the theory (eye movements to maximize the understanding of scenes). Below, we\naddress this concern and then cover in detail all other items raised by R1 & R3 (joint reviewing)\nand R2.\nMajor Concern: Reviewer 2’s request to include fixations on people is arguably the most\nimportant concern. First, we’d like to explain why we excluded fixations on people’s heads and\nbodies in our main analysis. Our thinking is that if we included the frequent fixations on people\nand explained these in terms of the contribution to scene understanding, the reaction would be:\n“This is obvious! We already know that humans fixate on people in scenes the most, and that\ncould be explained by many theories, not just the observer’s goal to understand scenes.”\nThus, our analysis focused on predicting the fixations on one of multiple objects in terms of their\ncontribution to scene understanding, which is arguably more challenging than predicting that\nobservers' most frequent fixations are directed at people. Our theory can predict which of the\nobjects is fixated on the most among many objects based on their contribution to understanding\nthe scene.\nHowever, we fully acknowledge the importance of the reviewer’s point and motivation: to\nunderstand how fixations on people in the scenes fit within the current theoretical framework of\neye movement to maximize scene understanding.\nTo investigate the role of fixations on people in the scenes, we repeated all our analyses but\nincluded people in the images. The endeavor included: 1) creating new scene understanding\nmaps (SUMs) by deleting each object and also people from the scenes and collecting scene\ndescriptions to re-generate the scene understanding maps; 2) Repeating all fixation analysis of\nour data for the four conditions (free viewing, scene description, objects search, object counting)\nwith the new SUMs including people; 3) Repeating all subsequent analyses: correlation of\nfixations across conditions (free viewing vs. scene description, vs. object search, and object\ncounting); 4) Assessing the correspondence of different scene component (people, scene\nunderstanding (SU) relevant object, perceive grasped/gazed at object, etc.) with the maximum\nvalue in the SUMs and the most fixated element in the scenes for the four conditions..\nThe newly reported results (Figure 8, Supplementary Figure 6) illustrate what R2 suggested in\ntheir review: “how fixations to humans are central to understanding the scene’s meaning.”\nWe found that the re-estimated scene understanding maps show that people in the scenes were\nthe most important element (i.e., when deleted, they altered the scene description the most in\n77% of the images, even more than the SU-relevant object). Below, we show an example of\nSUM when including the deletion of people and a comparison of the SUM estimated without\nincluding the deletion of people in the scenes (see Reply Figure 1). In addition, we found that\npeople in the scenes attracted the most fixations (see Reply Figure 2). Thus, the common\nfinding of frequent fixations on people is well accommodated within the theoretical framework of\neye movements to maximize scene understanding. Based on our definition, people are most\noften the most critical element in understanding a scene and attract the most fixations. The\ncritical object is the 2nd most important element to scene understanding and is the 2nd most\nfixated element in scenes.\nIn addition, R2 wanted to see some insight into the temporal dynamics of fixation selection\nacross various people/social cues and critical objects: “Additionally, understanding when in a\nsequence of fixations people are looked at would clarify how the visual system prioritizes scene\ninformation, offering a stronger theory of scene understanding.”\nWe quantified the frequency of fixations to people, objects perceived to be grasped/gazed at,\nand objects relevant and irrelevant to scene understanding. We focused on the images for\nwhich the perceived grasped/gazed at objects were different from the SU-relevant objects. The\nfindings provide interesting insight into the temporal dynamics of fixation selection. Starting with\nthe 2nd fixation, observers fixate on people in scenes the most. Also, observers’ early fixations\n(1st and 2nd) are more frequently directed to perceived-to-be-grasped or gazed-at objects than\nobjects relevant to scene understanding. Objects critical to the scene understanding\n(SU-relevant objects) capture an increasing number of fixations as the exploration progresses\nand exceed the to-be-grasped or gazed at object after the 6th and 7th fixations, but always\nbelow the fixations to people in the scene (see Reply Figure 2). These temporal dynamics are\nsimilar for the free viewing and scene description conditions but different for the object search\nand counting objects conditions. We believe these new results increase the impact of our\ntheoretical framework and methodology (Scene Understanding Maps by deleting objects and\npeople in scenes). We thank R2 for pushing us in this direction.\nThe revised manuscript's main changes to address R2's concerns include the new results\nsection, “Fixation to people and their role in scene understanding,” and the new Figure 8.\nReply Fig 1 (part of new Figure 8 in the revised manuscript). The top left image and its corresponding\nscene understanding map (SUM, top middle (with people), and right (without people) are computed by\ndeleting each object at a time and assessing the change in the scene description by human ratings the\nsimilarity with the gold standard description. Images below show the fixation heatmaps (across all\nobservers) for each condition, including fixations on people.\nReply Fig 2 (part of new Figure 8 in the revised manuscript). Cumulative fixations per image for different\nimage components: people, scene understanding relevant objects (SU-Relevant Objects), objects\nperceived to be grasped or gazed at, and scene understanding irrelevant objects (SU-irrelevant objects).\nThe figure also shows a cumulative fixation series on “SU-Relevant including people”, which counts\nfixations to the most critical component, whether an object or people.\nBelow, we address all individual reviewer concerns. We use the notation R.X.Y to refer to the\nreviewer (X) and comment number (Y). Because of the joint review, we refer to R1 and R3 as\nR1&3.\nR.1&3.1. How important is scene understanding for typical free-viewing behaviour? The paper\nconvincingly shows that scene understanding is a strong driver for the type of images used in\nthis study. The authors discuss that this may not generalise well to ‘empty’ scenes not implying\nany past, present or future actions, which we appreciate. However, it also seems unclear how\nstrong the effect is for other images depicting people and events, which are not designed to\nimply a crucial role for a single object. Scene understanding maps may capture fixation\nbehaviour which previous models fail to do, for other images of scenes and events as well, but\nwe don’t yet know this. This seems a relevant and exciting question, esp. because scene\nunderstanding maps are potentially image computable with the help of multimodal transformers.\nWe think this warrants further discussion. To be clear, we do not consider this a problem for the\nmain conclusion of the paper. It’s a virtue of the study that the authors deliberately designed\ntheir stimuli to be maximally sensitive to the hypothesized effect and disentangle it from other\nforms of saliency. However, it’s a relevant question for future research how important this effect\nis relative to other baseline models for everyday scenes, not specifically chosen for this\npurpose.\nResponse 1&3.1: Thanks for the great summary. Indeed, the reviewers are correct about the\npossibility of using multi-model large language models (MLLMs) to compute scene\nunderstanding maps. When we started this research three years ago, image caption models\nand MLLMs were not capable of accurately describing our Winograd Images. The rapid\ndevelopment of MLLMs with image segmentation and generative AI provides the opportunity to\nautomate the generation of the Scene Understanding Map (SUM). We are actively working on\nthis. Although the automated SUM is not as accurate as using human photoshopped images,\nhuman descriptions, and ratings, we are quickly working on problems and narrowing the gap.\nR1&3.2. - A related and important point is the nature of images and differences between image\npairs. We applaud the authors for making the stimuli available but encourage them to show\nmore examples in the paper and describe them in more depth. The current description of ‘small’\nchanges between images seems a bit misleading. Changes go beyond an object’s position _or_\nactor’s posture. Typically, there is a whole array of changes to imply a different narrative,\nincluding changes which will affect the low-level salience and local meaning of the object (e.g.\nthe laundry being placed more centrally or the projector being switched on in the example\ngiven). This of course is controlled for by the baseline models but seems worth pointing\nout. Also, some of the scenes seem hard to understand. The analysis focussing on misunderstood\nscenes used an image-wise deviation from the gold standard description of 1 SD as cut-off. We\nsuspect this may mean very different levels of (mis)understanding for different images. It would\nbe good to provide a feeling for this by showing some example descriptions and discuss the\nnature of the scenes in this light.\nResponse 1&3.2: We now specify that, in some instances, it meant replacing an object with\nanother one (See Methods, pages 22-23). The reviewers are correct that we have tried to\nminimize changes in the low-level saliency and local meaning, but in some of the images, there\nmight be some small changes in the objects that we replace. We clarify this in the methods\nsection.\nSome of the scenes are more challenging, but we found that a majority of observers (82.78%)\naccurately described the scenes. We have added this information in the Methods section (page\n23).\nRegarding the analysis focusing on misunderstood scenes, we have followed the reviewers’\nrequest and added image examples in the first part of the supplementary section to give an idea\nof the descriptions below and above 1 SD (descriptions for all images just below and above 1\nSD in Mendeley). In addition, to show the generality of the results (higher fixation numbers for\nSU-relevant objects for correct vs. incorrect descriptions), the supplementary Figure 5b now\nalso shows fixation frequency using human raters' majority decision to assess whether the\ndescription is correct or incorrect. Supplementary Figure 5d also shows how our results\ngeneralize to other description similarity deviations cut-offs (0 to 2.5 SD below mean similarity\nfor each image). Together, the results show the generality of the findings and their robustness\nto various criteria and ways to define correct vs. incorrect descriptions (human and LLM\nembeddings).\nR.1&3.3. The discussion points out that models like DeepGaze could in theory pick up on\ncontextual clues but may fail to do so because of the images they are trained on. Another\nimportant reason seems that these models are trained to predict fixation density maps, rather\nthan peaks in those maps. The main analysis of the paper uses metrics that these models\nsimply weren’t optimised for, and Fig S2 shows that they perform at the same level as scene\nunderstanding maps when considering AUCs for the whole map. This doesn’t take away from\nthe clear evidence for the importance of scene understanding for free-viewing over and above\nwhat these previous models can explain. But the lacking performance of these previous models\ncould be discussed with more nuance. A related question is how the models compare in terms\nof AUC when not leaving out the fixations attracted by people?\nResponse 1&3.3:\nThe reviewer is correct. DeepGaze’s emphasis on fixation density maps might make it less\ncapable of successfully predicting the most frequently fixated object (peak of the map). We have\nnow incorporated the reviewer’s observation in the discussion section (page 21). In addition, when we included fixation on people, the AUROC analysis once again shows that the\nDeepgaze, Meaning maps, and SUMs are not significantly different from each other, although\nSUMs were lower (reported in Supplementary Figure 6c).\nR&3.4: The abstract states that the ‘human brain’s default task during free viewing is to\nunderstand scenes‘. This may be a bit premature, given the study only tested WEIRD adults.\nThere is a plethora of studies showing individual, cultural and developmental differences in\nfree-viewing. These differences may mean that not everybody’s gaze is driven by scene\nunderstanding. Alternatively, differences in gaze may reflect differences in scene understanding.\nThe latter seems supported by recent findings regarding the connection between individual gaze\nand individual scene descriptions (Kollenda, Reher & de Haas, 2024 ). Either way, this point\nseems worth discussing and phrasing with a bit more caution.\nResponse 1&3.4:\nThe reviewers bring up an important point. Indeed, individual, cultural, and developmental\ndifferences contribute to variability in eye movements during free viewing of scenes. The\nquestion is whether this variability implies that the observers are not trying to understand the\nscene. We argue that the precise components of the scene that are important and contribute to\nthe scene understanding can vary across people. For example, a soccer fan looks at a soccer\nfield scene with a sideline to the side raising a flag and describes it as an offside being called.\nThey might also fixate on the linesman or her/his flag. A person unfamiliar with soccer might\ndescribe the scene as a soccer game, and the flag or the lineman will not be critical to\nunderstanding the scene, and are less likely to be fixated. Thus, observers try to understand the\nscene, but their cultural background and individual interests shape their understanding and\nfixations. Thus, observer or culture-specific SUMs might capture these eye movements. We\nadded a sentence about individual differences (page 22) to the discussion and cited the\nsuggested reference. We also mention cross-cultural differences.\nWe also followed the reviewer’s point and toned down the abstract. The revised abstract now\nstates: “Thus, we conclude that one important default task during free viewing for humans is to\nunderstand scenes reflected by frequent eye movements to objects that maximize accurate\nunderstanding even when these objects are not to be grasped or gazed at.” If the reviewers\nwould prefer further revisions of this concluding statement, we are happy to modify it further and\nconsider suggested phrasing.\nR1&3.5: The discussion states the prediction of future events as an important function of scene\nunderstanding. A recent paper by Roth et al. (2024 PsychScience) shows that gaze is attracted\nby perceived potential for change and seems relevant in this context.\nResponse 1&3.5: Thanks for pointing to this important relevant reference. We have\nincorporated the reference on pages 21-22.\nR1&3.6: The example image in Fig 1 does not seem to be part of the stim set uploaded to\nfigshare. Was it not used in the experiment?\nResponse 1&3.6: We used this example instead of one from our data set because it is a simple\nimage with few objects. One of the objects is the most salient, the other is the most meaningful\n(glasses), and the third (the remote control) is the most critical to scene understanding. The\nimages in the experiment had many more objects, making it more challenging to convey the\nideas we were trying to communicate. We prefer to use this simpler image that makes it\nstraightforward to explain the difference between the three types of objects.\nR1&3.7: Typo on page 14: ‘p ! 0.01’\nResponse 1&3.7: Thanks for pointing this out. We have fixed it now.\nR1&3.8: Fig 4a: The arrangement of the maps is potentially confusing, because the reader is\ntempted to group them in columns, while there is no one-to-one relationship between models\nand experimental conditions. Maybe consider a line between the upper (predicted) and lower\n(measured) rows of maps?\nResponse 1&3.8: Thanks for the suggestion. We have added a line to separate the two rows.\nR2.1: Overall Evaluation: This paper presents an interesting contribution, but the fixation data\nare over-interpreted. It is unsurprising that scene interpretation guides eye movements, and I\nwould hesitate to call this a theory, given that multiple fixations are required to bring\nhigh-resolution information to the brain—especially when relevant details are spatially\ndistributed. Moreover, in small scenes (as shown in the figures), where no fixation is needed to\ncapture the full visual angle, the free-viewing framework collapses, reducing the process to\ncovert attention. That said, the use of Winograd image pairs is a clever approach, effectively\ndisentangling competing explanations and supporting the role of scene interpretation in\nfixation behavior.\nResponse 2.1:\nThanks for the feedback. We understand that the reviewer is unsurprised that scene\ninterpretation guides eye movements. Perhaps the reviewer belongs to the subfield of scene\nnarratives, and our result might seem less surprising. The paper focuses on the “free viewing”\ntask in which participants are not given any instructions. Factors influencing eye movements\nduring free viewing have been debated for the past two decades, ranging from low-level visual\nfeatures like saliency to high-level semantic information like objects/ local meaning (Meaning\nmaps). Though our results make intuitive sense that people are trying to understand the scene,\nit has been a harder feat to prove this is true because low-level image features and high-level\nsemantic information are often highly correlated. It has been hard to explain what influences\nfixations concretely. By decoupling low-level saliency from high-level semantic information using\nthe Winograd images, we were able to concretely show that scene understanding does\ninfluence eye movements even while freely viewing.\nWe also believe that our conceptual framework and methodology are novel. In our view, the new\nframework merits being called a theory (compared to all the theories proposed in cognitive and\nperceptual psychology, but this is, of course, up for debate and opinion). The idea of measuring\nthe contribution of objects and people in the image to scene understanding by assessing the\nimpact of their removal on scene descriptions is new. The prediction that the most fixations will\nbe directed to the object/person that is most critical to scene understanding is, to our\nknowledge, also new. Empirical evidence showing the causal influence of these fixations on\nscene understanding has not been reported previously.\nR2.2. Major concern: A major limitation of the study in relation to the proposed theory is that it\nremoves fixations on people, even though humans are often central to scene interpretation.\nFaces, gaze direction, and hand actions provide crucial contextual information about events,\ninteractions, and object use, making them integral to understanding a scene’s meaning. By\nexcluding these fixations, the study risks underestimating the role of social elements in natural\nvision and may artificially simplify how scene understanding operates. While this approach\nisolates object-driven fixations from social biases, it also creates a paradox: if free-viewing\nsupports scene interpretation, ignoring people—the most semantically rich elements in many\nscenes—limits the generalizability of the findings. A more nuanced analysis could compare\nfixations in “people-rich” versus “object-rich” scenes or distinguish between fixations on people\nfor social attention versus functional scene understanding. Additionally, understanding when in a\nsequence of fixations people are looked at would clarify how the visual system prioritizes scene\ninformation, offering a stronger theory of scene understanding.\nResponse 2.2:\nPlease see the Major Concern reply above, in which we address this important point. Our\napproach used the dataset that we had and directly compared fixation on people and objects.\nWe also showed how incorporating people into our analysis shows that there is no paradox (see\nR2’s comment above). As R2 predicted, people are the most important element in\nunderstanding a scene. Our cumulative fixation frequency analysis provides insight into the\ntemporal dynamics of fixation selection to the different components of the scenes, including\npeople, social cues (objects perceived to be grasped or gazed at), and scene understanding\nrelevant objects.\nR.2.3. Recommendation: As it stands, this paper would be better suited for a specialist journal\nsuch as Journal of Vision or Vision Research rather than a broad-readership publication like\nNature Communications. However, if the authors were to revise their theory of scene\ninterpretation to explicitly incorporate social biases in fixations, the paper would become\nsignificantly more compelling and more aligned with their own fixation data. This would likely\nrequire revisiting their analyses and potentially conducting a control experiment, but I strongly\nencourage the authors to pursue this approach. Rather than omitting what may be some of the\nmost insightful findings, embedding social fixations into their theoretical framework would\nenhance both the impact and the generalizability of their conclusions.\nResponse 2.3:\nWe believe our paper's theoretical framework, conceptual change in thinking, experimental rigor,\nand innovative methodology for computing scene understanding maps merit publication in a\nbroad readership journal. As a fair comparison, the meaning maps paper (Hayes and\nHenderson, 2017) that preceded our work and addressed similar questions was published in\nNature Human Behavior.\nHaving said this, we fully recognize the importance of the reviewer’s point that extending the\nwork to address the role of fixations to people and social cues would greatly increase the\nimpact. We hope to have provided a useful expansion of the analysis to show how fixations to\npeople work well within the framework of scene understanding maps and present insight into the\ntemporal dynamics of fixation selection to people, social cues, and critical objects. We\nappreciate R2's comments, which pushed us to do additional analysis and integrate fixation with\npeople in our theoretical framework. The new data provides a broader understanding of how all\nthe components fit together. If there are any additional references that we are missing and that\nR2 deems appropriate, we are happy to include them.\nSincerely,\nMiguel P. Eckstein, PhD\nDepartment of Psychological & Brain Sciences\nUniversity of California, Santa Barbara\nShravan Murlidaran, PhD candidate\nDepartment of Psychological & Brain Sciences\nUniversity of California, Santa Barbara"}, {"round": 2, "reviewer_comments": {"reviewer_1": "We congratulate the authors to the revised version of what we consider a ground-breaking contribution.\nEven though this point was raised by reviewer 2, we particularly appreciate the analysis including people. In our view, this\nstrengthens and focusses the message of the paper: While people tend to be highly salient in general (and this is a good\nheuristic for gaze prediction), the authors provide compelling evidence for a novel explanation as to _why_ this is the case.\nCrucially, this extents to the decisive and much more challenging test case of scenarios in which inanimate objects compete\nfor attention.\nIt is important to re-iterate that after decades of salience models for static scenes, this is the first compelling evidence we\nknow that human gaze is guided by context-dependent understanding of global meaning. It also provides a convincing\nanswer to _the_ notorious question in the field: Which task are people solving when free-viewing.\nThe revisions addressed all our points well. Three remaining minor suggestions are:\n- The authors may consider adding a concluding paragraph at the end of the discussion which summarises the main\nmessage.\n- While we understand the reasoning behind using a conceptual, simple example image for Fig 1a, we think it would be\ngood to point out in the figure legend that this example wasn’t actually used in the experiments.\n- The example images in Figure S5 are small and don’t have enough resolution to provide useful information when zooming\nin. It would be good to replace them with larger versions with higher resolution.", "reviewer_2": "Overall Evaluation: The authors have substantively addressed my main concern about excluding people from the analysis.\nThey rebuilt the Scene Understanding Maps (SUMs) with people deletions included, re-ran all fixation analyses across\ntasks, added a dedicated results section (“Fixation to people and their role in scene understanding”) and new Figure 8. In the\nrevised results, as I expected, people are the most critical element for understanding the scene in ~78% (28/36) of images\nand attract the most fixations; SU-relevant objects are typically second. These additions bring the theory into alignment with\nthe known prominence of people in natural vision and with the reported fixation data.\nCoherence of the revised theory (people vs. objects): The framework now treats fixations as directed to whichever\ncomponent—person or object—maximizes scene understanding, which coherently explains both frequent fixations on\npeople and systematic fixations on SU-relevant objects. Temporal analyses show that early fixations (1st–2nd) are often\ndrawn to grasp/gaze-cued objects, but SU-relevant objects overtake them by the 6th–7th fixation, while fixations to people\nremain highest throughout free viewing and scene description. Task differences are sensible (patterns diverge for object\nsearch/counting). Overall, this integration is theoretically consistent and empirically supported.\nMain comments:\n(1) The Introduction (and to some extend the Abstract) are still very much object-centric. This matters because the results\nshow people are the most critical elements in ~78% of images and attract the most fixations–central to the paper’s message.\nThe Introduction should be rewritten with the prevalence on people signposted (and not discovered late in the\nIntroduction/deep in the Results section). In addition, clarify the unified prediction in the Introduction/Discussion:\nfixations maximize scene understanding across components, with people treated as just another (often dominant) component—tying\nthis to the new Figure 8 results.\n(2) When people are included, AUROC differences flatten (SUM ≈ DeepGaze ≈ Meaning Maps; SUM lower, ns). Please\nsurface this prominently and nuance claims of superiority.\n(3) The expanded analyses are still post-hoc on the same Winograd-style dataset. A short paragraph in Discussion should\nacknowledge limits for broader, people-rich everyday scenes and state predictions for a more heterogeneous corpus.", "reviewer_4": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature\nCommunications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career\nResearchers who co-review manuscripts."}, "author_response": "Dear Reviewers,\nWe thank the reviewers for their thoughtful new comments, which have helped us further\nimprove our revised paper. Below, we address the remaining individual reviewer concerns and\nsuggestions. We use the notation R.X.Y to refer to the reviewer (X) and comment number (Y).\nBecause of the joint review, we refer to R1 and R3 as R1&3.\nR.1&3.1. The authors may consider adding a concluding paragraph at the end of the discussion\nwhich summarises the main message.\nResponse 1&3.1: Thank you for the suggestion. We have now added a concluding statement at\nthe end of the discussion. The conclusion reads (see page 23):\n“To conclude, our findings suggest that during free viewing of scenes, eye movements are\ndirected to people and objects that are critical to understanding the scene, rather than to regions\nvisually salient or judged to be meaningful. The eye movement fixations serve a perceptual\nfunction, as they causally improve the accuracy of scene descriptions. The theory of eye\nmovements to maximize scene understanding and its empirical implementation through scene\nunderstanding maps provide a unified account of free viewing of scenes. They predict which\nobjects are fixated on frequently, explain the higher frequency of fixations on people over\nobjects, and frequent fixations on objects that are perceived to be gazed at or to be grasped.\nTogether, our findings suggest that an important default task for the human brain during free\nviewing is comprehending the visual world.”\nR1&3.2. While we understand the reasoning behind using a conceptual, simple example image\nfor Fig 1a, we think it would be good to point out in the figure legend that this example wasn’t\nactually used in the experiments.\nResponse 1&3.2: Thank you for the suggestion. We have added a line stating that the image is\nnot part of our study and is solely used for motivating our study (In the caption of Fig. 1a).\nR.1&3.3. The example images in Figure S5 are small and don’t have enough resolution to\nprovide useful information when zooming in. It would be good to replace them with larger\nversions with higher resolution.\nResponse 1&3.3: Thank you for the suggestion. We have resized the images.\nR2.1: The Introduction (and to some extend the Abstract) are still very much object-centric. This\nmatters because the results show people are the most critical elements in ~78% of images and\nattract the most fixations–central to the paper’s message. The Introduction should be rewritten\nwith the prevalence on people signposted (and not discovered late in the Introduction/deep in\nthe Results section). In addition, clarify the unified prediction in the Introduction/Discussion:\nfixations maximize scene understanding across components, with people treated as just another\n(often dominant) component—tying this to the new Figure 8 results.\nResponse 2.1: After re-reading the paper, the reviewer has an excellent point. We have\nexpanded the theory so that it refers to people and objects. We then disclose earlier in the\npaper that we will focus first on objects and then address the well-documented result of frequent\nfixations on people. We now first discuss this on page 4 (rather than the last part of the\nintroduction). We have also modified the abstract to reflect that the theory predicts fixations on\npeople or objects that are critical to understanding the scene. We also modified the discussion\nto reflect that the theory proposes that eye movements are directed to people and objects (page\n19, first part of the Discussion Section).\nR2.2. When people are included, AUROC differences flatten (SUM ≈ DeepGaze ≈ Meaning\nMaps; SUM lower, ns). Please surface this prominently and nuance claims of superiority.\nResponse 2.2: Thank you for the suggestion. We have added in the discussion section, where\nwe discuss Deep Gaze and Meaning Maps, the null result when people are included in the\nanalysis (pages 20 and 21). Because of this addition, we altered the order of the following two\nsections: The section titled “The role of fixations on people and social cues in the theoretical\nframework of scene understanding” and the section on Saliency, Deep Gaze and Meaning Maps\n(Page 20).\nR.2.3. The expanded analyses are still post-hoc on the same Winograd-style dataset. A short\nparagraph in Discussion should acknowledge limits for broader, people-rich everyday scenes\nand state predictions for a more heterogeneous corpus.\nResponse 2.3: The section “Generalization to other image types and influence of individual\nand cultural differences” discusses the limitations of the Winograd-style image and how results\nmight differ for images that do not contain people or that do not imply past or intended actions.\nWe have now added two sentences discussing people-rich scenes. Although we are not certain\nabout the results with people-rich scenes, our thinking is that the scene understanding maps\nmight be good at identifying which people are critical to the scene understanding and thus are\nfrequently fixated on, and which people in the scene are less frequently fixated because they\nare irrelevant to the comprehension of the scene. Of course, this prediction needs to be tested.\nWe have incorporated this sentence on pages 21 and 22.\nWe thank the reviewers again. The review process for this paper has been a perfect illustration\nof how the peer-review process can improve the quality and impact of a paper."}, {"round": 3, "reviewer_comments": {"reviewer_2": "I thank the authors for addressing my final comments. The paper is better balanced as a result. The paper will be a\nworthwhile contribution to the literature on scene understanding."}, "author_response": ""}]

Decision letter