Reviewerly/exHarmony huggingface dataset:
COCO-CursorSearch: A large-scale cursor movement dataset approximating eye movement in visual search

Eye movements are widely used as an overt measure of attention in visual search, and recent work introduced COCO-Search18 to help build models to predict search fixation and scanpath. But accurate eye-movement recording still requires an eye-tracker, limiting the creation of far larger datasets. Here we introduce COCO-CursorSearch, a dataset paralleling COCO-Search18 (same images and targets) except that search behavior was obtained using a cursor-contingent moving window instead of measuring eye movements. A high-resolution central window (3° in radius) was repositioned over an otherwise blurred version of the search image, with target-present determination depending on an alignment of the cursor “fovea” with the target. Our goal is to identify patterns of cursor-search movements that correlate highly with eye movements during search, thereby validating the cursor movement data collected online. We compared COCO-CursorSearch to COCO-Search18 using both spatial and spatial-temporal measures. For target-present trials, the cursor-search density map accounted for 94% of the inter-observer-consistency (IOC) from the eye fixation-density map in Correlation Coefficient (CC) and 84% in Normalized Scanpath Saliency (NSS). Moreover, cursor-search scanpath accounted for more than 88% of the eye scanpath IOC in seven scanpath comparison metrics. For target-absent trials, the cursor-search density map accounted for 85% of the eye fixation IOC in CC and 80% in NSS. The cursor-search scanpath also accounted for around or more than 80% eye scanpath IOC on five scanpath metrics. For both target-present and target-absent conditions, correlations between cursor search and eye search were significant on all the applicable metrics across both target categories and trials (all ps < .05). We conclude that cursor search movement closely approximates eye-fixation search movement in multiple measures, thereby validating using a cursor to collect visual search data. The results suggest that both measures capture the fundamental constraint of aligning a central “fovea” with a search target.
Recovering population-level model frequency for recognition memory models

Estimating and comparing signal detection models to account for empirical receiving operator characteristic (ROC) curves has been a pivotal method in the studies of recognition memory. The unequal-variance signal detection (UVSD) model, following the assumptions of the standard signal detection theory (SDT), assumes that recognition memory can be captured by memory strength with an additional parameter to capture the unequal variances of the signal and the noise distribution. In contrast, the dual-process signal detection (DPSD) model combines a high-threshold component for recollection and a SDT component for familiarity-driven processes to interpret the ROC curves. The comparison between the two models in previous work has produced some inconsistent results, which could originate from two sources. First, individual-level model comparison techniques could be biased toward one of the models due to their difference in model flexibility. Second, the process of recognition memory can be a population-level random effect in nature. Specifically, there could be individual differences in whether UVSD or DPSD is the true mechanism for recognition memory in the population. The present study has thus attempted to develop a framework for estimating the frequency of each model being the “true” model in the population (i.e. model frequency) by assessing and comparing candidate methods in terms of the recovery rate of the model frequency. We found that mean posterior model probability is the most effective method for the comparison between UVSD and DPSD, while the percentage of being the best model in the individual-level model comparisons produces the highest recovery rate. We have also extended the current approach to other domains of interests (e.g., detecting shifts of the mean in Gaussian distributions). In general, we propose that population-level model frequency can provide unique and important information in model comparisons and as a result should be adopted in future studies.
Slippage Correction in Mobile Head Mounted Eye-tracking Systems

In recent years there has been a shift from desktop-based eye tracking to mobile eye tracking to facilitate experimentation in naturalistic environments. One substantial challenge of mobile eye tracking is inadvertent movement of the eye tracking camera with respect to the head, or “slippage”. Studies have shown that slippage substantially reduces gaze accuracy. Slippage often accumulates over long recording sessions, and consequently constrains the duration and type of activities that can be recorded with mobile eye tracking. One solution is to perform gaze calibration multiple times across a session. However, this is cumbersome, and slippage between calibrations can still reduce gaze accuracy. In this study, to compensate for slippage we use a deep neural network to track the eyelids across frames of the eye video and use them to estimate the gradual drift of the eye with respect to the eye camera. We track the eyelids for frames with a given eye width, under the assumption that if there is no slippage, the position of the eyelids should not change across time for a fixed eye width. We use five keypoints along these selected eyelid outlines to calculate the horizontal and vertical slippage of the eye camera with respect to the first eye image after calibration. Using rigid body transformations to account for slippage, we correct the pupil estimates before using them to estimate gaze. We tested this method on data collected for the Visual Experience Database, in which participants wore a PupilCore head-mounted eye tracker augmented with world and stereo-depth cameras. The front-heavy design of this rig commonly resulted in slippage. Our method results in reduction of gaze error by 72%, even in sessions with substantial slippage, suggesting that this can be a viable tool to correct slippage and improve gaze tracking accuracy for long recordings of naturalistic behaviors.
Human visual cortex as a texture basis set for object perception

Humans can easily and quickly identify objects, an ability thought to be supported by category-selective regions in lateral occipital cortex (LO) and ventral temporal cortex (VTC). However, prior evidence for this claim has not distinguished whether category-selective regions represent objects or simply represent complex visual features regardless of spatial arrangement, i.e. texture. If category-selective regions directly support object perception, one would expect that human performance discriminating objects from textures with scrambled object features would be predicted by the representational geometry of category-selective regions. To test this claim, we leveraged an image synthesis approach that provides independent control over the complexity and spatial arrangement of visual features. In a conventional categorization task, we indeed find that BOLD responses from category-selective regions predict human behavior. However, in a perceptual task where subjects discriminated real objects from synthesized textures containing scrambled features, visual cortical representations failed to predict human performance. Whereas human observers were highly sensitive in detecting the real object, visual cortical representations were insensitive to the spatial arrangement of features and were therefore unable to identify the real object amidst feature-matched textures. We find the same insensitivity to feature arrangement and inability to predict human performance in a model of macaque inferotemporal cortex and in Imagenet-trained deep convolutional neural networks. How then might these texture-like representations support object perception? We found that an image-specific linear transformation of visual cortical responses yielded a representation that was more selective for natural feature arrangement, demonstrating that the information necessary to support object perception is accessible, though it requires additional neural computation. Taken together, our results suggest that the role of visual cortex is not to explicitly encode a fixed set of objects but rather to provide a basis set of texture-like features that can be infinitely reconfigured to flexibly identify new object categories.
Massive visual long-term memory is largely dependent on meaning

Empirical evidence suggests a massive, perhaps even unbounded capacity of visual long-term memory (LTM), as thousands of images can be stored and subsequently recognized with high precision. However, most studies examined massive visual LTM using meaningful items, therefore, the existence of a pure visual memory that is independent of conceptual information remains to be determined. In three experiments, participants viewed hundreds of images and then completed a memory test. Three types of stimuli were used: images of real-world objects, lightly-scrambled images of the same objects, and fully-scrambled objects. The scrambled versions preserved the low-level visual statistics of the objects while reducing their semantic meaning. In the memory-recognition phase, LTM was evaluated using a 4AFC test that included an old image, a new image, and two mirror transformations of these stimuli. As expected, the results revealed a superior memory for meaningful compared to scrambled images. Importantly, there was no hint for a massive memory capacity of the scrambled items. Additionally, while participants were able to distinguish memorized items from their mirrored versions when tested with the real-world objects, memory for visual properties was much weaker and close to chance-level for the lightly-scrambled stimuli, and no different than chance-level for the fully-scrambled stimuli. Overall, these results suggest that a ‘pure’ visual LTM is quite poor and far from having a massive capacity. Rather, meaning might be critical for the long-term memorization of visual properties, as it may serve as a "conceptual hook" that enables more efficient encoding and storage of visual information.
Detection of 3-D objects in the virtual reality space

Daily objects are generally composed of many sub-objects or components. We psychophysically examined the relationship between sensitivity to the main object and the sub-objects that comprise the main object. The main object was the frame of a cube made of 12 small cylinders, and the sub-objects were 4 cylinders placed in parallel (parallel object) and 4 cylinders placed as a square (square object). Those objects are determined in the virtual reality (VR) space and had a 3-D impression. The participants observed those objects as a 2-D image on a flat display or stereoscopic VR image through a head-mounted display. The overall orientation of those objects varied from 0 to 90 degrees. Each object was embedded in many randomly placed single cylinders in the VR space. To estimate detection sensitivity toward each object, the participants were asked to detect the objects from the cylinders as quickly and precisely as possible. In the VR presentation, the participants can freely move the head and body, while in the flat display condition, a head rest was used so that the head did not move. Detection sensitivity was the highest for the main cube and the parallel objects, but only when they were vertically placed. When those objects were oriented at 45 degrees or horizontal, or for the square object of any angle, detection sensitivity was lower. Therefore, the vertical structure in the main object was found to be the key for detection. Those characteristics were prominent when the objects were presented in the 3-D VR space than in the flat display. These results indicate that the detection sensitivity of 3-D objects would depend on the vertical component of the object in the allocentric reference of frame, especially when the viewer is moving the head and body.
Human perception of navigational affordances in real-world environments relies on multiple scene properties

In daily life, we move with ease through our immediate environment: we choose different paths using a variety of navigational actions, such as walking, swimming or climbing. How do we represent navigational affordances and which visual features drive these representations? Prior work suggests that cardinal orientations in the lower visual field are diagnostic of navigational paths for walking through indoor scenes, and that Convolutional Neural Networks (CNNs) trained for scene prediction also encode these features, thus showing representations for navigational affordances (Bonner & Epstein, 2017, 2018). Here, we investigated which visual features predict a larger range of navigational actions in both indoor and outdoor environments. In an online experiment, human participants (N=152) annotated navigable paths and associated actions (e.g., walking, biking, driving) in images of real-world scenes, along with several readily identifiable scene properties (e.g., objects, materials, spatial layout, scene category). Converging results suggest that navigational affordances are derived from a combination of different scene properties which are only partially captured by current computational models. First, principal component analysis reveals three major dimensions in the navigational action space. While there are obvious diagnostic scene properties for some actions, like water for swimming/boating, there is no simple mapping from a single scene property to each dimension. Second, representational similarity analysis shows that measured scene properties only account for a small part of navigational affordance representations (R2<.21), with CNNs trained for scene recognition capturing a similar amount of variance (R2=.21). Finally, explainable AI feature visualizations show that CNN representations rely on objects and sharp contrasts rather than navigational features. Together, these results show that there is an inherent structure in navigational affordances that has not been exploited so far, and suggest that computational models specifically trained for navigational tasks may be necessary to uncover the underlying representations.
Consistency in the paintings that people remember – The impact of memorability on art

Every piece of artwork is unique, and viewing art is often seen as a subjective experience. Additionally, many works of art are made to last in people's memories and leave an impact on the viewer. However, what makes a work of art memorable? We determined and studied the memorability of 4,021 paintings at the Art Institute of Chicago (AIC) both in an online and in-person task. In an online task, over 3,200 people participated in a continuous recognition task to determine the memorability scores of the paintings. Participants were significantly consistent in the pieces they remembered and forgot, and that memory performance was predictable by a neural network (ResMem; Needell & Bainbridge, 2021), showing that an objective memorability score can be quantified for a piece of art. We identified properties that influence memorability scores: paintings with text or unusual content tended to have higher memorability scores, and paintings with the lowest memorability scores were usually scenes or landscapes with little visual information and a darker color palette. Additionally, the paintings that caused the most false alarms tended to be beige landscapes or scenes. Importantly, famous pieces were judged as significantly more memorable by ResMem, suggesting that certain perceptual features of a painting can influence its success. For the in-person task, a separate set of participants walked through and observed 162 paintings on both the first and second floors of the American Art wing of the AIC. Participants answered a mobile experiment indicating which paintings they remembered seeing, intermixed with foils. Foil images were from the same region and time period as the targets, and many were by the same artists. We found that ResMem memorability scores were also able to predict in-person memory behavior, suggesting a consistent influence of images on our memories.
Visual recognition of single body parts in natural images

Visual information about the human body is naturally organized in terms of body parts and their spatial relationships. Previous behavioral studies of body perception have largely focused on the visual interpretation of whole bodies using simplified stimuli. This leaves open the question of how well human observers can recognize individual body parts given limited spatial context and temporal constraints, and given the complex variability in appearances typical of natural images. To answer this question, we measured the accuracies with which human observers can recognize individual body parts in static natural images as a function of the amount of spatial information and stimulus duration. In an online experiment, observers were asked to identify one of eight body joints from natural images seen through small square apertures. The apertures were cropped from 340 images selected from MPII Human Pose dataset at widths that ranged from 12 to 60 pixels. These test images were presented randomly at durations that varied from 50 milliseconds to 2000 milliseconds, with or without backward masking. We found that recognition accuracy grew approximately linearly with aperture size from chance levels at 12.5% at the smallest size, to between 40 and 60%, depending on duration, at the maximum size. At the shortest duration, accuracy systematically increased from chance to 50% as a function of size. As duration increased accuracy reached over 50% at 36 pixels, leveling off to about 75% after 500 milliseconds. Additional experiments using Leeds Sports Pose Dataset largely replicated these findings. Our results suggest that human observers can use low- and mid-level cues extracted at a fairly early stage of visual processing. In addition, our data provide a useful benchmark of relatively comprehensive observation on human performance with which to compare computational models of single body part recognition in natural images.
What can 5.17 billion regression fits tell us about the representational format of the high-level human visual system?

Deep neural network models are often taken to be direct models of the hierarchical visual system; under this framework, benchmarking efforts like BrainScore (Schrimpf et al., 2018) often seek a single model with the overall best brain predictivity. However, these models also provide unprecedented experimental opportunities to systematically explore visual representation formation, by manipulating either the visual diet, the architectural inductive biases, or the format pressures induced by the task, while holding other factors constant. Here we consider targeted comparisons from >110 models and leverage the most extensive fMRI data set collected on visual system responses to date (NSD) to explore which factors give rise to more or less brain-like representational formats. The factor which showed the biggest variation in visual system brain predictivity was the task. Holding both architecture and input constant, object categorization creates a more brain-like representation relative to other tasks like autoencoding, segmentation, and depth prediction. Self-supervised tasks (e.g. SimCLR, BarlowTwins, CLIP) showed comparable or improved brain-predictivity relative to architecture-matched supervised object categorization networks. In contrast, even extremely diverse architectures (e.g. CNNs, transformers, MLP-mixers), holding constant both the input and task of object categorization, showed little to no difference in brain predictivity. Notably, the analytical method employed (e.g. RSA with or without voxel-wise feature weighting) also had a dramatic impact on brain-predictivity magnitude. Each analysis method makes implicit theoretical commitments about the linking hypotheses between artificial neurons, voxel responses, and structure of population geometry, which warrant deeper consideration. Broadly, while these results provide a current snapshot of the best-fitting models of the human ventral visual stream, here we also offer controlled model comparison as a paradigm to advance our understanding the pressures guiding visual representation formation, with the aspiration of building increasingly stable insights with every highly-performant model that arrives on the scene.
Title: Where’s Waldo?: Analyzing visual search behaviour with a web-based eye tracking system

In the current study, we tested the temporal and spatial properties of a new web-based eye tracking program called Gazer, developed at the University of Victoria. Using the Gazer program, we tracked the eye movements of participants during a Where's Waldo visual search task. On each trial, participants were presented with a Where’s Waldo scene and asked to identify the location of the Waldo target. Once they located Waldo, participants terminated the search by pressing the space bar. They were then shown a grid of 3 x 3 squares and asked to indicate the corresponding square in which Waldo appeared. During the visual search, gaze locations were recorded at a sampling rate of 30 Hz and timing was locked to the space bar response. To correlate gaze time with search behaviour, we calculated dwell time as the proportion of gaze spent on the on-target square versus time spent on the off-target squares. Dwell time was analyzed 1000ms prior to response and our results showed participants spent significantly more time looking at the on-target square than off-target squares (p < .001). At a finer grain of temporal and spatial analysis, we computed the average Euclidean distance between the Gazer coordinates and the Waldo target in 20ms time bins. This analysis revealed that participants began to move their gaze toward Waldo’s position approximately 3000ms before their space bar response. At the level of individual differences, we found a robust negative correlation between visual search speed and accuracy where participants who were very fast at finding the Waldo target were also very accurate. In conclusion, we believe that the Gazer program is a viable system for conducting remote eye tracking research. Our findings indicate Gazer has sufficient temporal and spatial resolution to investigate the relationship between eye movements and visual search performance.