Reviewerly/exHarmony huggingface dataset:
Meta-awareness of anisotropic processing of visual information across the visual field.

Introduction: During the search, our ability to find the target depends on the detectability of the target relative to our fixation point (retinal eccentricity). While much research to date focuses on the bottom-up versus top-down visual processing dichotomy in search, little work has focused on the observer's meta-awareness concerning their ability to detect the target at different retinal eccentricities. This meta-awareness of peripheral detectability can mediate an observers' beliefs of the accuracy gains from eye movement exploration during search and influence termination of search (e.g., Lago et al., 2021). We investigated the observers' perceived detectability of small targets in the visual periphery. We assessed whether the observers are aware of vertical/horizontal anisotropies in target detectability (higher detectability in the horizontal direction vs. vertical eccentricities). Methods: Four observers estimated their ability to detect a target in the visual periphery along the four cardinal directions. The estimation task required them to maintain fixation on a small target placed at the center of the screen embedded in 1/f2.8 background noise while simultaneously moving a fiduciary mark radially in one of the four cardinal directions. They placed the fiduciary mark at a retinal eccentricity that the observer deemed would result in one of five percent correct detection (50%, 60%,70%, 80%, 90%, randomized across trials) in a yes/no detection task at that given eccentricity. Results: A two-way repeated-measure ANOVA demonstrated a significant interaction between direction and proportion correct threshold. We found the left and the right retinal eccentricity estimates further in the visual periphery relative to the upward and downward estimates at all proportion correct thresholds except for 90% (p < .05, FDR correction). Conclusion: observers are aware of their ability to detect a small target in the left and right peripheral visual fields better than in the upper and lower visual fields.
Scanpath prediction in dynamic real-world scenes based on object-based selection

Humans actively shift their gaze when viewing dynamic real-world scenes. While there is a long-standing interest in understanding this behavior, the complexity of natural scenes makes it difficult to analyze experimentally. During free viewing, it has long been thought that the targets of eye movements are selected based on bottom-up saliency, but evidence accumulates that objects play an important role in the selection process. Here, we use a computational scanpath prediction framework to systematically compare predictions of models that incorporate combinations of object and saliency information, to human eye-tracking data. We model saccades as sequential decision processes between potential targets. To investigate the relevance of object-based selection, we compare an object-based model in which saccades target semantic objects, with a location-based model in which saccades target individual pixel values. Target selection in both models depends on potential targets’ eccentricity, the previous scanpath history, and target relevance. Target relevance is implemented either based on the distance to the center (center bias), on saliency based on low-level features, or high-level saliency as predicted by a deep neural network. We optimize each model’s parameters with evolutionary algorithms and fit them to reproduce the saccade amplitude and fixation duration distributions of free-viewing eye-tracking data on videos of the VidCom dataset. We assess model performance with respect to spatial and temporal fixation behavior, including the proportion of fixations exploring the background, as well as detecting, inspecting, and revisiting objects. Human data were best predicted by the object-based model with low-level saliency, followed by the location-based model with high-level saliency and the object-based model combined with a center bias. The location-based model with low-level saliency or center bias mainly explores the background. These results support the view that object-level attentional units play an important role in human exploration behavior, while saliency helps to prioritize between objects.
“Attentional Fingerprints”: Real-world scene semantics capture individuating signatures in gaze behavior

As we look around an environment, we actively select which semantic information to attend and which to ignore. How systematic are these individual differences? In other words, does an individual’s pattern of semantic attention in one environment reliably and uniquely predict their attention in a new environment? Here, we tested whether “attentional fingerprints” exist in naturalistic visual behavior. Participants’ (n = 16) gaze was monitored while they actively explored real-world photospheres (n = 60) in VR. To model scene semantics, we introduced a novel approach combining human judgments and computational language modeling to capture affordance-based inferences available to first-person viewers. Specifically, we decomposed each photosphere into tiles and obtained a written description of each tile (MTurk participants) containing both label (a door) and affordance-based (could be opened) content. Each description was transformed using a context-sensitive NLP model (BERT) into a sentence-level semantic embedding. For each participant, we used a mixed regression model built on n-1 trials (gaze~semantics) to iteratively predict gaze in the left-out trial. We correlated participants’ predicted and actual gaze and tested whether within-subject correlation based on a participant’s own semantic model was higher, on average, than predictions made by all other participants’ semantic models (own-other difference, OOD). We find that within-subject models accurately predict gaze on left-out photospheres (r = 0.33, p < 0.001); crucially, within-subject models are also individuating (OOD scores, p < 0.001). Interestingly, our ability to individuate gaze does not simply rely on modeling large numbers of semantic labels. We find that OOD is greater when verbal descriptions contain label + affordance descriptions, relative to label-only descriptions (p = 0.039). Together, our results reveal “attentional fingerprints” in real-world visual behavior and highlight the potential for inferring individual differences in higher-order cognitive processes (action planning, inferential reasoning) and psychiatric traits (autism, anxiety) from gaze alone.
Ideal Searcher with Inter-Saccade Response Correlations

Introduction: The Bayesian ideal searcher (IS; Najemnik & Geisler, 2005) is an essential normative model to understand how humans make eye movements during search. The IS uses knowledge of variations of target detectability with retinal eccentricity, and calculates fixation locations that maximize target detection accuracy. The IS formulation assumes that the internal responses are independent across fixations which occurs with temporally changing image noise or dominant dynamic internal noise. When searching static images, human responses across fixations are often partially correlated. In addition, image-computable foveated search models (Akbas & Eckstein, 2017, Lago el al., 2020) result in response correlations across fixations that depend on the image spatial statistics and the saccade amplitudes. What are the optimal eye movements with inter-saccade correlations? We developed an IS that accounted for the inter-saccade correlations (IS-COR) and compared it to the standard IS. Method: We used an image-computable foveated search model and the covariance between template responses over fixations was used to predict the optimal fixations of the IS-COR. We varied the noise power spectra of the external noise (from white noise to 1/f^4) and the independent internal noise. Results: The IS-COR achieved higher localization accuracy than the IS for white noise (44.6%±2.2% vs. 22.4%±1.9%). The accuracy difference across models diminished with increasing independent internal noise. The IS-COR executed larger saccades than the IS model (mean amplitude of 5.088±0.035 degrees vs. 2.383±0.033 degrees). The mean saccade amplitudes of the IS-COR also varied with the statistical properties of the external noise, making larger saccades with low-pass noise (mean amplitude of 6.116±0.047 degrees). Conclusion: A novel ideal searcher (IS-COR) that predicts the optimal fixations with the more realistic conditions of inter-saccade correlations can be an important tool for calculating optimal eye movements for image-computable foveated search models and for understanding human eye movements.
The dynamics of scene understanding

Visual scene understanding involves processing and integration from different levels of visual tasks, including recognition of objects, actions and interactions. Here we study the dynamics of scene understanding over time. In particular, we study the time trajectory of scene interpretation, by controlling the exposure time with perceptual masking. 140 MTurk participants were instructed to provide a detailed free-recall description to 14 stimuli images portraying various interactions between animate agents (humans and pets) and other agents and objects. They were instructed to report the type of objects and agents in the image with their properties and inter-relations. For each image, subjects were assigned to one of seven exposure conditions: 50, 75, 100, 125, 200, 500 and 2000ms followed by a mask. A fixation cross at the center of the image frame appeared prior to image display. Participants had 15 minutes for task completion. Evaluation of the subjects’ responses was conducted by 4 scorers, who followed a detailed analysis protocol, which minimized subjective judgements. Preliminary results indicate consistent trends in the time evolution of scene perception: (i) human agents are reported earlier than objects and global scene description, even when objects appear at the center of fixation (e.g. ‘two men’ before ‘a park bench’); (ii) actions are reported earlier than the acted upon objects (e.g. ‘drinking’ before ‘cup’); (iii) for human agents, the number of agents is reported early, followed by age, and gender is reported on the average later (e.g. ‘two people’, before ‘two kids’, and then ‘two boys’). These findings are interesting from a modeling perspective since they do not fit the common scene understanding paradigm in computer vision, where objects are first detected and only then their inter-relations are processed. We will consider scene perception schemes that are more consistent with human dynamics of scene perception than current approaches.
An emerging landscape for the study of naturalistic visual memory

Our memories are often rich and visually vivid, sometimes even resembling their original percepts when called to mind. Yet, until recently, our methods for quantifying the visual content in memories have been unable to capture this wealth of detail, relying on simple, static stimuli, and testing memory with low-information visual recognition or verbal recall tasks. Because of this, we have been largely unable to answer fundamental questions such as what aspects of a visual event drive memory, or how the neural representations of perceived and recalled visual content compare. However, in recent years, new methods in quantifying visual memories have emerged, following the growth of naturalistic vision research more broadly. Instead of verbal recall, drawings can directly depict the visual content in memory, at a level of detail allowing us to simultaneously explore questions about object memory, spatial memory, visual-semantic interactions, and false memories. Social media is presenting new memory stimulus sets on the order of hundreds or thousands, allowing us to examine neural representations for diverse memories across years. And, the internet has also allowed us to identify surprising new phenomena in memory—such as the existence of shared visual false memories learned across people (the “Visual Mandela Effect”), or the existence of a population of individuals who lack visual recall in spite of intact perception (“aphantasia”). In this talk, I will present exciting new directions in the naturalistic study of visual memory and provide resources for those interested in pursuing their own studies of naturalistic memory.
Interpretable object dimensions in deep neural networks and their similarities to human representations

Convolutional neural networks (CNNs) have recently received a lot of attention in the vision sciences as candidate models of core visual object recognition. At the behavioral level, these models show near-human object classification performance, allow for oftentimes excellent prediction of object-related choices, and explain significant proportions of variance in object similarity judgments. Despite these parallels, CNNs continue to exhibit a performance gap in explaining object-based representations and behavior. Here we aimed at identifying what factors determine the similarities and differences between CNN and human object representations. Paralleling object similarity judgments in humans, we generated 20 million in-silico triplet odd-one-out choices on 22,248 natural object images, using the penultimate layer activations of a pretrained VGG-16 model. Next, we applied a gradient-based similarity embedding technique that yielded 57 sparse, non-negative dimensions that were hi ghly predictive of the CNN’s odd-one-out choices. These dimensions were interpretable, reflecting properties of objects that are both visual (e.g. color, shape, texture) and conceptual (e.g. high-level category, value) in nature. While recent work indicated that CNNs respond to the texture of an object rather than its shape, our results reveal robust shape-related dimensions, indicating that texture bias may not be a general representational limitation. To probe the representational content of individual dimensions, we developed a dimension prediction approach, allowing us to (1) generate optimal stimuli for individual dimensions, (2) reveal image regions for driving these dimensions, and (3) causally manipulate individual image features to identify the dimensions’ representational nature. Despite strong parallels between CNNs and humans, a one-to-one mapping of CNN dimensions to human representational dimensions revealed striking differences for a subset of images, reve aling novel image biases that limit a CNNs generalization ability. Together, this interpretability technique offers a powerful new approach for understanding the similarities and differences between representations derived from behavior and CNNs.
Machines lack the causal knowledge people use to visually interpret modified bodies

With recent developments in machine learning, machine face-processing systems have achieved extraordinary accuracy, largely built upon data-driven deep-learning models. Though promising, a critical aspect that limits both the performance and social responsibility of deployed face-processing systems is the inherent diversity of human appearance. Every human appearance reflects something unique about a person, including their heritage, identity, experiences, and visible manifestations of self-expression. Here, we compare the performance of human and artificial face-processing systems when presented with faces of those who deliberately modify their appearance. Body modification is a popular technique of human appearance alteration to signal affiliation in a group, to signal socioeconomic status, to better match local norms of beauty, and to communicate information about the personality. In Study 1, we use the sub-category of body modification from Distinctive Human Appearance Dataset to evaluate human vs. state-of-the-art face-detection models, the first step for face recognition systems, in their ability to detect faces in 137 images of people with body modifications. The results show that these face-detection algorithms, but not people, perform much poorer than their reported performances on other datasets. Because these systems lack a causal understanding of how faces can be modified and altered, it leads to misidentifications and demeaning predictions by face recognition technologies. In Study 2, a collaboration between technologists and leaders in the body modification community, we draw from interviews with members of the body modification community to contextualize the community's experiences with facial recognition technologies. In the case of the body modification community, the negative effect of the technology failures depends on the context in which face-recognition technology is deployed, e.g., in a personal electronic device or in the context of state surveillance.
The FreeMoCap Project - and - Gaze/Hand coupling during a combined three-ball juggling and balance task

We present a broad scale, long term, open science endeavor that aims to create a free, open source, low cost, research-grade full-body motion capture system alongside a research project that utilizes this tool to investigate visual motor coupling during a combined three-ball juggling and balance task. The FreeMoCap Project - The computer vision community is making tremendous advances in the field of markerless motion capture software. However, these advances often require a high floor of technical knowledge to be used effectively. This limits their utility to the scientific community and creates a near insurmountable barrier for the general population. The FreeMoCap system leverages emerging markerless motion capture software (e.g. OpenPose, MediaPipe, DeepLabCut, etc) to create a streamlined ‘one-click’ pipeline for 3D kinematic reconstruction of full-body human, animal, and robotic movement. This system works with arbitrary camera hardware and provides methods for synchronous recording of wired cameras (e.g. USB webcams) as well as the post-hoc synchronization of independent cameras (e.g. GoPros). The FreeMoCap project emphasizes ease-of-use, with the eventual goal of developing a system that will allow a 14-year-old with no technical training and no outside assistance to recreate a research-grade motion capture system for less than 100 US Dollars. Juggling/Balance Task - The FreeMoCap system was used to record a subject performing a three-ball juggling task while balancing on a ‘wobble-board’ platform. The full body (and hand) kinematic data produced by FreeMoCap was spatiotemporally calibrated with binocular gaze data recorded by a Pupil Labs mobile eye tracker in a manner analogous to methods used in Matthis, Yates, and Hayhoe (2018), Matthis, et al (PLoS Comp Bio, In Press) and Wirth and Matthis (VSS 2022). The resulting data reveals a tight coupling between gaze and the hand/juggling ball system as well as a complex relationship between the juggling and balance task.
Global information processing in feedforward deep networks

While deep neural networks are state-of-the-art models of many parts of the human visual system, here we show that they fail to process global information in a humanlike manner. First, using visual crowding as a probe into global visual information processing, we found that regardless of architecture, feedforward deep networks successfully model an elementary version of crowding, but cannot exhibit its global counterpart (“uncrowding”). It is not yet well-understood whether this limitation could be ameliorated by substantially larger and more naturalistic training conditions, or by attentional mechanisms. To investigate this, we studied models trained with the CLIP (Contrastive Language-Image Pretraining) procedure, which is a training procedure for a set of attention-based models intended for zero-shot classification of images. CLIP models are trained by self-supervised pairing of generated labels with image inputs on a composite dataset of approximately 400 million images. Due to this training procedure, CLIP models have shown to exhibit highly abstract representations, state-of-the-art performance in zero-shot classification, and to make classification errors that are more in line with the errors humans make than previous models. Despite these advances, we show, by fitting logistic regression models to the activations of layers in CLIP models, that training procedure, architectural differences, nor training dataset size can ameliorate feedforward networks’ inability to reproduce humanlike global information processing in an uncrowding task. This highlights an important aspect of visual information processing: feedforward computations alone are not enough to explain how visual information in humans is combined globally.
Facing Distortion

Face identification is a critical activity of daily living that may be impaired by blur or distortions caused by vision loss or prosopometamorphopsia. Distortions alter the spatial relationships among facial features, disrupting configural information. In this experiment, we examine how the spatial scale and amplitude of distortions affect the identification of upright faces, and, to examine the inversion effect, inverted faces. Bandpass filtered noise (Fpeak@1-32 cycles/face) was used to generate pixel shifts to distort calm and neutral faces from the IASLab database. The amplitude of distortion was varied using an adaptive staircase. 8 normally-sighted subjects were given unlimited time to identify which of 4 distorted faces matched the identity of an undistorted reference face, each presented in a 6.5°-9.8° ellipse in a 2*2 grid. Image cues were removed from each face by equalizing the luminance distribution and chrominance of the average face. There was a significant interaction between face orientation and distortion frequency (F5,35=4.5, p=0.003), where sensitivity as a function of distortion frequency monotonically increased for inverted faces but was U-shaped with a peak at 4-8 cycles/face for upright faces. A significant face inversion effect (p<0.05) was found when distortions were applied at the low and high distortion frequencies, but there was no significant inversion effect when distortions were applied at the middle critical frequencies (4-16 cycles/face). These results suggest that the face inversion effect for distortion is mediated by a narrow spatial band. The critical frequency is correlated with the distance between the eyes, consistent with a critical role for eye geometry in upright face identification.