Reviewerly/exHarmony huggingface dataset:
Interpretable mid-level encoding models of human visual cortex reveal associations between feature and semantic tuning for natural scene images

Populations of neurons in the ventral visual stream show preferential activation for specific categories, such as faces and buildings, as well as tuning for low- and mid-level visual features, such as spatial frequency, orientation, curvature, and color. Because these visual features tend to co-vary with semantic content in the statistics of natural images, one hypothesis is that the visual system utilizes the extraction of lower-level visual features as a mechanism for separating the representations of images with different high-level semantic meaning. Here, we investigate this question using a publicly available fMRI dataset in which participants (n=8) viewed a large number of naturalistic scene images (Natural Scenes Dataset; Allen et al., 2021). We constructed several voxel-wise encoding models that explicitly model sets of low- and mid-level visual features, including a Gabor model, a model of texture statistics based on a steerable pyramid representation (Portilla & Simoncelli, 2000), a contour model (Sketch Tokens; Lim, Zitnick, & Dollar, 2013), as well as a semantic model based on high-level image properties (e.g. animacy). Our encoding models were able to accurately predict held-out voxel responses in a range of early and high-level visual cortical areas, and exhibited a substantial amount of shared variance with AlexNet, a deep neural network (DNN) model that has commonly been used to model ventral stream areas. In addition, the high degree of interpretability of our model permitted us to investigate voxels’ selectivity for particular feature values, and how these feature preferences relate to the semantic information carried by each visual feature. Overall, our results suggest a framework in which the low- and mid-level feature tuning of visual cortical populations supports the separation of images according to their semantic meaning, and this separation increases with progressive stages of processing in the ventral visual stream.
Visual Working Memory Performance With Just 1 Item Predicts Nearly All of the Variance in Performance with 5 Items

What limits working memory performance when there are many items to be held in mind? Many models of working memory capacity focus on factors that are present primarily at high set sizes (e.g., interference between items; upper bounds on number of items that can be held in mind; etc). These models assume that performance is effectively ‘at ceiling’ when remembering just 1 item, and so little can be learned about working memory from such "sub-capacity" trials. Here we test this by taking an individual differences approach. We use the TCC model (Schurgin et al. 2020) and both continuous report and a specially designed 4-AFC task to measure visual working memory performance at set size 1 and set size 5, and ask how they are related. In 3 studies (all N>100) we find that performance at set size 1 explains 80-95% of the explainable variance in set size 5 performance. By contrast, a challenging mental rotation task explains <40% of the variance at set size 5, suggesting this is specific to working memory capacity. This raises important challenges for models who focus primarily on explaining high set sizes when considering the source of working memory limits (slots, interference, etc). It is most consistent with resource models, particularly those where the same fixed set of resources is given all to a single item at set size 1 or split among all 5 items at set size 5. This work also suggests that the focus on studying working memory 'capacity' only at high set sizes is likely counterproductive: the same capacity seems to be measurable at set size 1, and the largest variation between participants occurs at set size 1, and the largest drop in performance occurs in moving from set size 1 to set size 2.
Saccade Control During Gaze Following with Real-World Videos

Introduction: Gaze, head, and body postures are socially important cues to orient attention (Driver et al., 1999; Friesen et al., 2004) but have mainly been studied with static images (Azarian et al., 2017; Bayliss et al., 2004). Here, we investigate eye movements during dynamic gaze-following with videos and show a tight relationship of the observers’ eye movements with the gazer’s head velocity and peripheral gazed-target information. Methods: Twenty-five subjects observed 160 videos (duration 1.2s) of an individual directing their gaze to a target person (25% trials), a distractor person (25% trials), or an empty space (50% trials). Subjects were instructed to follow the gaze direction to detect the target person (yes/no task). Eye position was recorded using an Eyelink 1000. We used a pre-trained CNN model (Chong et al., 2020) to obtain objective estimates of each video frame's gaze direction and gaze point. Results: Saccade endpoint distance to the gazed location was lower when the target trials vs. absent or distractor trials, showing how observers integrate foveal gaze information and peripheral target information to guide saccades. Critically, we also observed that in 24% of trials, observers initiated a reverse saccade to the gazer after executing forward saccades that followed the gaze direction. Reverse saccade occurred on trials with significantly lower head velocity (0.06 deg/s vs. 0.11 deg/s) during the first 250ms of the video, t(24)= 10.9, p<0.001, compared to trials with no reverse saccade. The saccade endpoint error (relative to gazed person’s head) following the reverse saccade was significantly reduced from 3.5˚ to 2.9˚, p<0.001, showing the functional role of reverse saccades. Conclusion: The findings show how eye movements during gaze following are tightly coupled to the dynamics of the gazer’s head movement and peripheral information about likely gazed targets.
Hybrid search performance is better for target sets with greater memory strength

In hybrid search tasks, observers search the environment for multiple targets stored in memory (e.g., locating ingredients for a recipe in your pantry). Search performance improves linearly with the number of items in the search array but shows a logarithmic relationship with the number of targets in memory (Wolfe, 2012). Memory search is typically conceptualized as “searching through a mental list” (e.g., Wolfe, 2012). Here, we investigated how hybrid search performance is affected by the strength of the memory representations for target items. This is a critical issue because a strong effect of memory strength would provide an alternative view to the search-through-a-list metaphor. In each experiment, participants first memorized two sets of 16 targets and we increased the memory strength for one set by repeating items 8x (Exp. 1) or repeating and asking different questions about the items (e.g., “What is the primary use for this object?”) to encourage deeper processing (Exp. 2). After testing memory in an old/new recognition task, participants searched for the targets (50% present) in search arrays with 8 or 16 objects. Hybrid search performance was measured using inverse efficiency scores (IES = mean correct response time/proportion correct) to account for potential speed/accuracy tradeoffs. Across both experiments (overall N=80), IES scores on target absent trials were lower for the high strength memory condition than the low strength memory condition, indicating better search performance for target sets with stronger memory representations. A separate analysis revealed these effects were largely driven by differences in accuracy rather than response time. Broadly, these data contrast with the idea of a search through lists of items in memory. Instead, they suggest that differences in memory strength — and therefore differences in how accessible items are — can account for memory set size effects in hybrid search.
Learning to perceive the gist of cancer through perceptual training

Rapid extraction of global structural regularities provides us with general information about our visual environment, the so-called gist. Similarly, medical experts can detect the gist of abnormality from medical images. We investigated whether naïve observers could learn the gist of medical abnormality. Fifteen participants completed nine training sessions (=6480 trials), receiving categorical feedback (cancer present/developing or absent), where viewing time started at 2500 ms and gradually reduced towards 500 ms based on performance. They viewed unilateral mammograms of four categories: normal, obvious-abnormal, subtle-abnormal, and global signals of abnormality (mammograms with no visible lesions but from breasts contralateral to or years prior to development of cancer). Performance was measured on a 200-mammogram test-set viewed for 500 ms without feedback pre-training, post-training, and at retention (7-10 days). D’ was modulated by category (p=.001, ηp2=.41), with a trending session effect (p=.068, ηp2=.18), without interaction (p=.301, ηp2=.08). Log linear likelihood-derived AUCs show that 11 participants performed above chance after training but only 7 at retention. Sub-analysis of 9 learners (positive correlation of d’ across training) showed a significant session (p=.026, ηp2=.37) and category (p<.001, ηp2=.62) effect, without interaction (p=.924, ηp2=.03). While naïve observers showed higher performance for mammograms with obvious and subtle lesions, critically they learned to detect abnormal gist in mammograms without any localizable lesion. Interestingly, while a state-of-the-art breast cancer classifier recognised obvious and subtle lesions as malignant, it struggled with the global category, suggesting observers learned a complex global signal not easily captured by current DNNs. Gist of abnormality can be learned through perceptual/incidental learning in mammograms both with and without visible abnormalities, although there were individual differences. Gist was poorly retained, suggesting that perceptual tuning to gist is lost when not used converging with findings that radiologists’ gist performance correlates with cases reviewed in a year, not years of experience.
Adaptive gaze allocation when simultaneously manipulating and monitoring the environment

Many tasks we perform daily require multitasking. For example, we frequently perform an action while simultaneously monitoring a visual scene. Here we investigated eye-hand coordination while participants repeatedly performed a ball-drop task that involved inserting a ball in one of three slots using different grasp modes (fingertips, tweezers). We compared gaze allocation when the task was performed in isolation (single task) or while simultaneously monitoring a display of letters that randomly changed every 1.5 to 6 seconds and could only be detected with central vision (dual task). In the single task, participants typically fixated the ball during reach and shifted gaze to the slot once the ball was grasped with either fingertips or tweezers. In the dual task, gaze patterns differed between grasp modes. In many fingertip trials (48.2%), participants exclusively fixated the display (display-only). In most remaining trials (39%), participants shifted gaze from the display to the slot and back (display-slot-display). In most tweezer trials (60.7%), participants shifted gaze from the display to the ball, returning briefly to the display before shifting gaze to the slot and back to the display (display-ball-display-slot-display). Occasionally (19%), participants shifted gaze from the ball directly to the slot without returning to the display (display-ball-slot-display). Irrespective of task conditions, the onsets of ball and slot fixations were spatiotemporally coupled to action sub-goals: the time of ball grasp and slot entry. Interestingly, the chosen gaze pattern depended on the timing of the letter change. Participants were more likely to shift gaze away from the display (display-slot-display in fingertips and display-ball-slot-display in tweezer trials) when the action sub-goals occurred during a 1.5 s time window in which no second letter change could occur. These results indicate that participants exploited the statistics of the environment and adaptively allocated gaze to simultaneously support competing perceptual and motor tasks.
An empirical comparison of online and in-lab data collection using a data-driven method on Pack&amp;Go (VPixx Technologies)

In recent years, new challenges have emerged for vision scientists. Firstly, the growing awareness that most of our theories are based on samples that do not reflect diversity – namely, Western, Educated, Industrialized, Rich, and Democratic (WEIRD) samples – stresses the need to reach individuals as diverse as possible for future studies. Secondly, the COVID-19 has put great constraints on our capacity to bring participants to the lab. In reaction to these challenges, new technologies have been developed to allow researchers to collect data on the internet. These technologies, however, are often ill adapted to the experimental paradigms we have developed in the field. For instance, they are often not designed to allow modifications of stimuli as a function of participants’ responses. Moreover, they are not well adapted to the use of data-driven classification image methods. In the present study, we tested a new platform for online experiments: Pack&Go from VPixx Technologies. This platform runs Matlab/Psychtoolbox3 experiments online. We tested participants on a categorization task using Bubbles, a data-driven method allowing to reveal visual information utilization. In Phase 1, participants were tested in the lab, in three different conditions each comprising 1000 trials: 1) The experimental code was run locally; 2) The experiment was conducted on the same hardware, but the experimental code was run by Pack&Go; 3) The experiment was conducted on a different computer, and the experimental code was run by Pack&Go. In Phase 2, we tested participants that were recruited using a panel provider (Prolifics), and tested from their home using Pack&Go. In all conditions, the exact same experimental code was used, making it easy to compare the results across conditions. The pattern of findings was well replicated across conditions. The pros and cons of testing data-driven methods online will be discussed.
The multiple encoding benefit: encoding specificity does not hinder the retrieval generalizability of visual long-term memory

A robust method for enhancing visual long-term memory (VLTM) retrieval is to encode visual information over multiple opportunities, known as the multiple encoding benefit (MEB). An aspect of the MEB that may be overlooked is the potential detriments of encoding specificity. According to the encoding specificity principle, memory performance is best when the context at encoding matches that at retrieval. In a typical MEB experiment, visual information is not only encoded repeatedly in the same context but also retrieved in the same context. This raises the possibility that the MEB is contingent upon the context match between repeated encoding and retrieval. If so, the MEB may not extend to a new retrieval context, thus limiting the generalizability of memory retrieval. To examine the impact of encoding specificity on retrieval generalizability, we had participants encode a set of real-world objects presented on one of three nature scenes which served as the encoding context. Some objects were presented three times on the same scene (consistent encoding), while others were presented three times, each on a different scene (variable encoding). For each of the encoding conditions, we tested participant’s VLTM recognition by having them retrieve items in the same encoding context or in a brand-new context (a fourth scene). When comparing both encoding styles, we found that neither was more detrimental or effective for retrieval generalization. That is, regardless of consistent or variable encoding, VLTM performance was similar when items were retrieved in a new context. Furthermore, when comparing retrieval following consistent encoding, we found that performance was similar for items retrieved in the original encoding context and those retrieved in a new context. Therefore, encoding specificity in the MEB does not hinder retrieval generalizability and further demonstrates the enduring benefit of multiple encoding opportunities.
What limits visual long-term memory?

In visual working memory and visual attention, processing and/or remembering one item largely comes at the expense of all the other items (e.g., there is a ‘resource’ limit). By contrast, when we encode a new object in long-term memory, it does not seem to come directly at the expense of the other items in memory: Remembering the clothes your child is wearing today does not automatically squeeze out your memory of the food on your child’s breakfast plate. Yet at the same time, we do not perfectly or even partially remember the visual details of everything we encounter, or even everything we actively attend to and encode into long-term memory. So what determines which items survive in visual long-term memory and how precisely their visual features are remembered? In this talk, I’ll discuss several experiments detailing both perceptual constraints on visual long-term memory storage for visual features and the role of interference in limiting long-term memory for visual objects. I’ll suggest there is a rich interplay between perception and concepts in visual long-term memory, and that therefore long-term memory visual objects can be a helpful case study for vision scientists in understanding the structure of visual representations in general.
The Meaning of Learning by Young Children’s Encountering the Signs in Picturebook Play Spaces

Objectives This study was to explore the meaning of learning by young children’s encountering the signs in picturebook play spaces. Methods The free-play related with picturebook of 38 five-year-old children (21 boys and 17 girls) at A daycare center located in K city were observed for about 8 months from November 2, 2020 to July 30, 2021.. The children’s narratives accompanied in the picturebook play spaces were transcribed and analyzed in qualitative ways, of which analysis were supported by the data of interviews, the field notes and researcher diaries. Results The meaning of learning by young children’s encountering the signs of picturebook play spaces is, first, the moment of entanglement due to the connection with signs, which leads the children to keep learning through inter-wining imagination and signs, creating difference among the images, and orchestrating transformation and merging the plays. Second, the opening strength for the learning, in which the children take the lead in the learning through an endless desire for expression and through challenges and experiments in their own risk. Conclusions The children keep forming new learnings with their own unique ways, in which they attend and reflect on at the moment of encountering the signs, and they explore the entanglement of desires and challenges. This suggests that young children are empowered beings who can unlock the full power of growth and change.
Certification of photon Fock states using second-order nonlinearity

Certifying whether a photon Fock state is present or not while preserving its quantum state is an important task that proves useful in quantum technologies. The existing methods to perform such task apply to single photons only and consume costly quantum resources. In this paper, we propose a resource-efficient scheme for certification of an arbitrary (nonvacuum) photon Fock state by using second-order nonlinearity. In particular, we utilize the correlations in both photon number and polarization of a stimulated parametric downconversion process to herald the presence and the quantum state of an incoming Fock state. The proposed scheme works excellently with a near perfect certification fidelity even when using nonideal photodetectors. We then compare our scheme with the existing ones taking into account relevant figures of merit for each method. We also present applications of our scheme in overcoming transmission loss of photons in quantum channels as well as in preparation of NOON states, $(|N\ensuremath{\rangle}|0\ensuremath{\rangle}+|0\ensuremath{\rangle}|N\ensuremath{\rangle})/\sqrt{2}$. Our paper introduces to the optical toolbox a practical technique that is beneficial for quantum communication and quantum metrology.
Interdisciplinary Center on Open and Accessible Educational Resources: 10 Years Building Inclusive Education in Uruguay and Latin America

The year 2020 undoubtedly marks a turning point in the integration of digital technologies in education. The physical distancing necessary to contain the Covid-19 pandemic has highlighted the need to approach digital education from a holistic and critical perspective on its capacity for inclusion and exclusion. The Interdisciplinary Center for Open and Accessible Educational Resources (CIEDA) intends to consolidate and expand the approach to the subject of inclusive, open, and accessible digital education, in Uruguay and Latin America, implementing teaching, research, and third mission activities that provide scientific and technical knowledge for innovation, professional and academic practice, integrating diverse disciplines, producing a comprehensive approach, and giving rise to interdisciplinary and transdisciplinary collaboration. The interdisciplinary and intersectoral development in the university together with other social actors contribute to the informed development of the action of civil society and the government, in the creation of public policies, in permanent dialogue with reference centers of the region and the world. The center is an effort to consolidate the work that has been carried out since 2015 by the Interdisciplinary Group of Open and Accessible Educational Resources together with works of different stakeholders from society. Currently, the center is expanding its scope to three cross-cutting areas for the generation of open knowledge and human development: digital media and educational technologies, education and open science, ethics, and digital rights. On these transversal areas, interdisciplinary and intersectoral lines of work articulate efforts in teaching, research, and third mission, with social appropriation and transferability (products and services): skills for digital citizenship; data and artificial intelligence for education; diversity, equity, and inclusion in education; and critical digital pedagogies. The chapter presents the main developments made during a decade of work aimed at educational inclusion in Uruguay and in Latin America.
Improved Face Detection Algorithm under Mask Occlusion

Face detection technology is widely used in various fields, such as security systems and authentication systems. In the current stage of normalized epidemic prevention and control, people need to wear masks in public. A new approach is urgently needed to enable face detection when people wear masks. First, according to the public dataset Widerface and MAFA, this paper uses a Dlib-based way to obtain a self-built dataset and label them. Second, we have made the following innovations to the YOLOv3 face detection algorithm. The original Darknet-53 is replaced with ResNet50 as the backbone network, where DCN(Deformable Convolutional Network) enhances the robustness of the model to slightly deformed targets. At the same time, the FPN(Feature Pyramid Network) is replaced by BiFPN(Bidirectional Feature Network) and DIoU(Distance IoU) is introduced as a positioning loss function. Third, after changing faceNet's loss function to ArcFace, two models are trained using FaceNet: a mask-wearing face and a normal face. Experiments show that the face detection model proposed in this paper has an mAP50(mean Average Precision) of 94.68%, which is 2.53% higher than the original YOLOv3 model. Through multi-model with different weights selected by self-determination, the face recognition can maximize the use of the current facial feature points and achieve a more superior face identity recognition effect.