julilili42/papers huggingface dataset:
Using an Odd-One-Out Design Affects Consistency, Agreement and Decision Criteria in Similarity Judgement Tasks Involving Natural Images.
2022-12-05

Recently, similarity judgement tasks have been employed to estimate the perceived similarity of natural images (Hebart, Zheng, Pereira, & Baker, 2020). Such tasks typically take the form of triplet questions in which participants are presented with a reference image and two additional images and are asked to indicate which of the two is more similar to the reference. Alternatively, participants can be presented with three images and asked to indicate the odd one out. Though both questions are mathematically similar, they might affect participants' decision criteria, the agreement among observers, or the consistency of single observers—these possibilities have hitherto not been assessed. To address these issues, we presented four observers with triplets from three image sets designed to juxtapose different perceptual and conceptual features. Using a soft ordinal embedding algorithm—a machine learning version of a multidimensional scaling—we represented the images in a two-dimensional space such that the Euclidean distances between images reflected observers' choices. Agreement between observers was assessed through a leave-one-out procedure in which embeddings based on three observers served to predict the respective fourth observer's choices. Consistency was calculated as the proportion of identical choices in a repeat session. Here we show that design choices in similarity judgement tasks can indeed affect results. The odd-one-out design resulted in greater embedding accuracy, higher agreement among, and higher consistency within observers. Hence, an individual observer's choices could be better predicted in the odd-one-out than in the triplet design. However, predicting individual responses was only possible for image sets for which participants could report a predominant relationship. Otherwise, predictability dropped to close to chance level. Our results suggest that seemingly innocuous experimental variations—standard triplet versus odd-one-out—can have a strong influence on the resulting perceptual spaces. Furthermore, we note severe limitations regarding the predictive power of models relying on pooled observer data.
Electronic structure of colloidal 2H-MoS2 mono- and bilayers determined by spectroelectrochemistry
2022-12-05

We investigate the electronic structure of mono- and bilayers of colloidal 2H-MoS2 nanosheets synthesized by wet-chemistry using potential-modulated absorption spectroscopy (EMAS), differential pulse voltammetry (DPV) and electrochemical gating (ECG) measurements. We report the energetic positions of the conduction and valence band edges of the direct and indirect bandgap and observe strong bandgap renormalization effects, charge screening of the exciton as well as intrinsic n-doping of the as-synthesized material. We find two distinct transitions in the spectral regime associated with the C exciton, which overlap into a broad signal upon filling the conduction band. In contrast to the oxidation, the reduction of the nanosheets is largely reversible, enabling potential applications for reductive electrocatalysis. This work demonstrates that EMAS is a highly sensitive tool for determining the electronic structure of thin films with few nanometer thickness and that colloidal chemistry affords high-quality transition metal dichalcogenide nanosheets with an electronic structure comparable to that of exfoliated samples.
Using the DeepGaze III model to decompose spatial and dynamic contributions to fixation placement over time
2022-12-05

Humans view images in scanpaths of fixations, where they move their gaze over the image to explore interesting parts of the image. Which factors govern the principles of such scanpaths and how they change over time has been the subject of substantial research. The deep learning based DeepGaze III model currently sets the state-of-the-art in free-viewing scanpath prediction on natural images. It combines a spatial prediction module, which captures the influence of scene content on fixation placement, with a scanpath history module that captures the influence of earlier fixations and therefore the dynamics of the scanpath. Here, we conduct a series of ablation studies to train variants of DeepGaze III with no access to scene content, scanpath history or both and analyse how well fixations are predicted over the course of free-viewing scanpaths. We find that the overall predictability of fixations decays substantially over the course of scanpaths. Comparing the ablated models allows us to attribute this decay nearly completely to a decay in the influence of scene content: early fixations focus on the most salient image areas. Additionally, there is an influence of the initial fixation, which is mostly decayed after one saccade. Interestingly, the effect of scanpath history stays nearly constant over the scanpaths, suggesting that the dynamics of scene viewing don't change substantially over time. Due to the capacity of our models relative to the size of the dataset, we can assume that the models are able to capture all substantive ways in which fixations could depend on image content and previous fixations (in the context of free viewing). This avoids the possible confounder that the models are simply not good enough to predict fixations well. This study provides an example of how deep learning based models can be used to further our understanding of human behaviour.
Motion- and shape-based body-selectivity in macaque anterior inferotemporal cortex
2022-12-05

Visual analysis of body movements is an essential element of non-verbal social communication. fMRI studies have identified body patches in the macaque inferotemporal (IT) cortex that were activated more strongly by static images of bodies compared to images of faces and objects. However, the organization of “dynamic” body patches that are activated by dynamic bodies, and their feature selectivity at the single-cell level are still unknown. In this fMRI-guided electrophysiological study, we investigated the neural representations underlying the visual processing of dynamic bodies in the dynamic body patches of the macaque’s brain. We first localized the dynamic body patches using a set of 1 s long videos of dynamic monkey bodies, dynamic monkey faces, and dynamic artificial objects. We found several dynamic body patches that were neighboring face patches along and ventral to the Superior Temporal Sulcus (STS). Then, we recorded body-selective single cells, identified with the same videos as for the fMRI mapping, in an anterior dynamic body patch, ventral to the STS in two monkeys. The majority of neurons responded only during a segment of the video and showed high selectivity for different body videos. Most neurons had a similar response profile for the original videos and a version in which the body was reduced to its silhouette. To assess the contribution of motion, we also measured the response to static frames ("snapshots”) of the videos. Many neurons responded equally well to the static presentations of snapshots and the original videos, while others responded only to the videos, requiring motion. The latter cells were also selective for the order of the frames of a video. These findings suggest that the majority of body-selective cells in anterior IT are shape-selective and some of these cells, located ventral to the STS, also encode the snapshot sequence.
Eye tracking evidence for V1 Saliency Hypothesis from an anomalous visual search behavior
2022-12-05

Typically, humans find a target among uniformly oriented non-targets more quickly when this target is perpendicular, rather than parallel, to the non-targets. V1 Saliency Hypothesis (V1SH), which states that the primary visual cortex (V1) creates a saliency map to attract attention exogenously, predicts a special case in which exactly the opposite holds. Each visual item contains two equally-sized disks. Its orientation is that of the line connecting the disk centers. The target is a homo-pair of disks of the same color, black or white. Each non-target is a hetero-pair of one black and one white disk. The target is 45 degrees clockwise or counterclockwise from vertical, and parallel or perpendicular to uniformly oriented non-targets. With its on/off subfield of a receptive field activated by a white/black disk, a V1 neuron is more activated by a target/non-target parallel/perpendicular to its preferred orientation. Since a V1 neuron is more strongly inhibited by nearby V1 neurons tuned to similar orientations, it is more activated by a target parallel, rather than perpendicular, to the non-targets by being less suppressed, making the parallel target more salient according to V1SH. Our subjects had to press a button to report the homo-pair target’s location as quickly as possible. Report reaction times (RTs) were significantly shorter for targets parallel to the non-targets. Splitting each RT into components, RT_gaze and RT_lapse, for durations before and after the gaze reaching the target, this parallel advantage was only caused by RT_gaze for attentional shifts. In RT_gaze, the parallel advantage is stronger among trials in which the target-reaching saccade has a larger saccadic amplitude, consistent with the idea that saliency is stronger in peripheral vision. Our findings support the idea that the neural substrate for saliency is in V1 rather than higher brain areas which signal figure orientations of symbolic objects.
DeepGaze vs SceneWalk: what can DNNs and biological scan path models teach each other?
2022-12-05

Eye movements on natural scenes are driven by image content as well as by saccade dynamics and sequential dependencies. Recent research has seen a variety of models that aim to predict time-ordered fixation sequences, including statistical-, mechanistic-, and deep neural network (DNN) models, each with their own advantages and shortcomings. Here we show how a synthesis of different modeling frameworks may offer fresh insights into the underlying processes. Firstly, the explanatory power of biologically inspired models can help develop an understanding of mechanisms learned by DNNs. Secondly, DNN performance can be used to estimate data predictability and thereby help uncover new mechanisms. DeepGaze3 (DG3) is currently the best-performing DNN model for scan path predictions (Kümmerer & Bethge, 2020); SceneWalk (SW) is the best-performing biologically inspired dynamical model (Schwetlick et al., 2021). Both models can be fitted using maximum likelihood estimation and compute per-fixation likelihood predictions. Thus, we can analyze prediction divergence at the level of individual fixations. DG3 generally outperforms SW, indicating that the DNN is accounting for variance by learning mechanisms that are not yet included in the mechanistic SW model. Preliminary results show that SW tends to underestimate the probability of long, explorative saccades. In SW this behavior could be achieved by replacing the Gaussian attention span with a function with heavier tails or by implementing temporal attention span fluctuation. Furthermore, DG3 appears to compress previously unexplored areas, increasing likelihood for saccades to the region center. Once the region is fixated, DG3 broadens the local probability, consistent with a dualistic exploration-exploitation strategy. Adding corresponding mechanisms to SW may improve model performance and help develop more advanced dynamical models. Finding the synergies between different modeling approaches, specifically high-performing DNNs and more transparent dynamical models, is a valuable tool for improving our understanding of fixation selection during scene viewing.
The latency of reflexive saccades is influenced by contrast and correlates over the horizontal and vertical visual field plane
2022-12-05

The previous research reporting on the influence of stimulus contrast on eye movements showed decreasing latency in saccadic onset with increasing contrast of a visual stimulus, however considering only horizontally-oriented saccades. The current study targeted the latency of reflexive saccades to a Gabor patch (GP) presented at horizontal and vertical locations in three eccentricities (2°, 6.5°, 11°) to 12 participants. The selected spatial frequencies (SF) were 0.8, 1.4, 2.2, 4.3 cpd (cycles per degree), and the contrast ranged from 0.03 - 66%. The experimental procedure considered a GP appearing for maximum time of 0.5s (or disappeared as the gaze position exceeded the fixation area - 1°) randomly either above, below, left or right relatively to a central fixation mark. The monocular gaze position was captured by an infra-red eye tracker. The GP had a constant size of 1.4°, while its contrast was managed by the QUEST+ adaptive psychometric procedure, upon an outcome, whether a respective saccade towards the stimulus was detected or not. The QUEST+ was applied for each of the four tested locations individually. First the 95th percentile of the timing was sought for each location and SF individually over the three eccentricities, while applied Friedman’s test showed no significant effects of the SF and location χ2(15) = 2.1; p=0.52, χ2(15) = 2.1; p = 0.55 and χ2(15) = 1.2; p = 0.75 for each of them. However, we found exponentially decreasing latency with increasing contrast in both horizontal and vertical directions from approximately 0.48s to 0.30s (R2all > 0.74). Furthermore, the comparable trend was found for other tested percentiles as well (25%, 50%, 75%). In conclusion, we propose the possibility to adjust the presentation time of a visual stimulus accordingly to its parameters for better time performance in visual field-specific visual performance testing using reflexive saccades.
Norm-referenced neural mechanism for the recognition of facial expressions across fundamentally different face shapes
2022-12-05

Humans recognize facial expressions from highly different facial shapes, including comic figures that never have been seen before. Even though they are trained with huge numbers of pictures of faces (REF), this task is a challenge for popular deep neural network models of face recognition, while humans effortlessly accomplish such transfer. For the encoding of facial identity two different encoding mechanisms have been contrasted, norm-referenced and example-based encoding (Tsao et al., 2017, Leopold et al., 2006; Koyano et al, 2021). We demonstrate that norm-referenced encoding is suitable to account for facial expression recognition from fundamentally different head shapes without extensive training. METHODS: We propose a neural model consisting of two modules: 1) a standard deep neural network, which models the initial parts of the visual pathway and extracts a sparse set of facial landmarks that support optimally the classification of the learned facial expressions; 2) a simple neural network for norm-referenced encoding of expressions based on these landmark features. We compare this new model to a standard deep neural model with a stimulus set that presents the same 7 basic facial expressions on very different head shapes (human avatar, comic figure, etc.). RESULTS: Our model recognizes reliably the facial expressions across all tested head shapes, requiring training of the expressions only of one head shape and of the neutral expressions of all other head shapes. In contrast, a state-of-the art CNN architecture (ResNet50) trained with 400k+ facial images fail to classify expressions robustly on this data set. CONCLUSIONS: We presented a physiologically-inspired neural model for the ‘vectorized encoding’ (Beymer & Poggio, 1995) of facial expressions that accounts for transfer across very different head shapes, which is difficult to obtain with standard models.
Sensory tuning in neuronal movement commands: potential perceptual consequences
2022-12-05

We normally perceive a stable visual environment despite repetitive eye movements. To achieve such stability, visual processing integrates information across saccades, and laboratory hallmarks of such integration are robustly observed by presenting brief peri-movement probes. In one phenomenon, peri-saccadic suppression, visual sensitivity to such probes is strongly suppressed, and in another, peri-saccadic mislocalization, perceived probe locations are grossly erroneous. Both phenomena are believed to depend, at least in part, on corollary discharge associated with saccade-related neuronal movement commands. However, we found in a companion study (VSS 2022) that superior colliculus (SC) motor bursts, a known source of corollary discharge (Sommer & Wurtz, 2004), paradoxically exhibit strong sensory tuning in them. Therefore, here we investigated whether both peri-saccadic suppression and mislocalization depend on saccade-target visual properties. We asked eight humans to generate saccades to either high (5 cycles/deg) or low (0.5 cycles/deg) spatial frequency gratings. We always placed a high contrast target spot at grating center, to ensure matched saccades across image types. In one experiment (suppression), peri-saccadic probes (~12 ms duration) appeared at one of four fixed locations, which was indicated by subjects via button press (4AFC paradigm), relative to the saccade target. We varied probe contrasts across trials to calculate perceptual thresholds. In a second experiment (mislocalization), we used high contrast probes, and the subjects pointed (via mouse cursor) at their perceived locations. In both experiments, we observed significantly stronger peri-saccadic suppression and mislocalization for low spatial frequency saccade targets, consistent with our neuronal results (VSS 2022). This was despite matched saccade metrics and kinematics to trials with high spatial frequency targets. To the extent that peri-saccadic suppression and mislocalization depend on corollary discharge, our results suggest that such discharge relays more than just saccade vector properties to the visual system; saccade-target visual features can also be peri-saccadically processed.
The exception that proves the rule: Virulence gene expression at the onset ofPlasmodium falciparumblood stage infections
2022-12-05

Abstract Controlled human malaria infections (CHMI) are a valuable tool to study parasite gene expression in vivo under defined conditions. In previous studies, virulence gene expression was analyzed in samples from volunteers infected with the Plasmodium falciparum (Pf) NF54 isolate, which is of African origin. Here, we provide an in-depth investigation of parasite virulence gene expression in malaria-naïve European volunteers undergoing CHMI with the genetically distinct Pf 7G8 clone, originating in Brazil. Differential expression of var genes, encoding major virulence factors of Pf, PfEMP1s, was assessed in ex vivo parasite samples as well as in parasites from the in vitro cell bank culture that was used to generate the sporozoites (SPZ) for CHMI (Sanaria® PfSPZ Challenge (7G8)). We report broad activation of mainly B-type subtelomeric located var genes at the onset of a 7G8 blood stage infection in naïve volunteers, mirroring the NF54 expression study and suggesting that the expression of virulence-associated genes is generally reset during transmission from the mosquito to the human host. However, in 7G8 parasites, we additionally detected a continuously expressed single C-type variant, Pf7G8_040025600, that was most highly expressed in both pre-mosquito cell bank and volunteer samples, suggesting that 7G8, unlike NF54, maintains expression of some previously expressed var variants during transmission. This suggests that in a new host, the parasite may preferentially express the variants that previously allowed successful infection and transmission.
Does it fit? The impact of scene context on temporal and spatial characteristics of the object manipulation in a pick-and-place task
2022-12-05

Gaze is shown to be anticipatory to motor actions and is promising for understanding an individual’s intentions even before the action. The use of visual data in relation to the motor behavior is thus encouraging to engage an intuitive control of assistive technologies (prosthetic arm, exoskeleton) by people with impaired motor functions. Vast majority of studies investigating interaction with objects such as grasping and moving objects between different locations focused on isolated items. Yet, we daily interact with numerous objects in a rich environment. Emerging evidence demonstrates effect of scene context on human visual search behavior, but it remains unclear whether in a realistic context the eye-hand interaction remains the same. In the present study, we evaluated impact of the scene context on temporal and spatial characteristics of object manipulation in a pick-and-place task. To do so, three realistic experimental scenarios were implemented in VR: congruent (object fitted the context), incongruent (object didn’t fit the context), and isolated (no context was present). Three phases were examined: search, reach, and transport. Each of seven participants searched for the target object, picked it, and placed it on a predefined final location. Using linear mixed-effect model analysis, a significant increase of task performance duration and the scene coverage for incongruent condition was found. Furthermore, a significant effect of the scene context was found on the search phase indicated by longer search times in incongruent condition. The reach phase duration was longer for incongruent condition, but not significantly. Finally, transport phase was not affected by the scene context. Thus, in a pick-and-place task, this pilot study demonstrates impact of the scene context primarily on search behavior, but not significantly on reach and transport. The study brings insight to the development of assistive technologies for a pick-and-place task demonstrating impact of the realistic scene context.