Reviewerly/exHarmony huggingface dataset:
Modeling internal state changes in free-viewing and visual search scanpaths with gain control in DeepGaze III

The DeepGaze III model currently sets the state-of-the-art in predicting free-viewing human scanpaths on natural images by predicting future fixations from the observed image and recent fixation locations. Inspired by gain control mechanisms in Neuroscience, we introduce gain control layers into the network architecture which can modulate the activity in certain channels of the network depending on additional factors, such as observer biases or search targets. By comparing the prediction performance of the baseline model with the performance of such an extended model in terms of information gain, we can quantify the amount of information that additional factors contribute to fixation placement. Due to the modular DeepGaze III architecture, we can decompose the information gain into different components: (1) a first component affecting only the modulation amplitude of the fixation distribution, (2) a second component modulating which image features are salient, and (3) a third component affecting the scanpath dynamics. Applying this approach, we quantify how much a fixation’s index in a scanpath, subject identity and search targets affect scanpaths in free-viewing and visual search. For free-viewing, we find that fixation index and subject identity contribute to a similar degree to fixation placement. In the case of fixation index, this information is equally split into a part making the fixation density more uniform over time, and a part changing which image features are salient. The contribution of subject identity is mostly due to different subjects preferring different image features. For visual search on the COCO Search18 dataset, the search target increases the explained information by 18% compared to just the presented image, suggesting substantial similarities in fixation behavior across targets. Our work demonstrates how contrast gain control can be used as a very general and sample-efficient mechanism to flexibly modify neural network computation to account for additional factors of interest.
Unsupervised contrastive learning and supervised classification training have opposite effects on the human-likeness of CNNs during occluded object processing

Human observers can readily perceive and recognize visual objects even when occluding stimuli obscure much of the object from view. By contrast, state-of-the-art convolutional neural network (CNN) models of primate vision perform poorly at classifying occluded objects (Coggan and Tong, VSS, 2022). A key difference between biological and artificial visual systems is how they learn from visual examples. CNNs are typically trained using supervised methods to classify images by object category based on labelled data. By contrast, humans learn about objects with a broader range of learning objectives and fewer opportunities for supervised feedback. Here, we asked whether a more naturalistic approach to training CNNs might yield more occlusion-robust models that better predict human neural and behavioural responses to occluded objects. To address this question, we trained an array of CORnet-S model instances with either supervised classification or unsupervised contrastive learning. We also augmented the standard ImageNet dataset by superimposing artificial occluders onto the images. The contrastive learning objective was to produce similar unit activations in the highest layer to differently occluded instances of the same underlying object image. Once training was complete, each model was tested for occlusion robustness and compared to human behavioural and neural responses to occluded objects. For supervised models, we found that training on the occluded dataset led to substantial improvements in classification accuracy for novel occluded objects, relative to the standard dataset. Despite these improvements, occlusion-trained models performed worse at predicting both human behavioural and neural responses to occluded objects, suggesting that these supervised models learned a different type of occlusion-robust mechanism. By contrast, the layer-wise activity patterns found in the unsupervised, contrastively trained models exhibited stronger occlusion robustness and greater human-likeness than any other model, suggesting that human robustness to occlusion may be attributable in part to a natural, unsupervised visual learning environment.
Reconstructing facial motion across views using a multi-view face space.

While the concept of ‘face space’ (Valentine, 1991) has been highly influential in representing identity, it has seen fewer applications in representing within-person variation such as changes in appearance with lighting, expression, speech and pose. Recently, Burton and colleagues (2016) proposed that in addition to an identity-general space that captures between-individual variation, we may also have a separate, identity-specific space for each familiar individual that captures their within-person variation. We addressed the problem of viewpoint invariance using expression-based PCA spaces. As there is currently little evidence for a 3D representation in humans, we assessed the possibility of identity-specific, multi-view face spaces, based on a few prototypical 2D views, that can be used to represent and reconstruct facial motion across changes in viewpoint. When we concatenate video from five views, captured simultaneously, together into multi-view vectors and create a single multi-view PCA space, we are able to reconstruct facial motion across views remarkably well from a single input view. However, this model lacked biologically plausibility due to the need to ‘see’ multiple views simultaneously, so we then assessed more biologically plausible models. If we make a separate space for each viewpoint, we are able to reconstruction motion across views well, but only once the association between the components across views is learned. We also explored a model in which neighbouring views were concatenated in time, creating a single space with overlapping viewpoints. This avoided the requirement for separate, view-specific spaces, whilst still also allowing us to reconstruct motion across views well. These models, although a work in progress, show that a 3D representation might not be necessary for certain face-related tasks such as processing motion and speech across views, and that a few 2D representations might be sufficient.
Shared Representation of Different Material Categories: Transfer Learning of Crystals From Soaps

We encounter various materials every day (e.g., plastics, soaps, and stones), and often need to estimate attributes of novel materials in new environments. How do humans judge material properties across different categories of materials, which possess common but also distinctive visual characteristics? We hypothesize that the features humans use to perceive materials might be overlapping, although different materials have their peculiar visual characteristics pertinent to their physical properties. We previously demonstrated the unsupervised image synthesis model, StyleGAN, can generate perceptually convincing images of translucent objects (e.g., soaps). Here, using transfer learning, we test whether the model pretrained on a soap dataset provides critical information to learn a different material, crystals. Specifically, we transfer the pre-trained StyleGAN from a large soap dataset to a relatively small crystal dataset, via full model fine-tuning. With little training time, we obtain a new generator that synthesizes realistic crystals. The layer-wise latent spaces of both material domains show similar scale-specific semantic representations: early-layers represent coarse spatial-scale features (e.g., the object’s contour), middle-layers represent mid-level spatial-scale features (e.g., material), and later-layers represent fine spatial-scale features (e.g., color). Notably, by swapping the weights between the soap and crystal generators, we find that the multiscale generative processes of the materials (spanning from 4-by-4 to 1024-by-1024 resolution) mostly differ in the coarse spatial-scale convolution layers. Convolution weights at the 32-by-32 resolution generative stage determine the critical difference in the geometry of the two materials. Convolution weights at 64-by-64 resolution decode the common characteristics (e.g., translucency) between the materials. Moreover, without additional training, we could create new material appearances that have visual features from both training categories. Together, we show that there are overlapping latent image features among distinctive material categories, and that learning features from one material benefits learning a new material.
Naturalistic primate vision combines generative and discriminative computations

Biological and machine vision needs to be statistically and computationally efficient, enabling both robust and rapid perception from sensory evidence. The normative ideal, as embodied by the generative approach, interprets evidence optimally in the context of a statistical model of the world. This ideal is computationally expensive or even intractable for naturalistic and dynamic vision. The discriminative approach emphasizes rapid inference without the use of an explicit generative model. I will introduce generative and discriminative models as an explanatory framework characterizing visual inference at different levels of analysis. Models of primate vision can be understood as points in a vast space of possible inference models that contains classical algorithms like predictive coding, the feedforward cascade of filters in a convolutional neural network, along with more novel concepts from engineering such as amortized inference. I will clarify what kind of evidence is required to discern whether primate visual inference is generative. A key insight is that “more is different”: what seems viable as a visual inference algorithm for an abstracted toy task may not scale to naturalistic real-world vision. We should therefore scale our tasks to be more naturalistic and dynamic, exposing the brain’s unique combination of generative models and discriminative computations.
Spatial-frequency channels for object recognition by neural networks are twice as wide as those of humans

We compare how well human observers and neural networks recognize ImageNet images in filtered noise using the critical-band masking paradigm (Fletcher, 1940). We assess the spatial-frequency tuning of object recognition by measuring its noise tolerance at various frequencies. 16 observers and 4 neural networks performed 16-way categorization of 1050 ImageNet images perturbed by band-limited Gaussian noise of five strengths centered at seven spatial frequencies. We find that the noise sensitivity of object recognition is an inverted, U-shaped function of spatial frequency. Human observer performance is severely impaired over an octave-wide band. Such octave-wide selectivity has frequently appeared in the vision literature where it is called a “channel” (Solomon & Pelli, 1994). Here, this channel is revealed in heatmaps of human and network performance. A demo of this channel presents images perturbed with noise at various frequencies and strengths. Together with previous work (Majaj et al., 2002), this shows that the bandwidth of the human channel is conserved across diverse target kinds: real-world objects, gratings, and letters. We find the classic octave-wide frequency band for humans, and a two-octave-wide band for machines, both centered at ~28 cycles per image. At the peak, humans tolerate four times as much noise variance as networks do. When recognizing objects, networks are known to rely on texture cues whereas humans rely on shape (Geirhos et al., 2018). Data augmentation helps bridge this gap (Hermann et al., 2020, Geirhos et al., 2021, Muttenthaler et al., 2022). Our results show that perhaps the popular notion of texture-vs-shape bias simply reflects the width of the spatial-frequency channel.
Visual Search Patterns in Cerebral Visual Impairment (CVI) Are Driven by Saliency Cues When Exploring Naturalistic Scenes

Higher order visual perceptual deficits are a cardinal feature in individuals with cerebral visual impairment (CVI). We investigated visual search behavior in CVI participants (n=16; mean age=16.00 yrs ± 5.03 SD) and neurotypical controls (n=31; mean age=20.59 yrs ± 5.56 SD). Participants searched for a target in a series of 80 naturalistic LabelMe scenes (mixture of indoor and outdoor scenes, fixed presentation time of 4 sec) in response to either an image cue or text cue while gaze behavior was recorded with an eye tracker (Tobii 4C; 90 Hz). Target identification accuracy and reaction time (RT) were collected. Receiver operating characteristic (ROC) analyses comparing the distribution of eye gaze patterns to an image saliency (Graph-Based Visual Saliency) model and semantic saliency (Linguistic Analysis of Semantic Salience) model were also carried out. Compared to controls, CVI participants were less accurate and slower in finding the target for both the image cue (accuracycontrols=91.05% ± 11.70 SD, accuracyCVI=50.63% ± 33.15 SD, t(16.95)=4.72, p<0.001, d=1.89; RTcontrols=1.20 sec ± 0.11 SD, RTCVI=1.77 sec ± 0.34 SD, t(16.78)=-6.44, p<0.001, d=-2.59) and text cue (accuracycontrols=88.06% ± 10.03 SD, accuracyCVI=42.81% ± 30.10 SD, t(16.74)=5.84, p<0.001, d=2.35; RTcontrols=1.37 sec ± 0.18 SD, RTCVI=2.00 sec ± 0.53 SD, t(15.61)=-4.446, p<0.001, d=-1.87) tasks. For the image cue task, controls and CVI participants showed more similar ROC scores with respect to the image saliency model (ROCcontrols=0.80 ± 0.04 SD; ROCCVI=0.74 ± 0.08 SD, t(19.27)=3.31, p=0.002, d=1.23). In contrast, for the text cue task, there was a greater disparity in ROC scores with respect to the semantic model (ROCcontrols=0.83 ± 0.06 SD; ROCCVI=0.65 ± 0.12 SD, t(17.66)=5.58, p<0.001, d=2.16). These findings suggest that CVI participants tend to rely more on image saliency than semantic saliency cues during visual search of naturalistic scenes.
The brain predominantly represents attended semantics rather than global semantics in a naturalistic task

During visual search, selective attention shifts the representation of visual semantic information towards the attended category (Cukur et al. 2013). Here we sought to determine whether semantic tuning shifts occur intrinsically during naturalistic tasks. We used fMRI to record brain activity from five participants while they performed a taxi-driver task in a large virtual world (110-180 minutes of data per subject). Subjects freely viewed the stimulus while eye movements were recorded at 60 Hz. The video game engine provided ground-truth semantic segmentation of video frames. Attended semantics were operationally defined as the semantic content of the video frame within 2.5° of fixation, which was used as a proxy for top-down attention. Global semantics were operationally defined as the semantic content of the entire video frame. Banded ridge regression (Nunez-Elizalde et al., 2019, Dupré la Tour et al., 2022) was used to estimate voxelwise encoding models simultaneously for attended semantics and for global semantics, along with 31 additional feature spaces that captured other aspects of the taxi-driver task. A held-out dataset was used to test statistical significance, prediction accuracy, and generalization. The participants’ visual behavior caused the semantic content of attended semantics to be dramatically different from the content of global semantics. Analysis of voxelwise encoding models show that attended semantics accounts for 12 ± 4% (mean ± std across subjects) of the total explained variance in well-predicted voxels. In contrast, global semantics accounts for only 1 ± 0.2% of the total explained variance. These results suggest that visual behavior in a naturalistic task focusses attention on task-relevant semantic categories, and that visual semantic representations in the brain are further biased to favor task-relevant categories at the expense of categories that are not important for the task. Thus, attentional effects must be accounted for in studies of naturalistic tasks.
A method to align real and virtual objects for mixed reality investigations of visually guide grasping

Virtual Reality (VR) is a powerful tool for studying perception, allowing researchers to present stimuli whose appearance and physical behaviour deviate from reality. Here, we used VR to investigate visually-guided grasping in mixed reality environments. By aligning real and virtual environments, participants could view virtual versions of objects while touching their real counterparts at the same location. We asked participants to grasp the objects, while we tracked markers attached to their hands and to the objects using 8 Qualisys Miqus M5 motion-tracking cameras. Additionally, we recorded temporally aligned videos using 6 Qualisys Miqus Video cameras. Marker positions were streamed into a virtual environment containing a realistic replica of the real object, which matched the motion of the real object. To verify the alignment between real and virtual scenes, we rendered the virtual environment from the same viewpoints as the video cameras. For visualization purposes, we overlaid the virtual object renderings onto the video recordings, thus generating a new, mixed reality depiction of the scene. To quantify the alignment overlap, we computed the intersection over union of the object segmented out of video and rendered scenes, achieving an average alignment of 84±3 %. We also quantified the alignment accuracy as the distance in image space between real and rendered tracked object markers. We achieved an average accuracy of 6.5±0.5 pixels, which translated to a misalignment of 1.7±0.1 mm and 5.1±0.3 minutes of visual angle for an average viewing distance of 115 cm, validating the effectiveness of our alignment procedure. By rendering the aligned virtual objects with different textures, we can vary the appearance of the objects’ material composition. This allows us to perform novel explorations investigating how the visuomotor system adapts to conflicts between the visual appearance and physical properties of graspable objects.
Applications of Artificial Intelligence in Renewable Energy: a brief review

Renewable energy is a sustainable substitute to fossil fuels, which are depleting and attributing to global warming as well as greenhouse gas emissions. Renewable energy innovations including solar, wind, and geothermal have grown significantly and play a critical role in meeting energy demands recently. Consequently, Artificial Intelligence (AI) could further enhance the benefits of renewable energy systems. The combination of renewable technologies and AI could facilitate the development of smart grids that can better manage energy distribution and storage. AI thus has the potential to optimize the efficiency and reliability of renewable energy systems, reduce costs, and improve their overall performance. This study provides an overview of the applications of algorithms and models of AI as well as its advantages and challenges in renewable energy systems.
Deep Learning Enabled Keystroke Eavesdropping Attack Over Videoconferencing Platforms

The COVID-19 pandemic has significantly impacted people by driving people to work from home using communication tools such as Zoom, Teams, Slack, etc. The users of these communication services have exponentially increased in the past two years, e.g., Teams annual users reached 270 million in 2022 and Zoom averaged 300 million daily active users in videoconferencing platforms. However, using edging artificial intelligence techniques, new cyber attacking tools expose these services to eavesdropping or disruption. This work investigates keystroke eavesdropping attacks on physical keyboards using deep learning techniques to analyze the acoustic emanation of keystroke audios to identify victims' keystrokes. An accurate context-free inferring algorithm was developed that can automatically predict keystrokes during inputs. The experimental results demonstrated that the accuracy of keystroke inference approaches is around 90% over normal laptop keyboards.
Analyzing Compositional Style in the Music of The Legend of Zelda: Ocarina of Time

The soundtrack of The Legend of Zelda: “Ocarina of Time” is rife with fascinating and innovative musical techniques which facilitate player engagement and interactivity. The composer Koji Kondo’s unique style, which often includes the use of modes, chromatic harmony, parallel motion, and proximal voice leading techniques, differentiates this game from others of the same era and efficiently communicates important narrative information like emotion, setting, and trope. Using these musical topics makes Kondo’s works distinct and recognizable. These musical elements also contribute to the game’s overall goal of immersion. By analyzing the characteristics of Kondo’s work, similarly effective stylistic topics can be implemented in new, original compositions to accomplish the same feats as Kondo’s famed soundtracks. The purpose of the project is to develop my personal compositional style which is audibly identifiable while also functioning as nuanced video game music. Some of these nuances of game music include the added challenges of player engagement and the necessity of continuous music; the duration of the music is not predetermined like a live performance or film score. Achieving this goal involves the analysis and recreation of Kondo’s original works, including simulating the limitations of the 1990s era game console hardware. Also, to modernize the findings of this project, similar analysis and experimentation is applied to works of other game composers, exploring the growth and changes in compositional techniques in games throughout the past several decades. The culmination of this research is my own body of original compositions which aim to achieve the goals outlined above. These works include both acoustic and digital pieces of music, and a live interactive suite for medium ensemble.
Natural Language Processing of Radiology Reports: Predicting Metastatic Progression from Text Data

The project’s goal is to extract tumour measurement data from oncological radiology reports with equal, or improved accuracy of a human radiologist. The purpose is to streamline and improve the efficiency of cancer diagnosis, accomplishing this through methods in artificial intelligence. In this experiment, a collection of 85,218 colorectal, and lung radiology reports were used. After loading reports into a data frame, and a BioBERT (bidirectional encoder representations from transformers pre-trained on biomedical corpora) model into a virtual environment, a question is set to be answered by the model where the context of the question is each radiology report’s findings section of the specified organ. These inputs are tokenized and embedded into numerical values to map sentences to vectors of real numbers. Vectors are fed into the model, and an answer is output in natural language for human readability. Answers are stored in the data frame in their corresponding row from which they were derived. The model successfully answered questions about measurements of tumours written in free text in reports where tumours were present, and successfully ignored or did not report in cases where tumours were not present, or measurements were unchanged. In cases with multiple tumours, the model reported exclusively first listed measurements. In an updated version of this model, context will be run through sentence-wise to ensure equal attention to the context entirely. This project is evidence that using one question and selecting the findings portion of a radiology report for one organ as context in a question-answering model built using BioBERT is effective, and efficient in collecting measurements from radiology reports. This algorithm can be applied to other areas of medicine, or other fields entirely with a few model alterations. This project is a step forward in improving cancer diagnosis efficiency and improving medical AI.