Reviewerly/exHarmony huggingface dataset:
Hyper-flexible Convolutional Neural Networks based on Generalized Lehmer and Power Means

Convolutional Neural Network is one of the famous members of the deep learning family of neural network architectures, which is used for many purposes, including image classification. In spite of the wide adoption, such networks are known to be highly tuned to the training data (samples representing a particular problem), and they are poorly reusable to address new problems. One way to change this would be, in addition to trainable weights, to apply trainable parameters of the mathematical functions, which simulate various neural computations within such networks. In this way, we may distinguish between the narrowly focused task-specific parameters (weights) and more generic capability-specific parameters. In this paper, we suggest a couple of flexible mathematical functions (Generalized Lehmer Mean and Generalized Power Mean) with trainable parameters to replace some fixed operations (such as ordinary arithmetic mean or simple weighted aggregation), which are traditionally used within various components of a convolutional neural network architecture. We named the overall architecture with such an update as a hyper-flexible convolutional neural network. We provide mathematical justification of various components of such architecture and experimentally show that it performs better than the traditional one, including better robustness regarding the adversarial perturbations of testing data.
The Evolving Menace of Ransomware: A Comparative Analysis of Pre-pandemic and Mid-pandemic Attacks

Drawing upon direct interviews and secondary sources, this article presents a qualitative comparative analysis of 39 ransomware attacks, 26 of which occurred shortly before the outbreak of the COVID-19 pandemic and 13 of which took place during the pandemic. The research objective was to gain an understanding of how ransomware attacks changed tactics across this period. Using inductive content analysis, a number of key themes emerged, namely (1) ransomware attackers have adopted more sinister tactics and now commit multiple crimes to maximise their return; (2) the expanded attack surface caused by employees working from home has greatly aggravated the risk of malicious intrusion; (3) the preferred attack vectors have changed, with phishing and VPN exploits now to the fore; (4) failure to adapt common business processes from off-line to on-line interaction has created vulnerabilities; (5) the ongoing laissez-faire attitude toward cybersecurity and lack of preparedness continues to be a substantial problem; and (6) ransomware attacks now pose potentially severe consequences for individuals, whose personal data has become a central part of the game. Recommendations are proposed to address these issues.
Synthesizing Game Levels for Collaborative Gameplay in a Shared Virtual Environment

We developed a method to synthesize game levels that accounts for the degree of collaboration required by two players to finish a given game level. We first asked a game level designer to create playable game level chunks. Then, two artificial intelligence (AI) virtual agents driven by behavior trees played each game level chunk. We recorded the degree of collaboration required to accomplish each game level chunk by the AI virtual agents and used it to characterize each game level chunk. To synthesize a game level, we assigned to the total cost function cost terms that encode both the degree of collaboration and game level design decisions. Then, we used a Markov-chain Monte Carlo optimization method, called simulated annealing, to solve the total cost function and proposed a design for a game level. We synthesized three game levels (low, medium, and high degrees of collaboration game levels) to evaluate our implementation. We then recruited groups of participants to play the game levels to explore whether they would experience a certain degree of collaboration and validate whether the AI virtual agents provided sufficient data that described the collaborative behavior of players in each game level chunk. By collecting both in-game objective measurements and self-reported subjective ratings, we found that the three game levels indeed impacted the collaboration gameplay behavior of our participants. Moreover, by analyzing our collected data, we found moderate and strong correlations between the participants and the AI virtual agents. These results show that game developers can consider AI virtual agents as an alternative method for evaluating the degree of collaboration required to finish a game level.
RenderSR: A Lightweight Super-Resolution Model for Mobile Gaming Upscaling

Mobile game play can be a prime use case where an efficient SR network can lead to both performance boosts and power savings. In this paper, we present RenderSR (RSR), a bandwidth aware super-resolution network designed for use in mobile game upscaling. We explore how different factors affect the resulting image quality: color space, the inclusion of the depth channel, sharpening. With a 40K parameter size, RenderSR without sharpening achieves a PSNR value difference ranging -0.41 to 0.36dB from several much larger SR models. RenderSR with sharpening super resolved large objects such as rocks, buildings, tree trunks are almost identical to the ground truth. Based on our performance experiment, we propose that RenderSR upscales the GPU rendered image on NPU or DSP on the mobile SoC.
Adversarial Machine Learning Attacks Against Video Anomaly Detection Systems

Anomaly detection in videos is an important computer vision problem with various applications including auto-mated video surveillance. Although adversarial attacks on image understanding models have been heavily investigated, there is not much work on adversarial machine learning targeting video understanding models and no previous work which focuses on video anomaly detection. To this end, we investigate an adversarial machine learning attack against video anomaly detection systems, that can be implemented via an easy-to-perform cyber-attack. Since surveillance cameras are usually connected to the server running the anomaly detection model through a wireless network, they are prone to cyber-attacks targeting the wireless connection. We demonstrate how Wi-Fi deauthentication attack, a notoriously easy-to-perform and effective denial-of-service (DoS) attack, can be utilized to generate adversarial data for video anomaly detection systems. Specifically, we apply several effects caused by the Wi-Fi deauthentication attack on video quality (e.g., slow down, freeze, fast forward, low resolution) to the popular bench-mark datasets for video anomaly detection. Our experiments with several state-of-the-art anomaly detection models show that the attackers can significantly undermine the reliability of video anomaly detection systems by causing frequent false alarms and hiding physical anomalies from the surveillance system.
CIPPSRNet: A Camera Internal Parameters Perception Network Based Contrastive Learning for Thermal Image Super-Resolution

Thermal Image Super-Resolution (TISR) is a technique for converting Low-Resolution (LR) thermal images to High-Resolution (HR) thermal images. This technique has recently become a research hotspot due to its ability to reduce sensor costs and improve visual perception. However, current research does not provide an effective solution for multi-sensor data training, possibly driven by pixel mismatch and simple degradation setting issues. In this paper, we proposed a Camera Internal Parameters Perception Network (CIPPSRNet) for LR thermal image enhancement. The camera internal parameters (CIP) were explicitly modeled as a feature representation, the LR features were transformed into the intermediate domain containing the internal parameters information by perceiving CIP representation. The mapping between the intermediate domain and the spatial domain of the HR features was learned by CIPPSRNet. In addition, we introduced contrastive learning to optimize the pretrained Camera Internal Parameters Representation Network and the feature encoders. Our proposed network is capable of achieving a more efficient transformation from the LR to the HR domains. Additionally, the use of contrastive learning can improve the network’s adaptability to misalignment data with insufficient pixel matching and its robustness. Experiments on PBVS2022 TISR Dataset show that our network has achieved state-of-the-art performance for the Thermal SR task.
Characterizing Target-absent Human Attention

Human efficiency in finding a target in an image has attracted the attention of machine learning researchers, but what about when no target is there? Knowing how people search in the absence of a target, and when they stop, is important for Human-computer-interaction systems attempting to predict human gaze behavior in the wild. Here we report a rigorous evaluation of target-absent search behavior using the COCO-Search18 dataset to train state-of-the-art models. We focus on two specific aims. First, we characterize the presence of a target guidance signal in target-absent search behavior by comparing it to target-present guidance and free viewing. We do this by comparing how well a model trained on one type of fixation behavior (target-present, target-absent, free viewing) can predict behavior in either the same or different task. To compare target-absent search to free viewing behavior we created COCO-FreeView, a dataset of free-viewing fixations for the same images used in COCO-Search18. These comparisons revealed the existence of a target guidance signal in target-absent search, albeit one much less dominant compared to when a target actually appeared in an image, and that the target-absent guidance signal was similar to free viewing in that saliency and center bias were both weighted more than guidance from target features. Our second aim focused on the stopping criteria, a question intrinsic to target-absent search. Here we propose to train a foveated target detector whose target detection representation is sensitive to the relationship between distance from the fovea. Then combining the predicted target detection representation with other information such as fixation history and subject ID, our model outperforms the baselines in predicting when a person stops moving his attention during target-absent search.
Privacy Leakage of Adversarial Training Models in Federated Learning Systems

Adversarial Training (AT) is crucial for obtaining deep neural networks that are robust to adversarial attacks, yet recent works found that it could also make models more vulnerable to privacy attacks. In this work, we further reveal this unsettling property of AT by designing a novel privacy attack that is practically applicable to the privacy-sensitive Federated Learning (FL) systems. Using our method, the attacker can exploit AT models in the FL system to accurately reconstruct users’ private training images even when the training batch size is large. Code is available at https://github.com/zjysteven/PrivayAttack_AT_FL.
Ex-Model: Continual Learning from a Stream of Trained Models

Learning continually from non-stationary data streams is a challenging research topic of growing popularity in the last few years. Being able to learn, adapt, and generalize continually in an efficient, effective, and scalable way is fundamental for a sustainable development of Artificial Intelligent systems. However, an agent-centric view of continual learning requires learning directly from raw data, which limits the interaction between independent agents, the efficiency, and the privacy of current approaches. Instead, we argue that continual learning systems should exploit the availability of compressed information in the form of trained models. In this paper, we introduce and formalize a new paradigm named "Ex-Model Continual Learning" (ExML), where an agent learns from a sequence of previously trained models in-stead of raw data. We further contribute with three ex-model continual learning algorithms and an empirical setting comprising three datasets (MNIST, CIFAR-10 and CORe50), and eight scenarios, where the proposed algorithms are extensively tested. Finally, we highlight the peculiarities of the ex-model paradigm and we point out interesting future research directions.
Quality of Experience Oriented Adaptive Video Streaming for Edge Assisted Cellular Networks

HTTP adaptive streaming accounts for a large part of mobile Internet traffic. With the developing cellular communication technologies, the standard dynamic adaptive streaming over HTTP technique allows mobile terminals to adaptively configure the transmission rate of video streaming applications and achieve high quality of experiences (QoE) with many adaptive bitrate algorithms. However, existing schemes either neglect to consider radio access network (RAN) side conditions or only consider simple RAN information optimization schemes. In this letter, we propose a QoE-oriented adaptive video streaming scheme based on a dueling deep Q-learning network. The proposed scheme improves the QoE by jointly considering the physical layer transmission bandwidth and the higher layer buffer status. Through the numerical and prototyping results, we show that our proposed scheme outperforms the existing schemes, with the average QoE improvements of 12.6% to 28.8%.
Transformaly - Two (Feature Spaces) Are Better Than One

Anomaly detection is a well-established research area that seeks to identify samples outside of a predetermined distribution. An anomaly detection pipeline is comprised of two main stages: (1) feature extraction and (2) normality score assignment. Recent papers used pre-trained networks for feature extraction achieving state-of-the-art results. However, the use of pre-trained networks does not fully-utilize the normal samples that are available at train time. This paper suggests taking advantage of this information by using teacher-student training. In our setting, a pre-trained teacher network is used to train a student network on the normal training samples. Since the student network is trained only on normal samples, it is expected to deviate from the teacher network in abnormal cases. This difference can serve as a complementary representation to the pre-trained feature vector. Our method - T ransformaly - exploits a pre-trained Vision Transformer (ViT) to extract both feature vectors: the pre-trained (agnostic) features and the teacher-student (fine-tuned) features. We report state-of-the-art AUROC results in both the common unimodal setting, where one class is considered normal and the rest are considered abnormal, and the multimodal setting, where all classes but one are considered normal, and just one class is considered abnormal <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
VG-VAE: A Venatus Geometry Point-Cloud Variational Auto-Encoder

In this paper, we propose VG-VAE: Venatus Geometric Variational Auto-Encoder for capturing unsupervised hierarchical local and global geometric signatures in pointcloud. Recent research emphasises the significance of the underlying intrinsic geometry for pointcloud processing. Our contribution is to extract and analyse the morphology of the pointcloud using the proposed Geometric Proximity Correlator (GPC) and variational sampling of the latent. The extraction of local geometric signatures is facilitated by the GPC, whereas the extraction of global geometry is facilitated by variational sampling. Furthermore, we apply a naive mix of vector algebra and 3D geometry to extract the basic per-point geometric signature, which assists the unsupervised hypothesis. We provide statistical analyses of local and global geometric signatures. The impacts of our geometric features are demonstrated on pointcloud classification as downstream task using the classic pointcloud feature extractor PointNet. We demonstrate our analysis on ModelNet40 a benchmark dataset, and compare with state-of-the-art techniques.
User-Guided Variable Rate Learned Image Compression

We propose a learning-based image compression method that achieves any arbitrary input bitrate via user-guided bit allocation to preferred regions. We verify our hypothesis of incorporating user guidance for bitrate control by experimenting with alternatives that do not have any guidance. We conduct extensive evaluation on CelebA-HQ and CityScapes dataset using standard quantitative metrics and human studies showing that our single model for multiple bitrates achieves similar or better performance as com-pared to previous learned image compression methods that require re-training for each new bitrate.
A Modular Multimodal Architecture for Gaze Target Prediction: Application to Privacy-Sensitive Settings

Predicting where a person is looking is a complex task, requiring to understand not only the person’s gaze and scene content, but also the 3D scene structure and the per-son’s situation (are they manipulating? interacting or observing others? attentive?) to detect obstructions in the line of sight or apply attention priors that humans typically have when observing others. In this paper, we hypothesize that identifying and leveraging such priors can be better achieved through the exploitation of explicitly derived multimodal cues such as depth and pose. We thus propose a modular multimodal architecture allowing to combine these cues using an attention mechanism. The architecture can naturally be exploited in privacy-sensitive situations such as surveillance and health, where personally identifiable information cannot be released. We perform extensive experiments on the GazeFollow and VideoAttentionTarget public datasets, obtaining state-of-the-art performance and demonstrating very competitive results in the privacy setting case. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
ScanpathNet: A Recurrent Mixture Density Network for Scanpath Prediction

Understanding the mechanisms underlying human visual attention is an important research problem in cognitive neuroscience and computer vision. While existing models predict salient regions (i.e., saliency maps) and temporal sequences of eye fixations (i.e., scanpaths) in images, their designs often partially follow theoretical frameworks. Here, we introduce ScanpathNet, a deep learning model inspired by the latest theoretical model in neuroscience. It is ’guided’ by a dynamic priority map influenced by semantic content and fixation history. The model leverages convolutional neural networks to extract rich semantic features, convolutional long short-term memory networks to model the inhibition of return mechanism and sequential dependencies of fixations, and mixture density networks to predict probability distributions of fixations for each pixel. Simulated human scanpaths can then be generated by sequentially sampling the output of the proposed model. Despite its simplicity, ScanpathNet showed promising qualitative and quantitative scanpath prediction performance in extensive experiments on numerous eye-tracking benchmark datasets.
Self Supervised Scanpath Prediction Framework for Painting Images

In our paper, we propose a novel strategy to learn distortion invariant latent representation from painting pictures for visual attention modelling downstream task. In further detail, we design an unsupervised framework that jointly maximises the mutual information over different painting styles. To show the effectiveness of our approach, we firstly propose a lightweight scanpath baseline model and compare its performance to some state-of-the-art methods. Secondly, we train the encoder of our baseline model on large-scale painting images to study the efficiency of the proposed self-supervised strategy. The lightweight decoder proves effective in learning from the self-supervised pre-trained encoder with better performances than the end-to-end fine-tuned supervised baseline on two painting datasets, including a proposed new visual attention modelling dataset. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1 2</sup>
Multi-Head Distillation for Continual Unsupervised Domain Adaptation in Semantic Segmentation

Unsupervised Domain Adaptation (UDA) is a transfer learning task which aims at training on an unlabeled target domain by leveraging a labeled source domain. Beyond the traditional scope of UDA with a single source domain and a single target domain, real-world perception systems face a variety of scenarios to handle, from varying lighting conditions to many cities around the world. In this context, UDAs with several domains increase the challenges with the addition of distribution shifts within the different target domains. This work focuses on a novel framework for learning UDA, continuous UDA, in which models operate on multiple target domains discovered sequentially, without access to previous target domains. We propose MuHDi, for Multi-Head Distillation, a method that solves the catastrophic forgetting problem, inherent in continual learning tasks. MuHDi performs distillation at multiple levels from the previous model as well as an auxiliary target-specialist segmentation head. We report both extensive ablation and experiments on challenging multi-target UDA semantic segmentation benchmarks to validate the proposed learning scheme and architecture.
Strengthening the Transferability of Adversarial Examples Using Advanced Looking Ahead and Self-CutMix

Deep neural networks (DNNs) are vulnerable to adversarial examples generated by adding malicious noise imperceptible to a human. The adversarial examples successfully fool the models under the white-box setting, but the performance of attacks under the black-box setting degrades significantly, which is known as the low transferability problem. Various methods have been proposed to improve transferability, yet they are not effective against adversarial training and defense models. In this paper, we introduce two new methods termed Lookahead Iterative Fast Gradient Sign Method (LI-FGSM) and Self-CutMix (SCM) to address the above issues. LI-FGSM updates adversarial perturbations with the accumulated gradient obtained by looking ahead. A previous gradient-based attack is used for looking ahead during N steps to explore the optimal direction at each iteration. It allows the optimization process to escape the sub-optimal region and stabilize the update directions. SCM leverages the modified CutMix, which copies a patch from the original image and pastes it back at random positions of the same image, to preserve the internal information. SCM makes it possible to generate more transferable adversarial examples while alleviating the overfitting to the surrogate model employed. Our two methods are easily incorporated with the previous iterative gradient-based attacks. Extensive experiments on ImageNet show that our approach acquires state-of-the-art attack success rates not only against normally trained models but also against adversarial training and defense models.