Reviewerly/exHarmony huggingface dataset:
TALL: Thumbnail Layout for Deepfake Video Detection

The growing threats of deepfakes to society and cybersecurity have raised enormous public concerns, and increasing efforts have been devoted to this critical topic of deepfake video detection. Existing video methods achieve good performance but are computationally intensive. This paper introduces a simple yet effective strategy named Thumbnail Layout (TALL), which transforms a video clip into a pre-defined layout to realize the preservation of spatial and temporal dependencies. Specifically, consecutive frames are masked in a fixed position in each frame to improve generalization, then resized to sub-images and rearranged into a pre-defined layout as the thumbnail. TALL is model-agnostic and extremely simple by only modifying a few lines of code. Inspired by the success of vision transformers, we incorporate TALL into Swin Transformer, forming an efficient and effective method TALL-Swin. Extensive experiments on intra-dataset and cross-dataset validate the validity and superiority of TALL and SOTA TALL-Swin. TALL-Swin achieves 90.79% AUC on the challenging cross-dataset task, FaceForensics++ → CelebDF. The code is available at https://github.com/rainy-xu/TALL4Deepfake.
Exploring Data as Quantum Bits: Analysis of Quantum Image Encoding Patterns for NISQ era

The generall efficiency of a computational system, be it simple or complex, presentation of the data plays a crucial role in getting the most out of it. Classical information theory, quantum mechanics, and computer science collectively influence Quantum Information in comprehending how information can be encoded, transmitted, and modified using quantum systems. Quantum mechanics allows quantum information to incorporate a variety of properties, such as quantum encoding of classical information, storing coherent details in quantum superpositions, quantum computations and entanglement, which occurs when two or more qubits become highly correlated and share an unbreakable quantum state. There are numerous barriers and constraints in the broad field of quantum computation, ranging from dealing with Decoherence to ensuring error correction. Moreover, the qubit, a fundamental computational unit, is challenging to create, manipulate, and store. To ensure the quality of the Quantum Algorithms, the area of quantum data encoding has been thoroughly researched and explored as relevant results can be obtained by adequate encoding. In this paper, we reviewed the terminology and physical modeling of qubits, as well as the encoding approaches used to transform data from classical to quantum models. Furthermore, different encoding schemes for Quantum Image Processing were explored to determine the efficiency of quantum pixel representation(QPIXL) framework for effective application in NISQ era.
Internet of Things (IoT) Network Forensic Analysis Using the Raspberry Pi 4 Model B and Open-Source Tools

The Internet of Things’ (IoT) rapid growth has led to increased concerns regarding network security and the need for effective forensic analysis. This research focuses on investigating the capabilities of the Raspberry Pi 4 Model B and open-source tools for conducting IoT network forensic analysis and collecting digital evidence. The proposed system comprises three components: the IoT Physical Intrusion System Module, the Attack Module, and the Forensic Module. Within the IoT Physical Intrusion System, various attack scenarios, including Denial of Service, SSH Brute Force, and Man in the Middle, were successfully launched. IDS systems such as Wazuh Agent, Suricata, and Fail2ban log application were installed on the Raspberry Pi, effectively identifying IoT threats and providing alerts. The Wazuh Manager, serving as the forensic workstation, received the alerts and facilitated the analysis of attacks, presenting graphical representations. This research demonstrating the practicality of using micro-controller-based device for portable IoT network forensic analysis.
GAFlow: Incorporating Gaussian Attention into Optical Flow

Optical flow, or the estimation of motion fields from image sequences, is one of the fundamental problems in computer vision. Unlike most pixel-wise tasks that aim at achieving consistent representations of the same category, optical flow raises extra demands for obtaining local discrimination and smoothness, which yet is not fully explored by existing approaches. In this paper, we push Gaussian Attention (GA) into the optical flow models to accentuate local properties during representation learning and enforce the motion affinity during matching. Specifically, we introduce a novel Gaussian-Constrained Layer (GCL) which can be easily plugged into existing Transformer blocks to highlight the local neighborhood that contains fine-grained structural information. Moreover, for reliable motion analysis, we provide a new Gaussian-Guided Attention Module (GGAM) which not only inherits properties from Gaussian distribution to instinctively revolve around the neighbor fields of each point but also is empowered to put the emphasis on contextually related regions during matching. Our fully-equipped model, namely Gaussian Attention Flow network (GAFlow), naturally incorporates a series of novel Gaussian-based modules into the conventional optical flow framework for reliable motion analysis. Extensive experiments on standard optical flow datasets consistently demonstrate the exceptional performance of the proposed approach in terms of both generalization ability evaluation and online benchmark testing. Code is available at https://github.com/LA30/GAFlow.
Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception

We introduce the Aria Digital Twin (ADT) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> - an egocentric dataset captured using Aria glasses with extensive object, environment, and human level ground truth. This ADT release contains 200 sequences of real-world activities conducted by Aria wearers in two real indoor scenes with 398 object instances (344 stationary and 74 dynamic). Each sequence consists of: a) raw data of two monochrome camera streams, one RGB camera stream, two IMU streams; b) complete sensor calibration; c) ground truth data including continuous 6-degree-of-freedom (6DoF) poses of the Aria devices, object 6DoF poses, 3D eye gaze vectors, 3D human poses, 2D image segmentations, image depth maps; and d) photo-realistic synthetic renderings. To the best of our knowledge, there is no existing egocentric dataset with a level of accuracy, photo-realism and comprehensiveness comparable to ADT. By contributing ADT to the research community, our mission is to set a new standard for evaluation in the egocentric machine perception domain, which includes very challenging research problems such as 3D object detection and tracking, scene reconstruction and understanding, sim-to-real learning, human pose prediction - while also inspiring new machine perception tasks for augmented reality (AR) applications. To kick start exploration of the ADT research use cases, we evaluated several existing state-of-the-art methods for object detection, segmentation and image translation tasks that demonstrate the usefulness of ADT as a benchmarking dataset.
Multi3DRefer: Grounding Text Description to Multiple 3D Objects

We introduce the task of localizing a flexible number of objects in real-world 3D scenes using natural language descriptions. Existing 3D visual grounding tasks focus on localizing a unique object given a text description. However, such a strict setting is unnatural as localizing potentially multiple objects is a common need in real-world scenarios and robotic tasks (e.g., visual navigation and object rearrangement). To address this setting we propose Multi3DRefer, generalizing the ScanRefer dataset and task. Our dataset contains 61926 descriptions of 11609 objects, where zero, single or multiple target objects are referenced by each description. We also introduce a new evaluation metric and benchmark methods from prior work to enable further investigation of multi-modal 3D scene understanding. Furthermore, we develop a better baseline leveraging 2D features from CLIP by rendering object proposals online with contrastive learning, which outperforms the state of the art on the ScanRefer benchmark.
Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learning

Semi-supervised learning is attracting blooming attention, due to its success in combining unlabeled data. To mitigate potentially incorrect pseudo labels, recent frameworks mostly set a fixed confidence threshold to discard uncertain samples. This practice ensures high-quality pseudo labels, but incurs a relatively low utilization of the whole unlabeled set. In this work, our key insight is that these uncertain samples can be turned into certain ones, as long as the confusion classes for the top-1 class are detected and removed. Invoked by this, we propose a novel method dubbed ShrinkMatch to learn uncertain samples. For each uncertain sample, it adaptively seeks a shrunk class space, which merely contains the original top-1 class, as well as remaining less likely classes. Since the confusion ones are removed in this space, the re-calculated top-1 confidence can satisfy the pre-defined threshold. We then impose a consistency regularization between a pair of strongly and weakly augmented samples in the shrunk space to strive for discriminative representations. Furthermore, considering the varied reliability among uncertain samples and the gradually improved model during training, we correspondingly design two reweighting principles for our uncertain loss. Our method exhibits impressive performance on widely adopted benchmarks.
Sentiment Analysis for Shona

No sentiment analysis system existed for Shona yet—even though it is a Bantu language spoken by nearly 17 million people. Consequently, we collected ShonaSenti—a new corpus of 16,000 tweets in Shona covering 8 different topics. In this paper, we describe our distant supervised labelling strategies to support the annotators to categorize the collected tweets into the 5 sentiment classes of very negative, negative, neutral, positive and very positive. Moreover, we leveraged the Shona sentiment analysis corpus to develop first mono-lingual and cross-lingual sentiment analysis systems for Shona. Our best sentiment analysis systems are cross-lingual and mono-lingual Transformer-based systems which achieve accuracies of 84% on 3 sentiment classes and 70% on 5 sentiment classes.
HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustness

Convolutional Neural Networks (CNN) are known to exhibit poor generalization performance under distribution shifts. Their generalization have been studied extensively, and one line of work approaches the problem from a frequency-centric perspective. These studies highlight the fact that humans and CNNs might focus on different frequency components of an image. First, inspired by these observations, we propose a simple yet effective data augmentation method HybridAugment that reduces the reliance of CNNs on high-frequency components, and thus improves their robustness while keeping their clean accuracy high. Second, we propose HybridAugment++, which is a hierarchical augmentation method that attempts to unify various frequency-spectrum augmentations. HybridAugment++ builds on HybridAugment, and also reduces the reliance of CNNs on the amplitude component of images, and promotes phase information instead. This unification results in competitive to or better than state-of-the-art results on clean accuracy (CIFAR-10/100 and ImageNet), corruption benchmarks (ImageNet-C, CIFAR-10-C and CIFAR-100-C), adversarial robustness on CIFAR-10 and out-of-distribution detection on various datasets. HybridAugment and HybridAugment++ are implemented in a few lines of code, does not require extra data, ensemble models or additional networks <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Quantum-Improved Weather Forecasting: Integrating Quantum Machine Learning for Precise Prediction and Disaster Mitigation

In numerous industries, weather forecasting is essential for making informed decisions and mitigating the effects of extreme weather events. The complexity and chaos of weather systems, however, place restrictions on standard procedures, leading to errors and significant threats. We suggest the Quantum Improved Weather Forecast framework, which combines quantum machine learning methods with conventional methodologies, to solve these issues. By using quantum algorithms including quantum support vector machines, quantum neural networks, and quantum clustering, the QWF framework seeks to improve accuracy. Despite a few drawbacks, the QWF framework provides a method to revolutionize weather forecasting by enhancing prediction accuracy and facilitating improved catastrophe preparedness. It may prevent fatalities, safeguard critical infrastructure, and promote sustainable growth.
FlipNeRF: Flipped Reflection Rays for Few-shot Novel View Synthesis

Neural Radiance Field (NeRF) has been a mainstream in novel view synthesis with its remarkable quality of rendered images and simple architecture. Although NeRF has been developed in various directions improving continuously its performance, the necessity of a dense set of multi-view images still exists as a stumbling block to progress for practical application. In this work, we propose FlipNeRF, a novel regularization method for few-shot novel view synthesis by utilizing our proposed flipped reflection rays. The flipped reflection rays are explicitly derived from the input ray directions and estimated normal vectors, and play a role of effective additional training rays while enabling to estimate more accurate surface normals and learn the 3D geometry effectively. Since the surface normal and the scene depth are both derived from the estimated densities along a ray, the accurate surface normal leads to more exact depth estimation, which is a key factor for few-shot novel view synthesis. Furthermore, with our proposed Uncertainty-aware Emptiness Loss and Bottleneck Feature Consistency Loss, FlipNeRF is able to estimate more reliable outputs with reducing floating artifacts effectively across the different scene structures, and enhance the feature-level consistency between the pair of the rays cast toward the photo-consistent pixels without any additional feature extractor, respectively. Our FlipNeRF achieves the SOTA performance on the multiple benchmarks across all the scenarios.
PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization

In a joint vision-language space, a text feature (e.g., from "a photo of a dog") could effectively represent its relevant image features (e.g., from dog photos). Also, a recent study has demonstrated the cross-modal transferability phenomenon of this joint space. From these observations, we propose PromptStyler which simulates various distribution shifts in the joint space by synthesizing diverse styles via prompts without using any images to deal with source-free domain generalization. The proposed method learns to generate a variety of style features (from "a S <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∗</inf> style of a") via learnable style word vectors for pseudo-words S <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∗</inf> . To ensure that learned styles do not distort content information, we force style-content features (from "a S <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∗</inf> style of a [class]") to be located nearby their corresponding content features (from "[class]") in the joint vision-language space. After learning style word vectors, we train a linear classifier using synthesized style-content features. PromptStyler achieves the state of the art on PACS, VLCS, OfficeHome and DomainNet, even though it does not require any images for training.
LTV-Backed E-Signatures Using Post-Quantum Cryptography

Due to concerns over the evolving quantum computing scenario and its potential threat to existing cryptosystems, including Public-Key Infrastructure (PKI) used in electronic signatures, there is a need to enhance the eSign components with quantum-resistant algorithms. One such algorithm is CrystalsDilithium, based on lattice cryptography, which can perform PKI operations in the eSign system. In addition to being resistant to quantum attacks, Long-Term Validation (LTV) is also utilized to ensure the validity of a document for an extended period. This is achieved by referring to PAdES/CAdES profiles and a time stamping service, which provides an accurate and trustworthy record of when a particular electronic document, file, or transaction was created, modified, or sent. Given the significance of eSign as a crucial digital service worldwide, migrating to post-quantum components will strengthen the system’s security. Therefore, the proposed research provides implementation, analysis and experimentation to ensure the eSign system components are robust against quantum computing threats.
Learning Robust Representations with Information Bottleneck and Memory Network for RGB-D-based Gesture Recognition

Although previous RGB-D-based gesture recognition methods have shown promising performance, researchers often overlook the interference of task-irrelevant cues like illumination and background. These unnecessary factors are learned together with the predictive ones by the network and hinder accurate recognition. In this paper, we propose a convenient and analytical framework to learn a robust feature representation that is impervious to gesture-irrelevant factors. Based on the Information Bottleneck theory, two rules of Sufficiency and Compactness are derived to develop a new information-theoretic loss function, which cultivates a more sufficient and compact representation from the feature encoding and mitigates the impact of gesture-irrelevant information. To highlight the predictive information, we further integrate a memory network. Using our proposed content-based and contextual memory addressing scheme, we weaken the nuisances while preserving the task-relevant information, providing guidance for refining the feature representation. Experiments conducted on three public datasets demonstrate that our approach leads to a better feature representation and achieves better performance than state-of-the-art methods. The code of our method is available at: https://github.com/Carpumpkin/InBoMem.
Sentence Attention Blocks for Answer Grounding

Answer grounding is the task of locating relevant visual evidence for the Visual Question Answering task. While a wide variety of attention methods have been introduced for this task, they suffer from the following three problems: designs that do not allow the usage of pre-trained networks and do not benefit from large data pre-training, custom designs that are not based on well-grounded previous designs, therefore limiting the learning power of the network, or complicated designs that make it challenging to re-implement or improve them. In this paper, we propose a novel architectural block, which we term Sentence Attention Block, to solve these problems. The proposed block re-calibrates channel-wise image feature-maps by explicitly modeling inter-dependencies between the image feature-maps and sentence embedding. We visually demonstrate how this block filters out irrelevant feature-maps channels based on sentence embedding. We start our design with a well-known attention method, and by making minor modifications, we improve the results to achieve state-of-the-art accuracy. The flexibility of our method makes it easy to use different pre-trained backbone networks, and its simplicity makes it easy to understand and be re-implemented. We demonstrate the effectiveness of our method on the TextVQA-X, VQS, VQA-X, and VizWiz-VQA-Grounding datasets. We perform multiple ablation studies to show the effectiveness of our design choices.
Cost Explosion for Efficient Reinforcement Learning Optimisation of Quantum Circuits

Large scale optimisation of quantum circuits is a computationally challenging problem. Reinforcement Learning (RL) is a recent approach for learning strategies to optimise quantum circuits by increasing the reward of an optimisation agent. The reward is a function of the quantum circuit costs, such as gate and qubit counts, or circuit depth. Our goal is to improve the agent’s optimization strategy, by including hints about how quantum circuits are optimized manually: there are situations when the cost of a circuit should be allowed to temporary explode, before applying optimisations which significantly reduce the circuit’s cost. We bring numerical evidence, using Bernstein-Vazirani circuits, to support the advantage of this strategy. Our results are preliminary, and show that allowing cost explosions offers significant advantages for RL training, such as reaching optimum circuits. Cost explosion strategies have the potential to be an essential tool for RL of large-scale quantum circuit optimisation.
Sat2Density: Faithful Density Learning from Satellite-Ground Image Pairs

This paper aims to develop an accurate 3D geometry representation of satellite images using satellite-ground image pairs. Our focus is on the challenging problem of 3D-aware ground-views synthesis from a satellite image. We draw inspiration from the density field representation used in volumetric neural rendering and propose a new approach, called Sat2Density. Our method utilizes the properties of ground-view panoramas for the sky and non-sky regions to learn faithful density fields of 3D scenes in a geometric perspective. Unlike other methods that require extra depth information during training, our Sat2Density can automatically learn accurate and faithful 3D geometry via density representation without depth supervision. This advancement significantly improves the ground-view panorama synthesis task. Additionally, our study provides a new geometric perspective to understand the relationship between satellite and ground-view images in 3D space.
Qubit Control on IBM's Quantum Computing Devices

In this work, we demonstrate how quantum control methods can be applied in cloud-based quantum computers. In particular, we demonstrate the use of composite pulses to perform robust population transfer, single-qubit gates, and reduction of leakage out of the computational subspace. The experiments have been performed on one of IBM's quantum computers, demonstrating excellent agreement with theoretical predictions.