Reviewerly/exHarmony huggingface dataset:
Masked Retraining Teacher-Student Framework for Domain Adaptive Object Detection

Domain adaptive Object Detection (DAOD) leverages a labeled domain (source) to learn an object detector generalizing to a novel domain without annotation (target). Recent advances use a teacher-student framework, i.e., a student model is supervised by the pseudo labels from a teacher model. Though great success, they suffer from the limited number of pseudo boxes with incorrect predictions caused by the domain shift, misleading the student model to get sub-optimal results. To mitigate this problem, we propose Masked Retraining Teacher-student framework (MRT) which leverages masked autoencoder and selective retraining mechanism on detection transformer. Specifically, we present a customized design of masked autoencoder branch, masking the multi-scale feature maps of target images and reconstructing features by the encoder of the student model and an auxiliary decoder. This helps the student model capture target domain characteristics and become a more data-efficient learner to gain knowledge from the limited number of pseudo boxes. Furthermore, we adopt selective retraining mechanism, periodically re-initializing certain parts of the student parameters with masked autoencoder refined weights to allow the model to jump out of the local optimum biased to the incorrect pseudo labels. Experimental results on three DAOD benchmarks demonstrate the effectiveness of our method. Code can be found at https://github.com/JeremyZhao1998/MRT-release.
Rendering Humans from Object-Occluded Monocular Videos

3D understanding and rendering of moving humans from monocular videos is a challenging task. Despite recent progress, the task remains difficult in real-world scenarios, where obstacles may block the camera view and cause partial occlusions in the captured videos. Existing methods cannot handle such defects due to two reasons. First, the standard rendering strategy relies on point-point mapping, which could lead to dramatic disparities between the visible and occluded areas of the body. Second, the naive direct regression approach does not consider any feasibility criteria (i.e., prior information) for rendering under occlusions. To tackle the above drawbacks, we present OccNeRF, a neural rendering method that achieves better rendering of humans in severely occluded scenes. As direct solutions to the two drawbacks, we propose surface-based rendering by integrating geometry and visibility priors. We validate our method on both simulated and real-world occlusions and demonstrate our method’s superiority. Project page: https://cs.stanford.edu/~xtiange/projects/occnerf/
CC3D: Layout-Conditioned Generation of Compositional 3D Scenes

In this work, we introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts, trained using single-view images. Different from most existing 3D GANs that limit their applicability to aligned single objects, we focus on generating complex scenes with multiple objects, by modeling the compositional nature of 3D scenes. By devising a 2D layout-based approach for 3D synthesis and implementing a new 3D field representation with a stronger geometric inductive bias, we have created a 3D GAN that is both efficient and of high quality, while allowing for a more controllable generation process. Our evaluations on synthetic 3D-FRONT and real-world KITTI-360 datasets demonstrate that our model generates scenes of improved visual and geometric quality in comparison to previous works.
Downstream-agnostic Adversarial Examples

Self-supervised learning usually uses a large amount of unlabeled data to pre-train an encoder which can be used as a general-purpose feature extractor, such that downstream users only need to perform fine-tuning operations to enjoy the benefit of "large model". Despite this promising prospect, the security of pre-trained encoder has not been thoroughly investigated yet, especially when the pre-trained encoder is publicly available for commercial use.In this paper, we propose AdvEncoder, the first framework for generating downstream-agnostic universal adversarial examples based on the pre-trained encoder. AdvEncoder aims to construct a universal adversarial perturbation or patch for a set of natural images that can fool all the downstream tasks inheriting the victim pre-trained encoder. Unlike traditional adversarial example works, the pre-trained encoder only outputs feature vectors rather than classification labels. Therefore, we first exploit the high frequency component information of the image to guide the generation of adversarial examples. Then we design a generative attack framework to construct adversarial perturbations/patches by learning the distribution of the attack surrogate dataset to improve their attack success rates and transferability. Our results show that an attacker can successfully attack downstream tasks without knowing either the pre-training dataset or the downstream dataset. We also tailor four defenses for pre-trained encoders, the results of which further prove the attack ability of AdvEncoder. Our codes are available at: https://github.com/CGCL-codes/AdvEncoder.
SimMatchV2: Semi-Supervised Learning with Graph Consistency

Semi-Supervised image classification is one of the most fundamental problem in computer vision, which significantly reduces the need for human labor. In this paper, we introduce a new semi-supervised learning algorithm - Sim-MatchV2, which formulates various consistency regularizations between labeled and unlabeled data from the graph perspective. In SimMatchV2, we regard the augmented view of a sample as a node, which consists of a label and its corresponding representation. Different nodes are connected with the edges, which are measured by the similarity of the node representations. Inspired by the message passing and node classification in graph theory, we propose four types of consistencies, namely 1) node-node consistency, 2) node-edge consistency, 3) edge-edge consistency, and 4) edge-node consistency. We also uncover that a simple feature normalization can reduce the gaps of the feature norm between different augmented views, significantly improving the performance of SimMatchV2. Our SimMatchV2 has been validated on multiple semi-supervised learning benchmarks. Notably, with ResNet-50 as our backbone and 300 epochs of training, SimMatchV2 achieves 71.9% and 76.2% Top-1 Accuracy with 1% and 10% labeled examples on ImageNet, which significantly outperforms the previous methods and achieves state-of-the-art performance.
CLR: Channel-wise Lightweight Reprogramming for Continual Learning

Continual learning aims to emulate the human ability to continually accumulate knowledge over sequential tasks. The main challenge is to maintain performance on previously learned tasks after learning new tasks, i.e., to avoid catastrophic forgetting. We propose a Channel-wise Lightweight Reprogramming (CLR) approach that helps convolutional neural networks (CNNs) overcome catastrophic forgetting during continual learning. We show that a CNN model trained on an old task (or self-supervised proxy task) could be "reprogrammed" to solve a new task by using our proposed lightweight (very cheap) reprogramming parameter. With the help of CLR, we have a better stability-plasticity trade-off to solve continual learning problems: To maintain stability and retain previous task ability, we use a common task-agnostic immutable part as the shared "anchor" parameter set. We then add task-specific lightweight reprogramming parameters to reinterpret the outputs of the immutable parts, to enable plasticity and integrate new knowledge. To learn sequential tasks, we only train the lightweight reprogramming parameters to learn each new task. Reprogramming parameters are task-specific and exclusive to each task, which makes our method immune to catastrophic forgetting. To minimize the parameter requirement of reprogramming to learn new tasks, we make reprogramming lightweight by only adjusting essential kernels and learning channel-wise linear mappings from anchor parameters to task-specific domain knowledge. We show that, for general CNNs, the CLR parameter increase is less than 0.6% for any new task. Our method outperforms 13 state-of-the-art continual learning baselines on a new challenging sequence of 53 image classification datasets. Code and data are in the top link.
CIRI: Curricular Inactivation for Residue-aware One-shot Video Inpainting

Video inpainting aims at filling in missing regions of a video. However, when dealing with dynamic scenes with camera or object movements, annotating the inpainting target becomes laborious and impractical. In this paper, we resolve the one-shot video inpainting problem in which only one annotated first frame is provided. A naive solution is to propagate the initial target to the other frames with techniques like object tracking. In this context, the main obstacles are the unreliable propagation and the partially inpainted artifacts due to the inaccurate mask. For the former problem, we propose curricular inactivation to replace the hard masking mechanism for indicating the in-painting target, which is robust to erroneous predictions in long-term video inpainting. For the latter, we explore the properties of inpainting residue and present an online residue removal method in an iterative detect-and-refine manner. Extensive experiments on several real-world datasets demonstrate the quantitative and qualitative superiorities of our proposed method in one-shot video inpainting. More importantly, our method is extremely flexible that can be integrated with arbitrary traditional inpainting models, activating them to perform the reliable one-shot video inpainting task. Video demonstrations can be found in our supplement, and our code can be found at https://github.com/Arise-zwy/CIRI.
FB-BEV: BEV Representation from Forward-Backward View Transformations

View Transformation Module (VTM), where transformations happen between multi-view image features and Bird-Eye-View (BEV) representation, is a crucial step in camera-based BEV perception systems. Currently, the two most prominent VTM paradigms are forward projection and backward projection. Forward projection, represented by Lift-Splat-Shoot, leads to sparsely projected BEV features without post-processing. Backward projection, with BEV-Former being an example, tends to generate false-positive BEV features from incorrect projections due to the lack of utilization on depth. To address the above limitations, we propose a novel forward-backward view transformation module. Our approach compensates for the deficiencies in both existing methods, allowing them to enhance each other to obtain higher quality BEV representations mutually. We instantiate the proposed module with FB-BEV, which achieves a new state-of-the-art result of 62.4% NDS on the nuScenes test set. Code and models are available at https://github.com/NVlabs/FB-BEV.
Asymptotic Stability of Non-Demolition Quantum Trajectories with Measurement Imperfections

We consider the question of asymptotic stability of quantum trajectories undergoing quantum non-demolition imperfect measurement, that is to say the convergence of the estimated trajectory towards the true trajectory whose parameters and initial state are not necessarily known. We give conditions on the estimated initial state and regions of validity for the estimated parameters so that this convergence is ensured. We illustrate these results through numerical simulations on the physical example [1] and discuss the asymptotic stability for a more realistic general case where decoherence acts on the system. In this case, the evolution is described by new Kraus operators which do not satisfy the quantum non-demolition property.
Tokenization of the Common

The concept of the tragedy of the commons, originally rooted in economics, describes the depletion of shared resources due to self-interested actions by individuals. This work proposes a novel solution to address this economic challenge by leveraging tokens to capture its multidimensional nature. By utilising blockchain and DLTs, this decentralised approach aims to achieve a social optimum while promoting self-regulation. The paper presents a mathematical treatment of the tragedy of the commons, incorporating multi-dimensional tokens and exploring the divergence from the classic optimal solution, highlighting the potential of tokenisation in shaping a sustainable and efficient economy.
The music education and teaching innovation using blockchain technology supported by artificial intelligence

This exploration conducts innovative research on music education and teaching based on blockchain technology under AI and puts forward the general view of blockchain technology and its potential to contribute to future development. The essence, characteristics and development process of blockchain technology are discussed. Meanwhile, the thinking on the development of 'blockchain + education' is put forward, and the digital platform application of blockchain technology in music education and teaching is given. As the basis of system development, requirement analysis plays a crucial part in the subsequent system design and implementation. Analysing requirements can help people locate software functions more accurately and clearly. This exploration takes blockchain as the underlying technology support and relies on Fabric architecture to explore its application in digital education resource management.
A Study on Data-Driven Probability Estimator Design for Video Coding

Data-driven optimization is employed to study alternative approaches [1] to the probability estimator of the the Enhanced Compression Model (ECM) (which includes additional coding tools on top of the Versatile Video Coding standard). In ECM, each context model uses a weighted sum of two hypotheses for probability estimation with different associated adaptation rates. Four alternative approaches are studied:
Multi-Scale Information Can Do More: Attention Aggregation Mechanism for Image Compression

The semantic information obtained from large-scale computation in image compression is not practical. To solve this problem, we propose an Attention Aggregation Mechanism (AAM) for learning-based image compression, which is able to aggregate attention map from multiple scales and facilitate information embedding. Based on AAM and Swin Transformer, we design a Multi-scale Self-Attention Image Compression (MSAIC) approach as shown in Figure 1, which is using Multi-scale Swin Transformer blocks (MSTB) and convolutional layers stacks in the down-sampling encoder and up-sampling decoder to the best embed information. The MSTB is made up of convolution layers and three shifted window multi-head self-attention (SW-MSA) blocks, each of which has a different number of heads. In this way, a single MSTB block can aggregate three scales of attention map from tiny, middle and large patches. It can also function as a plug-and-play component to enhance CNN-based models. Meanwhile, we use the channel-wise autoregressive entropy model for the efficient and accurate entropy probability estimation. Extensive experiments show MSAIC is effective and attain the state-of-the-art (SOTA) compression performance, also outperform the SOTA method at the high bit rate.
Force-directed graph embedding with hops distance

Graph embedding has become an increasingly important technique for analyzing graph-structured data. By representing nodes in a graph as vectors in a low-dimensional space, graph embedding enables efficient graph processing and analysis tasks like node classification, link prediction, and visualization. In this paper, we propose a novel force-directed graph embedding method that utilizes the steady acceleration kinetic formula to embed nodes in a way that preserves graph topology and structural features. Our method simulates a set of customized attractive and repulsive forces between all node pairs with respect to their hop-distance. These forces are then used in Newton’s second law to obtain the acceleration of each node. The method is intuitive, parallelizable, and highly scalable. We evaluate our method on several graph analysis tasks and show that it achieves competitive performance compared to state-of-the-art unsupervised embedding techniques.
Video Anomaly Detection with Spatio-Temporal Inspired Deep Neural Networks (DNN)

Intelligent surveillance systems must be able to detect anomalies promptly to prevent malicious activity. It is common for deep learning methods to be used in video anomaly detection to focus on analyzing video streams from just one camera with a single scenario. Using large-scale training data with high complexity is necessary for these deep learning methods. This paper uses a spatiotemporal-inspired Deep Neural Network (DNN) to detect video anomalies. Rather than expensive optical flow calculations, a Deep Neural Network (DNN) is used for motion information in the proposed approach to achieve high recognition accuracy at a low computational cost. Experimental results on publicly available datasets demonstrate that the proposed model provides better input frame generation performance and is more accurate than existing approaches.
LsStk: Lightweight solution to preventing Stack from buffer overflow vulnerability

Buffer overflow is a significant vulnerability in computer systems, and despite numerous techniques and approaches proposed to protect programs from exploitation, many existing software and hardware-based prevention mechanisms have proven ineffective in mitigating this threat. The presented study introduces a lightweight prevention technique (LsStk). The approach involves implementing an encryption algorithm to safeguard the function return address, making it resilient to exploitation by adversaries. The algorithm integrated at the compiler level, utilizes epilogue and prologue functions to encrypt the return address and protect it from adversarial tampering. Given its minimal footprint, the solution is well-suited for adaptation in embedded IoT (Internet of Things) devices.To evaluate the effectiveness of our proposed technique, we conducted a statistical analysis using various 32-bit servers. We compared the performance of our method with existing prevention mechanisms. Our approach is designed to be adaptable at both the hardware CPU architecture level and as a software-level patch for the GCC compiler. Our analysis leads to the conclusion that our proposed technique significantly enhances security against such attacks.
Reconciling quality and timeliness in a sustainable open access model

It has taken seven years to bring this project to life, but STEMS is finally a reality, with the first issue published in 2023. STEMS is a peer-reviewed journal that publishes original research, reviews, and thought-provoking pieces on current research, policy, and topics that resonate with the community. The rapid release of preprints expedites timely knowledge dissemination, while our rigorous peer-review process safeguards against potential manuscript shortcomings. This is how it happened.
HWR200: New open access dataset of handwritten texts images in Russian

Handwritten text image datasets are highly useful for solving many problems using machine learning. Such problems include recognition of handwritten characters and handwriting, visual question answering, near-duplicate detection, search for text reuse in handwriting and many auxiliary tasks: highlighting lines, words, other objects in the text. The paper presents new dataset of handwritten texts images in Russian created by 200 writers with different handwriting and photographed in different environment1 . We described the procedure for creating this dataset and the requirements that were set for the texts and photos. The experiments with the baseline solution on fraud search and text reuse search problems showed results of results of 60% and 83% recall respectively and 5% and 2% false positive rate respectively on the dataset.
Hierarchical and Decentralised Federated Learning

Federated Learning (FL) is a recent approach for distributed Machine Learning (ML) where data are never communicated to a central node. Instead, an ML model (for example, a deep neural network) is initialized by a designated central (aggregation) node and shared with training nodes that have direct access to data of interest. These training nodes then perform small batches of training on their local data. Periodically, each training node submits ML model parameter/weight updates to the central node. The central node aggregates the parameters/weights to create a new global ML model that it then re-shares with the training nodes. This process can either take place indefinitely or be repeated until the ML model converges with respect to some evaluation metric (for example, mean average error, accuracy).
Zero-Shot Recommendations with Pre-Trained Large Language Models for Multimodal Nudging

We present a method for zero-shot recommendation of multimodal non-stationary content that leverages recent advancements in the field of generative AI. We propose rendering inputs of different modalities as textual descriptions and to utilize pre-trained LLMs to obtain their numerical representations by computing semantic embeddings. Once unified representations of all content items are obtained, the recommendation can be performed by computing an appropriate similarity metric between them without any additional learning. We demonstrate our approach on a synthetic multimodal nudging environment, where the inputs consist of tabular, textual, and visual data.
Deep Attention Q-Network for Personalized Treatment Recommendation

Tailoring treatment for severely ill patients is crucial yet challenging to achieve optimal healthcare outcomes. Recent advances in reinforcement learning offer promising personalized treatment recommendations. However, they often rely solely on a patient’s current physiological state, which may not accurately represent the true health status of the patient. This limitation hampers policy learning and evaluation, undermining the effectiveness of the treatment. In this study, we propose Deep Attention Q-Network for personalized treatment recommendation, leveraging the Transformer architecture within a deep reinforcement learning framework to efficiently integrate historical observations of patients. We evaluated our proposed method on two real-world datasets: sepsis and acute hypotension patients, demonstrating its superiority over state-of-the-art methods. The source code for our model is available at https://github.com/stevenmsm/RL-ICU-DAQN.