Reviewerly/exHarmony huggingface dataset:
FragmGAN: generative adversarial nets for fragmentary data imputation and prediction

Modern scientific research and applications very often encounter 'fragmentary data' which brings big challenges to imputation and prediction. By leveraging the structure of response patterns, we propose a unified and flexible framework based on Generative Adversarial Nets (GAN) to deal with fragmentary data imputation and label prediction at the same time. Unlike most of the other generative model based imputation methods that either have no theoretical guarantee or only consider Missing Completed At Random (MCAR), the proposed FragmGAN has theoretical guarantees for imputation with data Missing At Random (MAR) while no hint mechanism is needed. FragmGAN trains a predictor with the generator and discriminator simultaneously. This linkage mechanism shows significant advantages for predictive performances in extensive experiments.
Hi-SIGIR: Hierachical Semantic-Guided Image-to-image Retrieval via Scene Graph

Image-to-image retrieval, a fundamental task, aims at matching similar images based on a query image. Existing methods with convolutional neural networks are usually sensitive to low-level visual features, and ignore high-level semantic relationship information. This makes retrieving complicated images with multiple objects and various relationships a significant challenge. Although some works introduce the scene graph to capture the global semantic features of the objects and their relations, they ignore the local visual representations. In addition, due to the fragility of individual modal representations, poisoning attacks in adversarial scenarios are easily achieved, hurting the robustness of the visual-guided foundation image retrieval model. To overcome these issues, we propose a novel hierarchical semantic-guided image-to-image retrieval method via scene graph, called Hi-SIGIR. Specifically, to begin with, our proposed method generates the scene graph of an image. Then, our model extracts and learns both the visual and semantic features of the nodes and relations within the scene graphs. Next, these features are fused to obtain local information and sent to the graph neural network to obtain global information. Using these information, the similarity between the scene graphs of several images is calculated at both the local and global levels to perform image retrieval. Finally, we introduce a surrogate that calculates relevance in a cross-modal manner to understand image content better. Experimental evaluations on several wildly-used benchmarks demonstrate the superiority of the proposed method.
A Symbolic Characters Aware Model for Solving Geometry Problems

AI has made significant progress in solving math problems, but geometry problems remain challenging due to their reliance on both text and diagrams. In the text description, symbolic characters such as "ABC" often serve as a bridge to connect the corresponding diagram. However, by simply tokenizing symbolic characters into individual letters (e.g., 'A', 'B' and 'C'), existing works fail to study them explicitly and thus lose the semantic relationship with the diagram. In this paper, we develop a symbolic character-aware model to fully explore the role of these characters in both text and diagram understanding and optimize the model under a multi-modal reasoning framework. In the text encoder, we propose merging individual symbolic characters to form one semantic unit along with geometric information from the corresponding diagram. For the diagram encoder, we pre-train it under a multi-label classification framework with the symbolic characters as labels. In addition, we enhance the geometry diagram understanding ability via a self-supervised learning method under the masked image modeling auxiliary task. By integrating the proposed model into a general encoder-decoder pipeline for solving geometry problems, we demonstrate its superiority on two benchmark datasets, including GeoQA and Geometry3K, with extensive experiments. Specifically, on GeoQA, the question-solving accuracy is increased from 60.0% to 64.1%, achieving a new state-of-the-art accuracy; on Geometry3K, we reduce the question average solving steps from 6.9 down to 6.0 with marginally higher solving accuracy.
Decoupled Cross-Scale Cross-View Interaction for Stereo Image Enhancement in the Dark

Low-light stereo image enhancement (LLSIE) aims at improving the visual quality of stereo images captured in dark conditions. However, existing methods have shown limited success in detail recovery and illumination adjustment. This can be attributed to two main factors: 1) insufficient single-scale inter-view interaction hinders the exploitation of valuable cross-view cues; 2) lacking long-range dependency leads to the inability to deal with the spatial long-range effects caused by illumination degradation. To address these limitations, we propose a novel LLSIE model named Decoupled Cross-Scale Cross-View Interaction Network (DCI-Net). Our model introduces a key component called the Decoupled Interaction Module (DIM) designed to promote sufficient dual-view information exchange. DIM decouples the dual-view information interaction by discovering multi-scale cross-view correlations and further exploring cross-scale information flow. Furthermore, we present Spatial-channel Information Mining Block (SIMB) for intra-view feature extraction, and the benefits are twofold. One is long-range dependency capture to build spatial long-range relationship, and the other is expanded channel information refinement that enhances information flow in the channel dimension. Extensive experiments conducted on Flickr1024, KITTI 2012, KITTI 2015, and Middlebury datasets show that our method obtains better illumination adjustment and detail recovery, and achieves SOTA performance compared to other related methods.
High Fidelity Face Swapping via Semantics Disentanglement and Structure Enhancement

In this paper, we propose a novel Semantics and Structure-aware face Swapping framework (S2Swap) that exploits semantics disentanglement and structure enhancement for high fidelity face generation. Different from previous methods that either 1) suffer from degraded generation fidelity due to insufficient identity-attributes disentanglement or 2) neglect the importance of structure information for identity consistency, our approach can achieve local facial semantics disentanglement beyond global identity while boosting identity consistency through structure enhancement. Specifically, to achieve identity-attributes disentanglement, our S2Swap is designed from global-local perspectives. Firstly, an Oriented Identity Transfer module is proposed to globally disentangle target identity and attributes under global identity semantics prior. Such global disentanglement enables source identity transfer to the individual target identity. Secondly, a Local Semantics Disentanglement module is devised to disentangle local identity and identity-irrelevant facial semantics, providing local semantic compensation for the global counterpart. Moreover, to boost identity consistency, a Structure-Aware Head Modeling module is introduced to provide the desired face structure enhancement through an intuitive face sketch. Finally, considering the identity-attributes trade-off, we adaptively integrate semantics and structure information in a self-learning manner. Extensive experiments qualitatively and quantitatively show that our method outperforms SOTA face swapping methods in terms of both identity transfer and attribute preservation.
Learning Occlusion Disentanglement with Fine-grained Localization for Occluded Person Re-identification

Person re-identification (Re-ID) has been extensively investigated in recent years. However, many existing paradigms rely on holistic person regions for matching, disregarding the challenges posed by occlusions in real-world scenarios. Recent methods have explored occlusion augmentation or external semantic cues. Nevertheless, these approaches tend to be coarse-grained, discarding valuable semantic information in local regions when determining them as occlusions. In this paper, we propose a Fine-grained Occlusion Disentanglement Network (FODN) that can extract more information from limited person regions. Specifically, we propose a fine-grained occlusion augmentation scheme to generate diverse occlusion data and employ bilinear interpolation and downsampling strategies to obtain fine-grained occlusion labels. We then design an occlusion feature disentanglement Module that decouples norm and angle from features and supervises the occlusion-aware task using the aforementioned occlusion labeling and person re-identification tasks, respectively, resulting in more robust features. Additionally, we propose a dynamic local weight controller to balance the relative importance of various human body parts, thereby improving the model's ability to mine more effective local features from limited human body regions after occlusion removal. Comprehensive experiments on various person Re-ID benchmarks demonstrate the superiority of FODN over state-of-the-art methods.
Automatic Generation of Commercial Scenes

Commercial scenes such as markets and shops are everyday scenes for both virtual scenes and real-world interior designs. However, existing literature on interior scene synthesis mainly focuses on formulating and optimizing residential scenes such as bedrooms, living rooms, etc. Existing literature typically presents a set of relations among objects. It recognizes each furniture object as the smallest unit while optimizing a residential room. However, object relations become less critical in commercial scenes since shelves are often placed next to each other so pre-calculated relations of objects are less needed. Instead, interior designers resort to evaluating how groups of objects perform in commercial scenes, i.e., the smallest unit to be evaluated is a group of objects. This paper presents a system automatically synthesizes market-like commercial scenes in virtual environments. Following the rules of commercial layout design, we parameterize groups of objects as "patterns" contributing to a scene. Each pattern directly yields a human-centric routine locally, provides potential connectivity with other routines, and derives the arrangements of objects concerning itself according to the assigned parameters. In order to optimize a scene, the patterns are iteratively multiplexed to insert new routines or modify existing ones under a set of constraints derived from commercial layout designs. Through extensive experiments, we demonstrate the ability of our framework to generate plausible and practical commercial scenes.
SoK: A First Order Survey of Quantum Supply Dynamics and Threat Landscapes

Quantum computing, with its transformative computational potential, is gaining prominence in the technological landscape. As a new and exotic technology, quantum computers involve innumerable Intellectual Property (IP) in the form of fabrication recipe, control electronics and software techniques, to name a few. Furthermore, complexity of quantum systems necessitates extensive involvement of third party tools, equipment and services which could risk the IPs and the Quality of Service and enable other attack surfaces. This paper is a first attempt to explore the quantum computing ecosystem, from the fabrication of quantum processors to the development of specialized software tools and hardware components, from a security perspective. By investigating the publicly disclosed information from industry front runners like IBM, Google, Honeywell and more, we piece together various components of quantum computing supply chain. We also uncover some potential vulnerabilities and attack models and suggest defenses. We highlight the need to scrutinize the quantum computing supply chain further through the lens of security.
An Analysis of the RankSign Signature Scheme with Rank Multipliers

We investigate the application of rank multipliers as introduced by Loidreau to repair the 2017 NIST proposal RankSign. In RankSign, a signature is generated by interpreting the message to be signed as the syndrome of a low-rank parity-check (LRPC) code. Through knowledge of $$\mathcal {F}$$ , the low-dimensional subspace of $$\mathbb {F}_{q^m}\,$$ from which the elements of the parity-check matrix are drawn, this syndrome can be decoded to a low-rank signature $$\textbf{e}$$ . Thus, the security of RankSign crucially relies on the obfuscation of $$\mathcal {F}$$ , ideally making the public code indistinguishable from a random code. Unfortunately, RankSign was broken shortly after its submission by an attack exploiting low-rank codewords in the public code due to the right scrambler being chosen over the base field $$\mathbb {F}_{q}\,$$ . We propose to adapt RankSign by using the inverse of a rank multiplier as a right scrambler, increasing the minimum distance of both the public code $$\mathcal {C}_{pub}$$ and its dual $$\mathcal {C}_{pub}^{\perp }$$ . With this change, the public code contains significantly fewer codewords of low-rank weight, thus preventing the attack that broke the initial RankSign proposal.
Local Consensus Enhanced Siamese Network with Reciprocal Loss for Two-view Correspondence Learning

Recent studies of two-view correspondence learning usually establish an end-to-end network to jointly predict correspondence reliability and relative pose. We improve such a framework from two aspects. First, we propose a Local Feature Consensus (LFC) plugin block to augment the features of existing models. Given a correspondence feature, the block augments its neighboring features with mutual neighborhood consensus and aggregates them to produce an enhanced feature. As inliers obey a uniform cross-view transformation and share more consistent learned features than outliers, feature consensus strengthens inlier correlation and suppresses outlier distraction, which makes output features more discriminative for classifying inliers/outliers. Second, existing approaches supervise network training with the ground truth correspondences and essential matrix projecting one image to the other for an input image pair, without considering the information from the reverse mapping. We extend existing models to a Siamese network with a reciprocal loss that exploits the supervision of mutual projection, which considerably promotes the matching performance without introducing additional model parameters. Building upon MSA-Net [30], we implement the two proposals and experimentally achieve state-of-the-art performance on benchmark datasets.
Improving the adversarial robustness of quantized neural networks via exploiting the feature diversity

Quantized neural networks (QNNs) have become one of the most prevalent approaches in deep learning model compression due to their computational and storage efficiency. However, there is a lack of research specialized in the adversarial robustness of QNNs, which is important for applications in security-critical domains. Existing defenses focus on conventional full-precision networks, which can result in behavioral disparities and degrade the expected performance when directly transferred to QNNs. A novel defensive strategy promotes feature diversity through an orthogonal constraint, which can synergize well with quantization. Inspired by this intuition, we propose an orthogonal regularization with quantization to improve the adversarial robustness of QNNs in this paper. Moreover, we observe that quantization serves as an implicit regularization and is able to alleviate orthogonal degeneration. The proposed orthogonal regularization with quantization is validated on several typical network architectures and benchmark datasets. The results demonstrate that the proposed method can notably enhance adversarial robustness against both white-box and black-box attacks.
Educational Seismology through an Immersive Virtual Reality Game: Design, Development and Pilot Evaluation of User Experience

Virtual Reality (VR), especially in its immersive form, is a promising technology employed to support education and training in various fields. VR offers unique opportunities to experience situations and environments that are otherwise inaccessible or risky. Educational Seismology aims to inform and educate the public on earthquakes; to this end, the use of VR is investigated as an attractive solution. VRQuake is an immersive VR application designed and developed for Educational Seismology purposes. VRQuake is structured in five consecutive scenes and is organized as a game. It allows users to interact with virtual objects in real time and apply learned rules and good practices in reaction to an earthquake, thus providing a dynamic learning environment. A pilot evaluation of VRQuake is performed by volunteer university students who play the game and then answer a questionnaire with closed- and open-type questions referring mostly to the user experience. Analysis of the answers has shown positive results regarding usability, clarity and acceptance of the application. Answers are also encouraging as to the educational potential of VRQuake. Furthermore, qualitative analysis of open-type questions has contributed user suggestions and demands that point to interesting new directions for further improvement of user experience and learning outcomes.
Filling in the Blank: Rationale-Augmented Prompt Tuning for TextVQA

Recently, generative Text-based visual question answering (TextVQA) methods, which are often based on language models, have exhibited impressive results and drawn increasing attention. However, due to the inconsistencies in both input forms and optimization objectives, the power of pretrained language models is not fully explored, resulting in the need for large amounts of training data. In this work, we rethink the characteristics of the TextVQA task and find that scene text is indeed a special kind of language embedded in images. To this end, we propose a text-centered generative framework FITB (stands for Filling In The Blank), in which multimodal information is mainly represented in textual form and rationale-augmented prompting is involved. Specifically, an infilling-based prompt strategy is utilized to formulate TextVQA as a novel problem of filling in the blank with proper scene text according to the language context. Furthermore, aiming to prevent the model from language bias overfitting, we design a rough answer grounding module to provide visual rationales for promoting multimodal reasoning. Extensive experiments verify the superiority of FITB in both fully-supervised and zero-shot/few-shot settings. Notably, even with a saving of about 64M data, FITB surpasses the state-of-the-art method by 3.00% and 1.99% on TextVQA and ST-VQA datasets, respectively.
UER: A Heuristic Bias Addressing Approach for Online Continual Learning

Online continual learning aims to continuously train neural networks from a continuous data stream with a single pass-through data. As the most effective approach, the rehearsal-based methods replay part of previous data. Commonly used predictors in existing methods tend to generate biased dot-product logits that prefer to the classes of current data, which is known as a bias issue and a phenomenon of forgetting. Many approaches have been proposed to overcome the forgetting problem by correcting the bias; however, they still need to be improved in online fashion. In this paper, we try to address the bias issue by a more straightforward and more efficient method. By decomposing the dot-product logits into an angle factor and a norm factor, we empirically find that the bias problem mainly occurs in the angle factor, which can be used to learn novel knowledge as cosine logits. On the contrary, the norm factor abandoned by existing methods helps remember historical knowledge. Based on this observation, we intuitively propose to leverage the norm factor to balance the new and old knowledge for addressing the bias. To this end, we develop a heuristic approach called unbias experience replay (UER). UER learns current samples only by the angle factor and further replays previous samples by both the norm and angle factors. Extensive experiments on three datasets show that UER achieves superior performance over various state-of-the-art methods. The code is in https://github.com/FelixHuiweiLin/UER.
Mask-Guided Progressive Network for Joint Raindrop and Rain Streak Removal in Videos

Videos captured in rainy weather are unavoidably corrupted by both rain streaks and raindrops in driving scenarios, and it is desirable and challenging to recover background details obscured by rain streaks and raindrops. However, existing video rain removal methods often address either video rain streak removal or video raindrop removal, thereby suffer from degraded performance when deal with both simultaneously. The bottleneck is a lack of a video dataset, where each video frame contains both rain streaks and raindrops. To address this issue, we in this work generate a synthesized dataset, namely VRDS, with 102 rainy videos from diverse scenarios, and each video frame has the corresponding rain streak map, raindrop mask, and the underlying rain-free clean image (ground truth). Moreover, we devise a mask-guided progressive video deraining network (ViMP-Net) to remove both rain streaks and raindrops of each video frame. Specifically, we develop an intensity-guided alignment block to predict the rain streak intensity map and remove the rain streaks of the input rainy video at the first stage. Then, we predict a raindrop mask and pass it into a devised mask-guided dual transformer block to learn inter-frame and intra-frame transformer features, which are then fed into a decoder for further eliminating raindrops. Experimental results demonstrate that our ViMP-Net outperforms state-of-the-art methods on our synthetic dataset and real-world rainy videos. Our code is available at https://github.com/TonyHongtaoWu/ViMP-Net.
Cross-modal Unsupervised Domain Adaptation for 3D Semantic Segmentation via Bidirectional Fusion-then-Distillation

Cross-modal Unsupervised Domain Adaptation (UDA) becomes a research hotspot because it reduces the laborious annotation of target domain samples. Existing methods only mutually mimic the outputs of cross-modality in each domain, which enforces the class probability distribution agreeable in different domains. However, these methods ignore the complementarity brought by the modality fusion representation in cross-modal learning. In this paper, we propose a cross-modal UDA method for 3D semantic segmentation via Bidirectional Fusion-then-Distillation, named BFtD-xMUDA, which explores cross-modal fusion in UDA and realizes distribution consistency between outputs of two domains not only for 2D image and 3D point cloud but also for 2D/3D and fusion. Our method contains three significant components: Model-agnostic Feature Fusion Module (MFFM), Bidirectional Distillation (B-Distill), and Cross-modal Debiased Pseudo-Labeling (xDPL). MFFM is employed to generate cross-modal fusion features for establishing a latent space, which enforces maximum correlation and complementarity between two heterogeneous modalities. B-Distill is introduced to exploit bidirectional knowledge distillation which includes cross-modality and cross-domain fusion distillation, and well-achieving domain-modality alignment. xDPL is designed to model the uncertainty of pseudo-labels by self-training scheme. Extensive experimental results demonstrate that our method outperforms state-of-the-art competitors in several adaptation scenarios.