Miracle-dz/UniFAR-FTUs huggingface dataset:
Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instances

Despite the great progress of Visual Question Answering (VQA), current VQA models heavily rely on the superficial correlation between the question type and its corresponding frequent answers (i.e., language priors) to make predictions, without really understanding the input. In this work, we define the training instances with the same question type but different answers as superficially similar instances, and attribute the language priors to the confusion of VQA model on such instances. To solve this problem, we propose a novel training framework that explicitly encourages the VQA model to distinguish between the superficially similar instances. Specifically, for each training instance, we first construct a set that contains its superficially similar counterparts. Then we exploit the proposed distinguishing module to increase the distance between the instance and its counterparts in the answer space. In this way, the VQA model is forced to further focus on the other parts of the input beyond the question type, which helps to overcome the language priors. Experimental results show that our method achieves the state-of-the-art performance on VQA-CP v2. Codes are available at Distinguishing-VQA.
Valley-Spin Hall Effect-Based Nonvolatile Memory With Exchange-Coupling-Enabled Electrical Isolation of Read and Write Paths

Valley-spin hall (VSH) effect in monolayer WSe2 has been shown to exhibit highly beneficial features for nonvolatile memory (NVM) design. Key advantages of VSH-based magnetic random access memory (VSH-MRAM) over spin orbit torque (SOT)-MRAM include access transistor-less compact bit-cell and low-power switching of perpendicular magnetic anisotropy (PMA) magnets. Nevertheless, large device resistance in the read path (<inline-formula> <tex-math notation="LaTeX">$R_{S}$ </tex-math></inline-formula>) due to low mobility of WSe2 and Schottky contacts deteriorates sense margin (SM), offsetting the benefits of VSH-MRAM. To address this limitation, we propose another flavor of VSH-MRAM that (while inheriting most of the benefits of VSH-MRAM) achieves lower <inline-formula> <tex-math notation="LaTeX">$R_{S}$ </tex-math></inline-formula> in the read path by electrically isolating the read and write terminals. This is enabled by coupling VSH with electrically isolated but magnetically coupled PMA magnets via interlayer exchange coupling. Designing the proposed devices using object-oriented micromagnetic framework (OOMMF) simulation, we ensure the robustness of the exchange-coupled PMA system under process variations. To maintain a compact memory footprint, we share the read access transistor across multiple bit-cells. Compared with the existing VSH-MRAMs, our design achieves 39%–42% and 36%–46% reduction in read time and energy, respectively, along with <inline-formula> <tex-math notation="LaTeX">$1.1\times - 1.3\times $ </tex-math></inline-formula> larger SM at a comparable area. This comes at the cost of <inline-formula> <tex-math notation="LaTeX">$1.7\times $ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$2.0\times $ </tex-math></inline-formula> increase in write time and energy, respectively. Thus, the proposed design is suitable for applications in which reads are more dominant than writes.
ActiveNeRF: Learning where to See with Uncertainty Estimation

Recently, Neural Radiance Fields (NeRF) has shown promising performances on reconstructing 3D scenes and synthesizing novel views from a sparse set of 2D images. Albeit effective, the performance of NeRF is highly influenced by the quality of training samples. With limited posed images from the scene, NeRF fails to generalize well to novel views and may collapse to trivial solutions in unobserved regions. This makes NeRF impractical under resource-constrained scenarios. In this paper, we present a novel learning framework, ActiveNeRF, aiming to model a 3D scene with a constrained input budget. Specifically, we first incorporate uncertainty estimation into a NeRF model, which ensures robustness under few observations and provides an interpretation of how NeRF understands the scene. On this basis, we propose to supplement the existing training set with newly captured samples based on an active learning scheme. By evaluating the reduction of uncertainty given new inputs, we select the samples that bring the most information gain. In this way, the quality of novel view synthesis can be improved with minimal additional resources. Extensive experiments validate the performance of our model on both realistic and synthetic scenes, especially with scarcer training data. Code will be released at \url{https://github.com/LeapLabTHU/ActiveNeRF}.
Scene Graph Modification as Incremental Structure Expanding

A scene graph is a semantic representation that expresses the objects, attributes, and relationships between objects in a scene. Scene graphs play an important role in many cross modality tasks, as they are able to capture the interactions between images and texts. In this paper, we focus on scene graph modification (SGM), where the system is required to learn how to update an existing scene graph based on a natural language query. Unlike previous approaches that rebuilt the entire scene graph, we frame SGM as a graph expansion task by introducing the incremental structure expanding (ISE). ISE constructs the target graph by incrementally expanding the source graph without changing the unmodified structure. Based on ISE, we further propose a model that iterates between nodes prediction and edges prediction, inferring more accurate and harmonious expansion decisions progressively. In addition, we construct a challenging dataset that contains more complicated queries and larger scene graphs than existing datasets. Experiments on four benchmarks demonstrate the effectiveness of our approach, which surpasses the previous state-of-the-art model by large margins.
Improving Topic Segmentation by Injecting Discourse Dependencies

Recent neural supervised topic segmentation models achieve distinguished superior effectiveness over unsupervised methods, with the availability of large-scale training corpora sampled from Wikipedia. These models may, however, suffer from limited robustness and transferability caused by exploiting simple linguistic cues for prediction, but overlooking more important inter-sentential topical consistency. To address this issue, we present a discourse-aware neural topic segmentation model with the injection of above-sentence discourse dependency structures to encourage the model make topic boundary prediction based more on the topical consistency between sentences. Our empirical study on English evaluation datasets shows that injecting above-sentence discourse structures to a neural topic segmenter with our proposed strategy can substantially improve its performances on intra-domain and out-of-domain data, with little increase of model’s complexity.
A-priori analysis of the quasicontinuum method in one dimension

The quasicontinuum method is a coarse-graining technique for reducing the complexity of atomistic simulations in a static and quasistatic setting. In this paper we give an a-priori error analysis for the quasicontinuum method in one dimension. We consider atomistic models with Lennard-Jones type long range interactions and a practical QC formulation. First, we prove the existence, the local uniqueness and the stability with respect to discrete W1,∞-norm of elastic and fractured atomistic solutions. We then used a fixed point technique to prove the existence of quasicontinuum approximation which satisfies an optimal a-priori error bound. The first-named author acknowledges the financial support received from the European research project HPRN-CT-2002-00284: New Materials, Adaptive Systems and their Nonlinearities. Modelling, Control and Numerical Simulation, and the kind hospitality of Carlo Lovadina (University of Pavia).
Discriminative Sampling of Proposals in Self-Supervised Transformers for Weakly Supervised Object Localization

Drones are employed in a growing number of visual recognition applications. A recent development in cell tower inspection is drone-based asset surveillance, where the autonomous flight of a drone is guided by localizing objects of interest in successive aerial images. In this paper, we propose a method to train deep weakly-supervised object localization (WSOL) models based only on image-class labels to locate object with high confidence. To train our localizer, pseudo labels are efficiently harvested from a self-supervised vision transformers (SSTs). However, since SSTs decompose the scene into multiple maps containing various object parts, and do not rely on any explicit super-visory signal, they cannot distinguish between the object of interest and other objects, as required WSOL. To address this issue, we propose leveraging the multiple maps generated by the different transformer heads to acquire pseudo-labels for training a deep WSOL model. In particular, a new Discriminative Proposals Sampling (DiPS) method is introduced that relies on a CNN classifier to identify discriminative regions. Then, foreground and background pixels are sampled from these regions in order to train a WSOL model for generating activation maps that can accurately localize objects belonging to a specific class. Empirical results11Our code is available: https://github.com/shakeebmurtaza/dips on the challenging TelDrone dataset indicate that our proposed approach can outperform state-of-art methods over a wide range of threshold values over produced maps. We also computed results on CUB dataset, showing that our method can be adapted for other tasks.
Panoramic Vision Transformer for Saliency Detection in 360° Videos

360$^\circ$ video saliency detection is one of the challenging benchmarks for 360$^\circ$ video understanding since non-negligible distortion and discontinuity occur in the projection of any format of 360$^\circ$ videos, and capture-worthy viewpoint in the omnidirectional sphere is ambiguous by nature. We present a new framework named Panoramic Vision Transformer (PAVER). We design the encoder using Vision Transformer with deformable convolution, which enables us not only to plug pretrained models from normal videos into our architecture without additional modules or finetuning but also to perform geometric approximation only once, unlike previous deep CNN-based approaches. Thanks to its powerful encoder, PAVER can learn the saliency from three simple relative relations among local patch features, outperforming state-of-the-art models for the Wild360 benchmark by large margins without supervision or auxiliary information like class activation. We demonstrate the utility of our saliency prediction model with the omnidirectional video quality assessment task in VQA-ODV, where we consistently improve performance without any form of supervision, including head movement.
The Bogomolov--Pantev resolution, an expository account

Before the work cited above, and that of Abramovich and de Jong [1] (appearing at roughly the same time) the only proof of this theorem was as a corollary of the famous result of Hironaka [5]. These new proofs were inspired by the recent work of de Jong [6], which Bogomolov and Pantev combine with a beautiful idea of Belyi [2] “simplifying” the ramification locus of a covering of P by successively folding up the P onto itself, over a fixed base. This latter step unfortunately only works in characteristic zero, limiting the scope of the argument (Abramovich and de Jong’s paper gives some results even in characteristic p). Hence, we work over the field of complex numbers; the argument also works (with suitable modifications about rationality) over any field of characteristic zero. The outline of the argument we follow is the same as that of the paper of Bogomolov and Pantev; however we offer different (and we hope simpler) proofs of the corresponding lemmas. To begin with, their argument using Grassmannians is replaced by an application of Noether normalisation in Section 1. Belyi’s argument to reduce the degree of individual components of the ramification locus is presented in purely algebraic form in Section 1, Lemmas 3 and 4. The presentation of Bogomolov and Pantev refers to “semi-stable families of pointed curves
DifferSketching

Multiple sketch datasets have been proposed to understand how people draw 3D objects. However, such datasets are often of small scale and cover a small set of objects or categories. In addition, these datasets contain freehand sketches mostly from expert users, making it difficult to compare the drawings by expert and novice users, while such comparisons are critical in informing more effective sketch-based interfaces for either user groups. These observations motivate us to analyze how differently people with and without adequate drawing skills sketch 3D objects. We invited 70 novice users and 38 expert users to sketch 136 3D objects, which were presented as 362 images rendered from multiple views. This leads to a new dataset of 3,620 freehand multi-view sketches, which are registered with their corresponding 3D objects under certain views. Our dataset is an order of magnitude larger than the existing datasets. We analyze the collected data at three levels, i.e., sketch-level, stroke-level, and pixel-level, under both spatial and temporal characteristics, and within and across groups of creators. We found that the drawings by professionals and novices show significant differences at stroke-level, both intrinsically and extrinsically. We demonstrate the usefulness of our dataset in two applications: (i) freehand-style sketch synthesis, and (ii) posing it as a potential benchmark for sketch-based 3D reconstruction. Our dataset and code are available at https://chufengxiao.github. io/DifferSketching/.
RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection

The task of out-of-distribution (OOD) detection is crucial for deploying machine learning models in real-world settings. In this paper, we observe that the singular value distributions of the in-distribution (ID) and OOD features are quite different: the OOD feature matrix tends to have a larger dominant singular value than the ID feature, and the class predictions of OOD samples are largely determined by it. This observation motivates us to propose \texttt{RankFeat}, a simple yet effective \texttt{post hoc} approach for OOD detection by removing the rank-1 matrix composed of the largest singular value and the associated singular vectors from the high-level feature (\emph{i.e.,} $\mathbf{X}{-} \mathbf{s}_{1}\mathbf{u}_{1}\mathbf{v}_{1}^{T}$). \texttt{RankFeat} achieves the \emph{state-of-the-art} performance and reduces the average false positive rate (FPR95) by 17.90\% compared with the previous best method. Extensive ablation studies and comprehensive theoretical analyses are presented to support the empirical results.
"Guess what I'm doing": Extending legibility to sequential decision tasks

In this paper we investigate the notion of legibility in sequential decision tasks under uncertainty. Previous works that extend legibility to scenarios beyond robot motion either focus on deterministic settings or are computationally too expensive. Our proposed approach, dubbed PoL-MDP, is able to handle uncertainty while remaining computationally tractable. We establish the advantages of our approach against state-of-the-art approaches in several simulated scenarios of different complexity. We also showcase the use of our legible policies as demonstrations for an inverse reinforcement learning agent, establishing their superiority against the commonly used demonstrations based on the optimal policy. Finally, we assess the legibility of our computed policies through a user study where people are asked to infer the goal of a mobile robot following a legible policy by observing its actions.
Generating detailed saliency maps using model-agnostic methods

The emerging field of Explainable Artificial Intelligence focuses on researching methods of explaining the decision making processes of complex machine learning models. In the field of explainability for Computer Vision, explanations are provided as saliency maps, which visualize the importance of individual pixels of the input to the model’s prediction. In this work we focus on a perturbation-based, model-agnostic explainability method called RISE and propose two modifications replacement of square occlusions with polygonal occlusions based on cells of a Voronoi mesh, and addition of an informativeness guarantee to the occlusion mask generator. These modifications, collectively called VRISE (Voronoi-RISE), are meant to, respectively, improve the accuracy of maps generated using large occlusions, and accelerate convergence of saliency maps in cases where sampling density is set to extreme values. We perform a quantitative comparison of accuracy of saliency maps produced by VRISE and RISE on the validation split of ILSVRC2012 using a saliency-guided content insertion/deletion metric and a pointing metric based on bounding boxes. Our results show a modest but universal improvement of accuracy measured by insertion/deletion metric, but performance in pointing game is improved only in configurations using large occlusions. We also notice greater improvement for images containing multiple object instances. Additionally, we explore the space of configurable parameters for RISE and VRISE on a small dataset and make observations regarding relationships between configuration of the algorithms and accuracy of results. We also describe and demonstrate two effects observed over the course of experimentation, stemming from the random sampling approach of RISE: "feature slicing" and "saliency misattribution"
MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning

Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL). However, non-stationarity remains a significant challenge in fully decentralized learning. In the paper, we tackle the non-stationarity problem in the simplest and fundamental way and propose multi-agent alternate Q-learning (MA2QL), where agents take turns updating their Q-functions by Q-learning. MA2QL is a minimalist approach to fully decentralized cooperative MARL but is theoretically grounded. We prove that when each agent guarantees $\varepsilon$-convergence at each turn, their joint policy converges to a Nash equilibrium. In practice, MA2QL only requires minimal changes to independent Q-learning (IQL). We empirically evaluate MA2QL on a variety of cooperative multi-agent tasks. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.
Inference of nonlinear causal effects with GWAS summary data

Large-scale genome-wide association studies (GWAS) have offered an exciting opportunity to discover putative causal genes or risk factors associated with diseases by using SNPs as instrumental variables (IVs). However, conventional approaches assume linear causal relations partly for simplicity and partly for the availability of GWAS summary data. In this work, we propose a novel model {for transcriptome-wide association studies (TWAS)} to incorporate nonlinear relationships across IVs, an exposure/gene, and an outcome, which is robust against violations of the valid IV assumptions, permits the use of GWAS summary data, and covers two-stage least squares as a special case. We decouple the estimation of a marginal causal effect and a nonlinear transformation, where the former is estimated via sliced inverse regression and a sparse instrumental variable regression, and the latter is estimated by a ratio-adjusted inverse regression. On this ground, we propose an inferential procedure. An application of the proposed method to the ADNI gene expression data and the IGAP GWAS summary data identifies 18 causal genes associated with Alzheimer's disease, including APOE and TOMM40, in addition to 7 other genes missed by two-stage least squares considering only linear relationships. Our findings suggest that nonlinear modeling is required to unleash the power of IV regression for identifying potentially nonlinear gene-trait associations. Accompanying this paper is our Python library \texttt{nl-causal} (\url{https://nonlinear-causal.readthedocs.io/}) that implements the proposed method.
Bayesian Power Spectrum Analysis of the First-Year Wilkinson Microwave Anisotropy Probe Data

We present the first results from a Bayesian analysis of the Wilkinson Microwave Anisotropy Probe (WMAP) first-year data using a Gibbs sampling technique. Using two independent, parallel supercomputer codes, we analyze the WMAP Q, V, and W bands. The analysis results in a full probabilistic description of the information the WMAP data set contains about the power spectrum and the all-sky map of the cosmic microwave background anisotropies. We present the complete probability distributions for each Cℓ, including any non-Gaussianities of the power spectrum likelihood. While we find good overall agreement with the previously published WMAP spectrum, our analysis uncovers discrepancies in the power spectrum estimates at low-ℓ multipoles. For example, we claim that the best-fit ΛCDM model is consistent with the C2 inferred from our combined Q+V+W analysis, with a 10% probability of an even larger theoretical C2. Based on our exact analysis, we can therefore attribute the "low-quadrupole issue" to a statistical fluctuation.
FR: Folded Rationalization with a Unified Encoder

Conventional works generally employ a two-phase model in which a generator selects the most important pieces, followed by a predictor that makes predictions based on the selected pieces. However, such a two-phase model may incur the degeneration problem where the predictor overfits to the noise generated by a not yet well-trained generator and in turn, leads the generator to converge to a sub-optimal model that tends to select senseless pieces. To tackle this challenge, we propose Folded Rationalization (FR) that folds the two phases of the rationale model into one from the perspective of text semantic extraction. The key idea of FR is to employ a unified encoder between the generator and predictor, based on which FR can facilitate a better predictor by access to valuable information blocked by the generator in the traditional two-phase model and thus bring a better generator. Empirically, we show that FR improves the F1 score by up to 10.3% as compared to state-of-the-art methods.
Learning to Answer Semantic Queries over Code

During software development, developers need answers to queries about semantic aspects of code. Even though extractive question-answering using neural approaches has been studied widely in natural languages, the problem of answering semantic queries over code using neural networks has not yet been explored. This is mainly because there is no existing dataset with extractive question and answer pairs over code involving complex concepts and long chains of reasoning. We bridge this gap by building a new, curated dataset called CodeQueries, and proposing a neural question-answering methodology over code. We build upon state-of-the-art pre-trained models of code to predict answer and supporting-fact spans. Given a query and code, only some of the code may be relevant to answer the query. We ﬁrst experiment under an ideal setting where only the relevant code is given to the model and show that our models do well. We then experiment under three pragmatic considerations: (1) scaling to large-size code, (2) learning from a limited number of examples and (3) robustness to minor syntax errors in code. Our results show that while a neural model can be resilient to minor syntax errors in code, increasing size of code, presence of code that is not relevant to the query, and reduced number of training examples limit the model performance. We are releasing our data and models 1 to facilitate future work on the proposed problem of answering semantic queries over code.