Miracle-dz/UniFAR-FTUs huggingface dataset:
SepTr: Separable Transformer for Audio Spectrogram Processing

Following the successful application of vision transformers in multiple computer vision tasks, these models have drawn the attention of the signal processing community. This is because signals are often represented as spectrograms (e.g. through Discrete Fourier Transform) which can be directly provided as input to vision transformers. However, naively applying transformers to spectrograms is suboptimal. Since the axes represent distinct dimensions, i.e. frequency and time, we argue that a better approach is to separate the attention dedicated to each axis. To this end, we propose the Separable Transformer (SepTr), an architecture that employs two transformer blocks in a sequential manner, the first attending to tokens within the same time interval, and the second attending to tokens within the same frequency bin. We conduct experiments on three benchmark data sets, showing that our separable architecture outperforms conventional vision transformers and other state-of-the-art methods. Unlike standard transformers, SepTr linearly scales the number of trainable parameters with the input size, thus having a lower memory footprint. Our code is available as open source at https://github.com/ristea/septr.
Domain Generalization by Mutual-Information Regularization with Pre-trained Models

Domain generalization (DG) aims to learn a generalized model to an unseen target domain using only limited source domains. Previous attempts to DG fail to learn domain-invariant representations only from the source domains due to the significant domain shifts between training and test domains. Instead, we re-formulate the DG objective using mutual information with the oracle model, a model generalized to any possible domain. We derive a tractable variational lower bound via approximating the oracle model by a pre-trained model, called Mutual Information Regularization with Oracle (MIRO). Our extensive experiments show that MIRO significantly improves the out-of-distribution performance. Furthermore, our scaling experiments show that the larger the scale of the pre-trained model, the greater the performance improvement of MIRO. Source code is available at https://github.com/kakaobrain/miro.
Estimation for the reaction term in semi-linear SPDEs under small diffusivity

We consider the estimation of a non-linear reaction term in the stochastic heat or more generally in a semi-linear stochastic partial differential equation (SPDE). Consistent inference is achieved by studying a small diffusivity level, which is realistic in applications. Our main result is a central limit theorem for the estimation error of a parametric estimator, from which confidence intervals can be constructed. Statistical efficiency is demonstrated by establishing local asymptotic normality. The estimation method is extended to local observations in time and space, which allows for non-parametric estimation of a reaction intensity varying in time and space. Furthermore, discrete observations in time and space can be handled. The statistical analysis requires advanced tools from stochastic analysis like Malliavin calculus for SPDEs, the infinite-dimensional Gaussian Poincar\'e inequality and regularity results for SPDEs in $L^p$-interpolation spaces.
Domain Representative Keywords Selection: A Probabilistic Approach

We propose a probabilistic approach to select a subset of a target domain representative keywords from a candidate set, contrasting with a context domain. Such a task is crucial for many downstream tasks in natural language processing. To contrast the target domain and the context domain, we adapt the two-component mixture model concept to generate a distribution of candidate keywords. It provides more importance to the distinctive keywords of the target domain than common keywords contrasting with the context domain. To support the representativeness of the selected keywords towards the target domain, we introduce an optimization algorithm for selecting the subset from the generated candidate distribution. We have shown that the optimization algorithm can be efficiently implemented with a near-optimal approximation guarantee. Finally, extensive experiments on multiple domains demonstrate the superiority of our approach over other baselines for the tasks of keyword summary generation and trending keywords selection.
Single-image Human-body Reshaping with Deep Neural Networks

In this paper, we present NeuralReshaper, a novel method for semantic reshaping of human bodies in single images using deep generative networks. To achieve globally coherent reshaping effects, our approach follows a ﬁt-then-reshape pipeline, which ﬁrst ﬁts a parametric 3D human model to a source human image and then reshapes the ﬁtted 3D model with respect to user-speciﬁed semantic attributes. Previous methods rely on image warping to transfer 3D reshaping effects to the entire image domain and thus often cause distortions in both foreground and background. Instead, to achieve more realistic reshaping results, we resort to generative adversarial nets conditioned on the source image and a 2D warping ﬁeld induced by the re-shaped 3D model. Speciﬁcally, we separately encode the foreground and background information in the source image using a two-headed U-net-like generator, and guide the information ﬂow from the foreground branch to the background branch via feature space warping. Furthermore, to deal with the lack-of-data problem that no paired data exist (i.e., the same human bodies in varying shapes), we introduce a novel weakly-supervised strategy to train our network. Besides, unlike previous methods that often require manual efforts to correct undesirable artifacts caused by incorrect body-to-image ﬁtting, our method is fully automatic. Extensive experiments on both indoor and outdoor datasets demonstrate the superiority of our method over previous approaches.
A Track-Wise Ensemble Event Independent Network for Polyphonic Sound Event Localization and Detection

Polyphonic sound event localization and detection (SELD) aims at detecting types of sound events with corresponding temporal activities and spatial locations. In this paper, a trackwise ensemble event independent network with a novel data augmentation method is proposed. The proposed model is based on our previous proposed Event-Independent Network V2 and is extended by conformer blocks and dense blocks. The track-wise ensemble model with track-wise output format is proposed to solve an ensemble model problem for track-wise output format that track permutation may occur among different models. The data augmentation approach contains several data augmentation chains, which are composed of random combinations of several data augmentation operations. The method also utilizes log-mel spectrograms, intensity vectors, and Spatial Cues-Augmented Log-Spectrogram (SALSA) for different models. We evaluate our proposed method in the Task of the L3DAS22 challenge and obtain the top ranking solution with a location-dependent F-score to be 0.699. Source code is released1.
A 3D Molecule Generative Model for Structure-Based Drug Design

We study a fundamental problem in structure-based drug design — generating molecules that bind to specific protein binding sites. While we have witnessed the great success of deep generative models in drug design, the existing methods are mostly string-based or graph-based. They are limited by the lack of spatial information and thus unable to be applied to structure-based design tasks. Particularly, such models have no or little knowledge of how molecules interact with their target proteins exactly in 3D space. In this paper, we propose a 3D generative model that generates molecules given a designated 3D protein binding site. Specifically, given a binding site as the 3D context, our model estimates the probability density of atom’s occurrences in 3D space — positions that are more likely to have atoms will be assigned higher probability. To generate 3D molecules, we propose an auto-regressive sampling scheme — atoms are sampled sequentially from the learned distribution until there is no room for new atoms. Combined with this sampling scheme, our model can generate valid and diverse molecules, which could be applicable to various structure-based molecular design tasks such as molecule sampling and linker design. Experimental results demonstrate that molecules sampled from our model exhibit high binding affinity to specific targets and good drug properties such as drug-likeness even if the model is not explicitly optimized for them.
How Well Does Self-Supervised Pre-Training Perform with Streaming Data?

Prior works on self-supervised pre-training focus on the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained. Unfortunately, such a problem setting is often impractical if not infeasible since many real-world tasks rely on sequential learning, e.g., data are decentralized or collected in a streaming fashion. In this paper, we conduct the first thorough and dedicated investigation on self-supervised pre-training with streaming data, aiming to shed light on the model behavior under this overlooked setup. Specifically, we pre-train over 500 models on four categories of pre-training streaming data from ImageNet and DomainNet and evaluate them on three types of downstream tasks and 12 different downstream datasets. Our studies show that, somehow beyond our expectation, with simple data replay or parameter regularization, sequential self-supervised pre-training turns out to be an efficient alternative for joint pre-training, as the performances of the former are mostly on par with those of the latter. Moreover, catastrophic forgetting, a common issue in sequential supervised learning, is much alleviated in sequential self-supervised learning (SSL), which is well justified through our comprehensive empirical analysis on representations and the sharpness of minima in the loss landscape. Our findings, therefore, suggest that, in practice, for SSL, the cumbersome joint training can be replaced mainly by sequential learning, which in turn enables a much broader spectrum of potential application scenarios.
Monotonic Differentiable Sorting Networks

Differentiable sorting algorithms allow training with sorting and ranking supervision, where only the ordering or ranking of samples is known. Various methods have been proposed to address this challenge, ranging from optimal transport-based differentiable Sinkhorn sorting algorithms to making classic sorting networks differentiable. One problem of current differentiable sorting methods is that they are non-monotonic. To address this issue, we propose a novel relaxation of conditional swap operations that guarantees monotonicity in differentiable sorting networks. We introduce a family of sigmoid functions and prove that they produce differentiable sorting networks that are monotonic. Monotonicity ensures that the gradients always have the correct sign, which is an advantage in gradient-based optimization. We demonstrate that monotonic differentiable sorting networks improve upon previous differentiable sorting methods.
Some general fixed-point theorems for nonlinear mappings connected with one Cauchy theorem

In this work, using a new geometrical approach we study to the existence of the fixed-point of mappings that independence of the smoothness, and also of their singlevalues or multi values. This work proved the theorems that generalize in some sense the Brouwer and Schauder fixed-point theorems, and also such type results in multi-valued cases. One can reckon this approach is based on the generalization of the one theorem Cauchy and on the convexity properties of sets. As the used approach is based on the geometry of the image of the examined mappings that are independent of the topological properties of the space we could to prove the general results for almost every vector space. The general results we applied to the study of the nonlinear equations and inclusions in VTS, and also by applying these results are investigated different concrete nonlinear problems. Here provided also sufficient conditions under which the conditions of the theorems will fulfill.
A quantitative Neumann lemma for finitely generated groups

We study the coset covering function $\mathfrak{C}(r)$ of a finitely generated group: the number of cosets of infinite index subgroups needed to cover the ball of radius $r$. We show that $\mathfrak{C}(r)$ is of order at least $\sqrt{r}$ for all groups. Moreover, we show that $\mathfrak{C}(r)$ is linear for a class of amenable groups including virtually nilpotent and polycyclic groups, and that it is exponential for property (T) groups.
DSRRTracker: Dynamic Search Region Refinement for Attention-based Siamese Multi-Object Tracking

Many multi-object tracking (MOT) methods follow the framework of"tracking by detection", which associates the target objects-of-interest based on the detection results. However, due to the separate models for detection and association, the tracking results are not optimal.Moreover, the speed is limited by some cumbersome association methods to achieve high tracking performance. In this work, we propose an end-to-end MOT method, with a Gaussian filter-inspired dynamic search region refinement module to dynamically filter and refine the search region by considering both the template information from the past frames and the detection results from the current frame with little computational burden, and a lightweight attention-based tracking head to achieve the effective fine-grained instance association. Extensive experiments and ablation study on MOT17 and MOT20 datasets demonstrate that our method can achieve the state-of-the-art performance with reasonable speed.
Hierarchies of resources for measurement-based quantum computation

For certain restricted computational tasks, quantum mechanics provides a provable advantage over any possible classical implementation. Several of these results have been proven using the framework of measurement-based quantum computation (MBQC), where nonlocality and more generally contextuality have been identified as necessary resources for certain quantum computations. Here, we consider the computational power of MBQC in more detail by refining its resource requirements, both on the allowed operations and the number of accessible qubits. More precisely, we identify which Boolean functions can be computed in non-adaptive MBQC, with local operations contained within a finite level in the Clifford hierarchy. Moreover, for non-adaptive MBQC restricted to certain subtheories such as stabiliser MBQC, we compute the minimal number of qubits required to compute a given Boolean function. Our results point towards hierarchies of resources that more sharply characterise the power of MBQC beyond the binary of contextuality vs non-contextuality.
Neural Predictor for Black-Box Adversarial Attacks on Speech Recognition

Recent works have revealed the vulnerability of automatic speech recognition (ASR) models to adversarial examples (AEs), i.e., small perturbations that cause an error in the transcription of the audio signal. Studying audio adversarial attacks is therefore the first step towards robust ASR. Despite the significant progress made in attacking audio examples, the black-box attack remains challenging because only the hard-label information of transcriptions is provided. Due to this limited information, existing black-box methods often require an excessive number of queries to attack a single audio example. In this paper, we introduce NP-Attack, a neural predictor-based method, which progressively evolves the search towards a small adversarial perturbation. Given a perturbation direction, our neural predictor directly estimates the smallest perturbation that causes a mistranscription. In particular, it enables NP-Attack to accurately learn promising perturbation directions via gradient-based optimization. Experimental results show that NP-Attack achieves competitive results with other state-of-the-art black-box adversarial attacks while requiring a significantly smaller number of queries. The code of NP-Attack is available online.
Testing black holes in non-linear electrodynamics from the observed quasi-periodic oscillations

Quasi-periodic oscillations (QPOs), in particular, the ones with high frequencies, often observed in the power spectrum of black holes, are useful in understanding the nature of strong gravity since they are associated with the motion of matter in the vicinity of the black hole horizon. Interestingly, these high frequency QPOs (HFQPOs) are observed in commensurable pairs, the most common ratio being 3:2. Several theoretical models are proposed in the literature which explain the HFQPOs in terms of the orbital and epicyclic frequencies of matter rotating around the central object. Since these frequencies are sensitive to the background spacetime, the observed HFQPOs can potentially extract useful information regarding the nature of the same. In this work, we investigate the role of regular black holes with a Minkowski core, which arise in gravity coupled to non-linear electrodynamics, in explaining the HFQPOs. Regular black holes are particularly interesting as they provide a possible resolution to the singularity problem in general relativity. We compare the model dependent QPO frequencies with the available observations of the quasi-periodic oscillations from black hole sources and perform a χ2 analysis. Our study reveals that most QPO models favor small but non-trivial values of the non-linear electrodynamics charge parameter. In particular, black holes with large values of non-linear electrodynamics charge parameter are generically disfavored by present observations related to QPOs.
Simulating Bandit Learning from User Feedback for Extractive Question Answering

We study learning from user feedback for extractive question answering by simulating feedback using supervised data. We cast the problem as contextual bandit learning, and analyze the characteristics of several learning scenarios with focus on reducing data annotation. We show that systems initially trained on few examples can dramatically improve given feedback from users on model-predicted answers, and that one can use existing datasets to deploy systems in new domains without any annotation effort, but instead improving the system on-the-fly via user feedback.
One After Another: Learning Incremental Skills for a Changing World

Reward-free, unsupervised discovery of skills is an attractive alternative to the bottleneck of hand-designing rewards in environments where task supervision is scarce or expensive. However, current skill pre-training methods, like many RL techniques, make a fundamental assumption - stationary environments during training. Traditional methods learn all their skills simultaneously, which makes it difficult for them to both quickly adapt to changes in the environment, and to not forget earlier skills after such adaptation. On the other hand, in an evolving or expanding environment, skill learning must be able to adapt fast to new environment situations while not forgetting previously learned skills. These two conditions make it difficult for classic skill discovery to do well in an evolving environment. In this work, we propose a new framework for skill discovery, where skills are learned one after another in an incremental fashion. This framework allows newly learned skills to adapt to new environment or agent dynamics, while the fixed old skills ensure the agent doesn't forget a learned skill. We demonstrate experimentally that in both evolving and static environments, incremental skills significantly outperform current state-of-the-art skill discovery methods on both skill quality and the ability to solve downstream tasks. Videos for learned skills and code are made public on https://notmahi.github. io/disk
Next-to-leading order scalar contributions to $\mu\rightarrow e$ conversion

Within a class of models in which lepton flavor violation is induced dominantly by scalar particle exchanges, we estimate the $\mu \to e$ conversion rate in several nuclei. We include next-to-leading order (NLO) terms in the one- and two-nucleon interactions in chiral effective theory, rectifying some incorrect results in the previous literature. We provide an uncertainty budget for the conversion rates and we find that NLO contributions affect the amplitudes at the level of $10\%$, which could be larger than the uncertainty on the leading order couplings, dominated by the strange and non-strange nucleon sigma terms. We study the implications of our results for testing Higgs-mediated CLFV in the future by combining results from various experimental searches, such as $\mu \to e$ conversion in multiple target nuclei and $\mu \to e \gamma$.
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration

Households across the world contain arbitrary objects: from mate gourds and coffee mugs to sitars and guitars. Considering this diversity, robot perception must handle a large variety of semantic objects without additional fine-tuning to be broadly applicable in homes. Recently, zero-shot models have demonstrated impressive performance in image classification of arbitrary objects (i.e., classifying images at inference with categories not explicitly seen during training). In this paper, we translate the success of zero-shot vision models (e.g., CLIP) to the popular embodied AI task of object navigation. In our setting, an agent must find an arbitrary goal object, specified via text, in unseen environments coming from different datasets. Our key insight is to modularize the task into zero-shot object localization and exploration. Employing this philosophy, we design CLIP on Wheels (CoW) baselines for the task and evaluate each zero-shot model in both Habitat and RoboTHOR simulators. We find that a straightforward CoW, with CLIP-based object localization plus classical exploration, and no additional training , often outperforms learnable approaches in terms of success, efficiency, and robustness to dataset distribution shift. This CoW achieves 6.3% SPL in Habitat and 10.0% SPL in RoboTHOR, when tested zero-shot on all categories. On a subset of four RoboTHOR categories considered in prior work, the same CoW shows a 16.1 percentage point improvement in Success over the learnable state-of-the-art baseline.
Mitigating Gender Bias in Machine Translation through Adversarial Learning

Machine translation and other NLP systems often contain significant biases regarding sensitive attributes, such as gender or race, that worsen system performance and perpetuate harmful stereotypes. Recent preliminary research suggests that adversarial learning can be used as part of a model-agnostic bias mitigation method that requires no data modifications. However, adapting this strategy for machine translation and other modern NLP domains requires (1) restructuring training objectives in the context of fine-tuning pretrained large language models and (2) developing measures for gender or other protected variables for tasks in which these attributes must be deduced from the data itself. We present an adversarial learning framework that addresses these challenges to mitigate gender bias in seq2seq machine translation. Our framework improves the disparity in translation quality for sentences with male vs. female entities by 86% for English-German translation and 91% for English-French translation, with minimal effect on translation quality. The results suggest that adversarial learning is a promising technique for mitigating gender bias in machine translation.