Miracle-dz/UniFAR-FTUs huggingface dataset:
Targeted deep learning: Framework, methods, and applications

Deep learning systems are typically designed to perform for a wide range of test inputs. For example, deep learning systems in autonomous cars are supposed to deal with traffic situations for which they were not specifically trained. In general, the ability to cope with a broad spectrum of unseen test inputs is called generalization. Generalization is definitely important in applications where the possible test inputs are known but plentiful or simply unknown, but there are also cases where the possible inputs are few and unlabeled but known beforehand. For example, medicine is currently interested in targeting treatments to individual patients; the number of patients at any given time is usually small (typically one), their diagnoses/responses are still unknown, but their general characteristics (such as genome information, protein levels in the blood, and so forth) are known before the treatment. We propose to call deep learning in such applications targeted deep learning. In this paper, we introduce a framework for targeted deep learning, and we devise and test an approach for adapting standard pipelines to the requirements of targeted deep learning. The approach is very general yet easy to use: It can be implemented as a simple data‐preprocessing step. We demonstrate on a variety of real‐world data that our approach can indeed render standard deep learning faster and more accurate when the test inputs are known beforehand.
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model

Inspired by biological evolution, we explain the rationality of Vision Transformer by analogy with the proven practical Evolutionary Algorithm (EA) and derive that both of them have consistent mathematical representation. Analogous to the dynamic local population in EA, we improve the existing transformer structure and propose a more efficient EAT model, and design task-related heads to deal with different tasks more flexibly. Moreover, we introduce the spatial-filling curve into the current vision transformer to sequence image data into a uniform sequential format. Thus we can design a unified EAT framework to address multi-modal tasks, separating the network architecture from the data format adaptation. Our approach achieves state-of-the-art results on the ImageNet classification task compared with recent vision transformer works while having smaller parameters and greater throughput. We further conduct multi-model tasks to demonstrate the superiority of the unified EAT, e.g., Text-Based Image Retrieval, and our approach improves the rank-1 by +3.7 points over the baseline on the CSS dataset.
The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications. Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work. However, despite humans being the target end-users, most attribution methods were only evaluated on proxy automatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et al. 2018). In this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine-grained classification, and when an image is natural or adversarial (i.e., contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-AI teams compared to AI alone. Importantly, we found automatic attribution-map evaluation measures to correlate poorly with the actual human-AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-loop applications and to rethink the existing evaluation metrics.
Weighted Training for Cross-Task Learning

In this paper, we introduce Target-Aware Weighted Training (TAWT), a weighted training algorithm for cross-task learning based on minimizing a representation-based task distance between the source and target tasks. We show that TAWT is easy to implement, is computationally efficient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees. The effectiveness of TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER). As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of fine-tuning.
Towards mental time travel: a hierarchical memory for reinforcement learning agents

Reinforcement learning agents often forget details of the past, especially after delays or distractor tasks. Agents with common memory architectures struggle to recall and integrate across multiple timesteps of a past event, or even to recall the details of a single timestep that is followed by distractor tasks. To address these limitations, we propose a Hierarchical Chunk Attention Memory (HCAM), which helps agents to remember the past in detail. HCAM stores memories by dividing the past into chunks, and recalls by first performing high-level attention over coarse summaries of the chunks, and then performing detailed attention within only the most relevant chunks. An agent with HCAM can therefore"mentally time-travel"-- remember past events in detail without attending to all intervening events. We show that agents with HCAM substantially outperform agents with other memory architectures at tasks requiring long-term recall, retention, or reasoning over memory. These include recalling where an object is hidden in a 3D environment, rapidly learning to navigate efficiently in a new neighborhood, and rapidly learning and retaining new object names. Agents with HCAM can extrapolate to task sequences much longer than they were trained on, and can even generalize zero-shot from a meta-learning setting to maintaining knowledge across episodes. HCAM improves agent sample efficiency, generalization, and generality (by solving tasks that previously required specialized architectures). Our work is a step towards agents that can learn, interact, and adapt in complex and temporally-extended environments.
Efficient and Modular Implicit Differentiation

Automatic differentiation (autodiff) has revolutionized machine learning. It allows to express complex computations by composing elementary ones in creative ways and removes the burden of computing their derivatives by hand. More recently, differentiation of optimization problem solutions has attracted widespread attention with applications such as optimization layers, and in bi-level problems such as hyper-parameter optimization and meta-learning. However, so far, implicit differentiation remained difficult to use for practitioners, as it often required case-by-case tedious mathematical derivations and implementations. In this paper, we propose automatic implicit differentiation, an efficient and modular approach for implicit differentiation of optimization problems. In our approach, the user defines directly in Python a function $F$ capturing the optimality conditions of the problem to be differentiated. Once this is done, we leverage autodiff of $F$ and the implicit function theorem to automatically differentiate the optimization problem. Our approach thus combines the benefits of implicit differentiation and autodiff. It is efficient as it can be added on top of any state-of-the-art solver and modular as the optimality condition specification is decoupled from the implicit differentiation mechanism. We show that seemingly simple principles allow to recover many existing implicit differentiation methods and create new ones easily. We demonstrate the ease of formulating and solving bi-level optimization problems using our framework. We also showcase an application to the sensitivity analysis of molecular dynamics.
Document-level Event Extraction via Heterogeneous Graph-based Interaction Model with a Tracker

Document-level event extraction aims to recognize event information from a whole piece of article. Existing methods are not effective due to two challenges of this task: a) the target event arguments are scattered across sentences; b) the correlation among events in a document is non-trivial to model. In this paper, we propose Heterogeneous Graph-based Interaction Model with a Tracker (GIT) to solve the aforementioned two challenges. For the first challenge, GIT constructs a heterogeneous graph interaction network to capture global interactions among different sentences and entity mentions. For the second, GIT introduces a Tracker module to track the extracted events and hence capture the interdependency among the events. Experiments on a large-scale dataset (Zheng et al, 2019) show GIT outperforms the previous methods by 2.8 F1. Further analysis reveals is effective in extracting multiple correlated events and event arguments that scatter across the document.
A splitting Hamiltonian Monte Carlo method for efficient sampling

We propose a splitting Hamiltonian Monte Carlo (SHMC) algorithm, which can be computationally eﬃcient when combined with the random mini-batch strategy. By splitting the potential energy into numerically nonstiﬀ and stiﬀ parts, one makes a proposal using the nonstiﬀ part of U , followed by a Metropolis rejection step using the stiﬀ part that is often easy to compute. The splitting allows eﬃcient sampling from systems with singular potentials (or distributions with degenerate points) and/or with multiple potential barriers. In our SHMC algorithm, the proposal only based on the nonstiﬀ part in the splitting is generated by the Hamiltonian dynamics, which can be potentially more eﬃcient than the overdamped Langevin dynamics. We also use random batch strategies to reduce the computational cost to O (1) per time step in generating the proposals for problems arising from many-body systems and Bayesian inference, and prove that the errors of the Hamiltonian induced by the random batch approximation is O ( √ ∆ t ) in the strong and O (∆ t ) in the weak sense, where ∆ t is the time step. Numerical experiments are conducted to verify the theoretical results and the computational eﬃciency of the proposed algorithms in practice.
Macroscopic turbulent flow via hard sphere potential

In recent works, we proposed a hypothesis that the turbulence in gases could be produced by particles interacting via a potential, and examined the proposed mechanics of turbulence formation in a simple model of two particles for a variety of different potentials. In this work, we use the same hypothesis to develop new fluid mechanics equations which model turbulent gas flow on a macroscopic scale. The main difference between our approach and the conventional formalism is that we avoid replacing the potential interaction between particles with the Boltzmann collision integral. Due to this difference, the velocity moment closure, which we implement for the shear stress and heat flux, relies upon the high Reynolds number condition, rather than the Newton law of viscosity and the Fourier law of heat conduction. The resulting system of equations of fluid mechanics differs considerably from the standard Euler and Navier–Stokes equations. A numerical simulation of our system shows that a laminar Bernoulli jet of an argon-like hard sphere gas in a straight pipe rapidly becomes a turbulent flow. The time-averaged Fourier spectra of the kinetic energy of this flow exhibit Kolmogorov’s negative five-thirds power decay rate.
Component Behaviour and Excess of Random Bipartite Graphs Near the Critical Point

The binomial random bipartite graph $G(n,n,p)$ is the random graph formed by taking two partition classes of size $n$ and including each edge between them independently with probability $p$. It is known that this model exhibits a similar phase transition as that of the binomial random graph $G(n,p)$ as $p$ passes the critical point of $\frac{1}{n}$. We study the component structure of this model near to the critical point. We show that, as with $G(n,p)$, for an appropriate range of $p$ there is a unique `giant' component and we determine asymptotically its order and excess. We also give more precise results for the distribution of the number of components of a fixed order in this range of $p$. These results rely on new bounds for the number of bipartite graphs with a fixed number of vertices and edges, which we also derive.
Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this article, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content.
Stochastic Flows on Non-compact Manifolds

Here we look at the existence of solution flows of stochastic differential equations on noncompact manifolds and the properties of the solutions in terms of the geometry and topology of the underlying manifold itself. We obtain some results on “strong p-completeness” given conditions on the derivative flow, and thus given suitable conditions on the coefficients of the stochastic differential equations. In particular a smooth flow of Brownian motion exists on submanifolds of R whose second fundamental forms are bounded. Another class of results we obtain is on homotopy vanishing given strong moment stability. We also have results on obstructions to moment stability by cohomology. Also we obtain formulae for d(e 1 2 t4hφ) for differential form φ in terms of a martingale and the form itself, not just its derivative, extending Bismut’s formula.
Data Acquisition for Improving Machine Learning Models

The vast advances in Machine Learning (ML) over the last ten years have been powered by the availability of suitably prepared data for training purposes. The future of ML-enabled enterprise hinges on data. As such, there is already a vibrant market offering data annotation services to tailor sophisticated ML models. In this paper, inspired by the recent vision of online data markets and associated market designs, we present research on the practical problem of obtaining data in order to improve the accuracy of ML models. We consider an environment in which consumers query for data to enhance the accuracy of their models and data providers who possess data make them available for training purposes. We first formalize this interaction process laying out the suitable framework and associated parameters for data exchange. We then propose two data acquisition strategies that consider a trade-off between exploration during which we obtain data to learn about the distribution of a provider's data and exploitation during which we optimize our data inquiries utilizing the gained knowledge. In the first strategy, Estimation and Allocation (EA), we utilize queries to estimate the utilities of various predicates while learning about the distribution of the provider's data; then we proceed to the allocation stage in which we utilize those learned utility estimates to inform our data acquisition decisions. The second algorithmic proposal, named Sequential Predicate Selection (SPS), utilizes a sampling strategy to explore the distribution of the provider's data, adaptively investing more resources to parts of the data space that are statistically more promising to improve overall model accuracy. We present a detailed experimental evaluation of our proposals utilizing a variety of ML models and associated real data sets exploring all applicable parameters of interest. Our results demonstrate the relative benefits of the proposed algorithms. Depending on the models trained and the associated learning tasks we identify trade-offs and highlight the relative benefits of each algorithm to further optimize model accuracy.
STEP: Spatial-Temporal Network Security Event Prediction

Network security events prediction helps network operators to take response strategies from a proactive perspective, and reduce the cost caused by network attacks, which is of great significance for maintaining the security of the entire network. Most of the existing event prediction methods rely on temporal characteristics and are dedicated to exploring time series predictions, but ignoring the spatial relationship between hosts. This paper combines the temporal and spatial characteristics of security events and proposes a spatial-temporal event prediction model, named STEP. In particular, STEP formulates the security events prediction into a spatial-temporal sequence prediction. STEP utilizes graph convolution operation to capture the spatial characteristics of hosts in the network, and adopts the long short term memory (LSTM) to capture the dynamic temporal dependency of events. This paper verifies the proposed STEP scheme on two public data sets. The experimental results show that the prediction accuracy of security events under STEP is higher than that of benchmark models such as LSTM, ConvLSTM. Besides, STEP achieves high prediction accuracy when we predict events from different lengths of sequence.
Kurtosis-Limited Sphere Shaping for Nonlinear Interference Noise Reduction in Optical Channels

Nonlinear interference (NLI) generated during the propagation of an optical waveform through the fiber depends on the fourth order standardized moment of the channel input distribution, also known as kurtosis. Probabilistically-shaped inputs optimized for the linear Gaussian channel have a Gaussian-like distribution with high kurtosis. For optical channels, this leads to an increase in NLI power and consequently, a decrease in effective signal-to-noise ratio (SNR). In this work, we propose kurtosis-limited enumerative sphere shaping (K-ESS) as an algorithm to generate low-kurtosis shaped inputs. Numerical simulations at a shaping blocklength of 108 amplitudes demonstrate that with K-ESS, it is possible to increase the effective SNRs by 0.4 dB in a single-span single-channel scenario at 400 Gbit/s. K-ESS offers also a twofold decrease in frame error rate with respect to Gaussian-channel-optimal sphere shaping.
Fitting Structural Equation Models via Variational Approximations

Abstract Structural equation models are commonly used to capture the relationship between sets of observed and unobservable variables. Traditionally these models are fitted using frequentist approaches, but recently researchers and practitioners have developed increasing interest in Bayesian inference. In Bayesian settings, inference for these models is typically performed via Markov chain Monte Carlo methods, which may be computationally intensive for models with a large number of manifest variables or complex structures. Variational approximations can be a fast alternative; however, they have not been adequately explored for this class of models. We develop a mean field variational Bayes approach for fitting elemental structural equation models and demonstrate how bootstrap can considerably improve the variational approximation quality. We show that this variational approximation method can provide reliable inference while being significantly faster than Markov chain Monte Carlo methods.
Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithm

Many engineering problems have multiple objectives, and the overall aim is to optimize a non-linear function of these objectives. In this paper, we formulate the problem of maximizing a non-linear concave function of multiple long-term objectives. A policy-gradient based model-free algorithm is proposed for the problem. To compute an estimate of the gradient, a biased estimator is proposed. The proposed algorithm is shown to achieve convergence to within an $\epsilon$ of the global optima after sampling $\mathcal{O}(\frac{M^4\sigma^2}{(1-\gamma)^8\epsilon^4})$ trajectories where $\gamma$ is the discount factor and $M$ is the number of the agents, thus achieving the same dependence on $\epsilon$ as the policy gradient algorithm for the standard reinforcement learning.
A randomized algorithm for the stabilization of switched nonlinear systems under restricted switching

This paper deals with input/output-to-state stability (IOSS) of switched nonlinear systems in the discrete-time setting. We present an algorithm to construct periodic switching signals that obey pre-specified restrictions on admissible switches between the subsystems and admissible dwell times on the subsystems, and identify sufficient conditions on the individual subsystems, the admissible switches and admissible dwell times under which a switching signal obtained from our algorithm preserves stability of a switched system with overwhelming probability. We recover our earlier result on probabilistic techniques for the design of switching signals that preserve global asymptotic stability of switched linear systems under sufficient conditions on the properties of the individual subsystems and the admissible dwell times on the subsystems.