Miracle-dz/UniFAR-FTUs huggingface dataset:
Adversarial Imitation Learning with Preferences

Designing an accurate and explainable reward function for many Reinforcement Learning tasks is a cumbersome and tedious process. Instead, learning policies directly from the feedback of human teachers naturally integrates human domain knowledge into the policy optimization process. Different feedback modalities, such as demonstrations and preferences, provide distinct benefits and disadvan-tages. For example, demonstrations convey a lot of information about the task but are often hard or costly to obtain from real experts while preferences typically contain less information but are in most cases cheap to generate. However, existing methods centered around human feedback mostly focus on a single teaching modality, causing them to miss out on important training data while making them less intuitive to use. In this paper we propose a novel method for policy learning that incorporates two different feedback types, namely demonstrations and preferences . To this end, we make use of the connection between discriminator training and density ratio estimation to incorporate preferences into the popular Adversarial Imitation Learning paradigm. This insight allows us to express loss functions over both demonstrations and preferences in a unified framework. Besides expert demonstrations, we are also able to learn from imperfect ones and combine them with preferences to achieve improved task performance. We experimentally validate the effectiveness of combining both preferences and demonstrations on common benchmarks and also show that our method can efficiently learn challenging robot manipulation tasks.
Geometrically regularized autoencoders for non-Euclidean data

Regularization is almost de rigueur when designing autoencoders that are sparse and robust to noise. Given the recent surge of interest in machine learning problems involving non-Euclidean data, in this paper we address the regularization of autoencoders on curved spaces. We show that by ignoring the underlying geometry of the data and applying standard vector space regularization techniques, autoencoder performance can be severely degraded, or worse, training can fail to converge. Assuming that both the data space and latent space can be modeled as Riemannian manifolds, we show how to construct regularization terms in a coordinate-invariant way, and develop geometric generalizations of the denoising autoencoder and reconstruction contractive autoencoder such that the essential properties that enable the estimation of the derivative of the log-probability density are preserved. Drawing upon various non-Euclidean data sets, we show that our geometric autoencoder regularization techniques can have important performance advantages over vector-spaced methods while avoiding other breakdowns that can result from failing to account for the underlying geometry.
DFlow: Learning to Synthesize Better Optical Flow Datasets via a Differentiable Pipeline

Comprehensive studies of synthetic optical flow datasets have attempted to reveal what properties lead to accuracy improvement in learning-based optical flow estimation. However, manually identifying and verifying the properties that contribute to accurate optical flow estimation require large-scale trial-and-error experiments with iteratively generating whole synthetic datasets and training on them, i.e . , impractical. To address this challenge, we propose a differentiable optical flow data generation pipeline and a loss function to drive the pipeline, called DFlow. DFlow efficiently synthesizes a dataset effective for a target domain without the need for cumbersome try-and-errors. This favorable property is achieved by proposing an efficient dataset comparison method that uses neural networks to approximately encode each dataset and compares the proxy networks instead of explicitly comparing datasets in a pairwise way. Our experiments show the competitive performance of our DFlow against the prior arts in pre-training. Furthermore, compared to competing datasets, DFlow achieves the best fine-tuning performance on the Sintel public benchmark with RAFT.
Average Sensitivity of Decision Tree Learning

A decision tree is a fundamental model used in data mining and machine learning. In practice, the training data used to construct a decision tree may change over time or contain noise, and a drastic change in the learned tree structure owing to such data perturbation is unfavorable. For example, in data mining, a change in the tree implies a change in the extracted knowledge, which raises the question of whether the extracted knowledge is truly reliable or is only a noisy artifact. To alleviate this issue, we design decision tree learning algorithms that are stable against insignificant perturbations in the training data. Specifically, we adopt the notion of average sensitivity as a stability measure, and design an algorithm with low average sensitivity that outputs a decision tree whose accuracy is close to the optimal decision tree. The experimental results on real-world datasets demonstrate that the proposed algorithm enables users to select suitable decision trees considering the trade-off between average sensitivity and accuracy.
Learning About Progress From Experts

Many important tasks involve some notion of long-term progress in multiple phases: e.g. to clean a shelf it must be cleared of items, cleaning products applied, and then the items placed back on the shelf. In this work, we explore the use of expert demonstrations in long-horizon tasks to learn a monotonically increasing function that summarizes progress. This function can then be used to aid agent exploration in environments with sparse rewards. As a case study we consider the NetHack environment, which requires long-term progress at a variety of scales and is far from being solved by existing approaches. In this environment, we demonstrate that by learning a model of long-term progress from expert data containing only observations, we can achieve efficient exploration in challenging sparse tasks, well beyond what is possible with current state-of-the-art approaches. We have made the curated gameplay dataset used in this work available at https://github.com/deepmind/nao_top10.
Mycotoxin Testing Paradigm: Challenges and Opportunities for the Future.

Mycotoxins are one of the great global challenges to agri-food and feed safety. Industry requires fast, reliable, and economical testing methods for the most important regulated mycotoxins to manage this problem. Climate change and changes in agricultural practice are complicating this situation, triggering the movement of some mycotoxins into new regions, which are unprepared for their management. Modern LC-tandem MS (LC-MS/MS) instruments have addressed this analytical challenge, but such instruments are expensive and require highly qualified personnel and dedicated facilities. As a result of these limitations, traditional LC-MS/MS is not amenable for use on farms or at small to midsized processing facilities, such as a grain elevator. To address the need for on-site rapid testing, the mycotoxin community has focused on antibody-based and spectrophotometric approaches. The development of innovative technologies such as miniaturized MS would allow for the acquisition of more information on mixtures of toxins present in a sample at costs comparable to those of the existing rapid methods such as ELISA. The capital costs are higher, but it would reduce per-sample testing costs and time requirements and provide better value for money while maintaining the accuracy and selectivity achieved in a laboratory setting. In this article, we review the available techniques and contrast them in the context of three main criteria: method performance, speed of analysis, and cost. We define the integration of these three parameters as the "mycotoxin testing paradigm. "
A CMDP-within-online framework for Meta-Safe Reinforcement Learning

Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience. However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings. In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting. We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the taskaveraged optimality gap and constraint satisfaction improve with task-similarity in a static environment, or task-relatedness in a dynamic environment. Several technical challenges arise when making this framework practical. To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by offpolicy stationary distribution corrections. Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle. Finally, experiments are conducted to demonstrate the effectiveness of our approach.
Learning in temporally structured environments

Natural environments have temporal structure at multiple timescales. This property is reﬂected in biological learning and memory but typically not in machine learning systems. We advance a multiscale learning method in which each weight in a neural network is decomposed as a sum of subweights with different learning and decay rates. Thus knowledge becomes distributed across different timescales, enabling rapid adaptation to task changes while avoiding catastrophic interference. First, we prove previous models that learn at multiple timescales, but with complex coupling between timescales, are equivalent to multiscale learning via a reparameterization that eliminates this coupling. The same analysis yields a new characterization of momentum learning, as a fast weight with a negative learning rate. Second, we derive a model of Bayesian inference over 1 /f noise, a common temporal pattern in many online learning domains that involves long-range (power law) autocorrelations. The generative side of the model expresses 1 /f noise as a sum of diffusion processes at different timescales, and the inferential side tracks these latent processes using a Kalman ﬁlter. We then derive a variational approximation to the Bayesian model and show how it is an extension of the multiscale learner. The result is an optimizer that can be used as a drop-in replacement in an arbitrary neural network architecture. Third, we evaluate the ability of these methods to handle nonstationarity by testing them in online prediction tasks characterized by 1 /f noise in the latent parameters. We ﬁnd that the Bayesian model signiﬁcantly outperforms online stochastic gradient descent and two batch
Zeroth-Order Optimization with Trajectory-Informed Derivative Estimation

Zeroth-order (ZO) optimization, in which the derivative is unavailable, has recently succeeded in many important machine learning applications. Existing algorithms rely on finite difference (FD) methods for derivative estimation and gradient descent (GD)-based approaches for optimization. However, these algorithms suffer from query inefficiency because many additional function queries are required for derivative estimation in their every GD update, which typically hinders their deployment in real-world applications where every function query is expensive. To this end, we propose a trajectory-informed derivative estimation method which only employs the optimization trajectory (i.e., the history of function queries during optimization) and hence can eliminate the need for additional function queries to estimate a derivative. Moreover, based on our derivative estimation, we propose the technique of dynamic virtual updates, which allows us to reliably perform multiple steps of GD updates without reapplying derivative estimation. Based on these two contributions, we introduce the zeroth-order optimization with trajectory-informed derivative estimation (ZORD) algorithm for query-efficient ZO optimization. We theoretically demonstrate that our trajectory-informed derivative estimation and our ZORD algorithm improve over existing approaches, which is then supported by our real-world experiments such as black-box adversarial attack, non-differentiable metric optimization, and derivative-free reinforcement learning.
Revisiting Populations in multi-agent Communication

Despite evidence from sociolinguistics that larger groups of speakers tend to 1 develop more structured languages, the use of populations has failed to yield 2 significant benefits in emergent multi-agent communication. In this paper we 3 reassess the validity of the standard training protocol and illustrate its limitations. 4 Specifically, we analyze population-level communication at the equilibrium in 5 sender-receiver Lewis games. We find that receivers co-adapt to senders they 6 are interacting with, which limits the effect of the population. Informed by this 7 analysis, we propose an alternative training protocol based on “partitioning” agents. 8 Partitioning isolates sender-receiver pairs, limits co-adaptation, and results in a new 9 global optimization objective where agents maximize (1) their respective "internal" 10 communication accuracy and (2) their alignment with other agents. In experiments, 11 we find that agents trained in partitioned populations are able to communicate 12 successfully with new agents which they have never interacted with and tend to 13 develop a shared language. Moreover, we observe that larger populations develop 14 languages that are more compositional. Our findings suggest that scaling up to 15 populations in multi-agent communication can be beneficial, but that it matters 16 how we scale up. 17
Making Better Decision by Directly Planning in Continuous Control

By properly utilizing the learned environment model, model-based reinforcement learning methods can improve the sample efficiency for decision-making problems. Beyond using the learned environment model to train a policy, the success of MCTS-based methods shows that directly incorporating the learned environment model as a planner to make decisions might be more effective. However, when action space is of high dimension and continuous, directly planning according to the learned model is costly and non-trivial. Because of two challenges: (1) the infinite number of candidate actions and (2) the temporal dependency between actions in different timesteps. To address these challenges, inspired by Differential Dynamic Programming (DDP) in optimal control theory, we design a novel Policy Optimization with Model Planning (POMP) algorithm, which incorporates a carefully designed Deep Differential Dynamic Programming (D3P) planner into the model-based RL framework. In D3P planner, (1) to effectively plan in the continuous action space, we construct a locally quadratic programming problem that uses a gradient-based optimization process to replace search. (2) To take the temporal dependency of actions at different timesteps into account, we leverage the updated and latest actions of previous timesteps (i.e., step 1 , · · · , h − 1 ) to update the action of the current step (i.e., step h ), instead of updating all actions simultaneously. We theoretically prove the convergence rate for our D3P planner and analyze the effect of the feedback term. In practice, to effectively apply the neural network based
Towards One-shot Neural Combinatorial Solvers: Theoretical and Empirical Notes on the Cardinality-Constrained Case

One-shot non-autoregressive neural networks, different from RL-based ones, have been actively adopted for solving combinatorial optimization (CO) problems, which can be trained by the objective score in a self-supervised manner. Such methods have shown their superiority in efficiency (e.g. by parallelization) and potential for tackling predictive CO problems for decision-making under uncertainty. While the discrete constraints often become a bottleneck for gradient-based neural solvers, as currently handled in three typical ways: 1) adding a soft penalty in the objective, where a bounded violation of the constraints cannot be guaranteed, being critical to many constraint-sensitive scenarios; 2) perturbing the input to generate an approximate gradient in a black-box manner, though the constraints are exactly obeyed while the approximate gradients can hurt the performance on the objective score; 3) a compromise by developing soft algorithms whereby the output of neural networks obeys a relaxed constraint, and there can still occur an arbitrary degree of constraint-violation. Towards the ultimate goal of establishing a general framework for neural CO solver with the ability to control an arbitrarysmall degree of constraint violation, in this paper, we focus on a more achievable and common setting: the cardinality constraints, which in fact can be readily encoded by a differentiable optimal transport (OT) layer. Based on this observation, we propose OT-based cardinality constraint encoding for end-to-end CO problem learning with two variants: Sinkhorn and Gumbel-Sinkhorn, whereby their violation of the constraints can be exactly characterized and bounded by our theoretical results. On synthetic and real-world CO problem instances, our methods surpass the state-of-the-art CO network and are comparable to (if not superior to) the commercial solver Gurobi. In particular, we further showcase a case study of applying our approach to the predictive portfolio optimization task on real-world asset price data, improving the Sharpe ratio from 1.1 to 2.0 of a strong LSTM+Gurobi baseline under the classic predict-then-optimize paradigm.
Any-scale Balanced Samplers for Discrete Space

The locally balanced informed proposal has proved to be highly effective for sampling from discrete spaces. However, its success relies on the “local” factor, which ensures that whenever the proposal distribution is restricted to be near the current state, the locally balanced weight functions are asymptotically optimal and the gradient approximations are accurate. In seeking a more efficient sampling algorithm, many recent works have considered increasing the scale of the proposal distributions, but this causes the “local” factor to no longer hold. Instead, we propose any-scale balanced samplers to repair the gap in non-local proposals. In particular, we substitute the locally balanced function with an any-scale balanced function that can self-adjust to achieve better efficiency for proposal distributions at any scale. We also use quadratic approximations to capture curvature of the target distribution and reduce the error in the gradient approximation, while employing a Gaussian integral trick with a special estimated diagonal to efficiently sample from the quadratic proposal distribution. On various synthetic and real distributions, the proposed sampler substantially outperforms existing approaches.
Evaluation of the foal survival score in a Danish‐Swedish population of neonatal foals upon hospital admission

Abstract Background It is highly desirable to assess the probability of survival in sick neonatal foals upon admission. The foal survival score (FSS) is a published scoring system used to estimate the probability of survival in hospitalized neonatal foals <4 days old. Hypothesis/Objectives To evaluate the ability of the FSS to predict survival in older foals from a geographically different area compared to the original study. Animals Five‐hundred ninety hospitalized neonatal foals ≤14 days of age. Methods Retrospective Danish‐Swedish multicenter study that included details of signalment, history, clinical examination, laboratory results, necropsy findings, and outcome. Scores and score variables were compared between survivors and nonsurvivors using logistic regression. The optimal cutoff and its test parameters were calculated using a receiver operator characteristic curve. Results Prematurity, cold extremities, ≥2 infectious or inflammatory sites, blood glucose concentration, and total white blood cell counts were significantly associated with nonsurvival (P ≤ .02). The optimal cutoff to predict survival was ≥6, resulting in sensitivity 78%, specificity 58%, 92% positive predictive value, and 31% negative predictive value. The test performed equally well in foals <4 days old compared to those 4‐14 days old. Conclusions and Clinical Importance Using the suggested optimal cutoff of ≥6, the FSS performed moderately well and may aid in early determination of prognosis for survival. However, the FSS did perform differently in another population and therefore should be assessed under local conditions so that its diagnostic potential is not overestimated.
ON THE ALGEBRAIC PROPERTIES OF THE HUMAN ABO-BLOOD GROUP INHERITANCE PATTERN

We generate an algebra on blood phenotypes with multiplication based on the human ABO-blood group inheritance pattern. We assume that gametes are not chosen randomly during meiosis. We investigate some of the properties of this algebra, namely, the set of idempotents, lattice of ideals and the associative enveloping algebra.
Predicting Risk Areas of Classical Scrapie in China Based on Environmental Suitability

Classical scrapie is a transmissible spongiform encephalopathy that attacks the central nervous system of sheep and goats. Since its discovery in the 18th century, the disease has caused enormous economic losses and public health impacts in continental Europe. In the late 20th century, classical scrapie began to spread to places, such as Asia and the Americas, becoming a disease of global concern. In this study, based on prion occurrence records and high-resolution environmental layers, a risk assessment of classical scrapie in China was performed using a maximum entropy model. The model achieved an area under the curve value of 0.906 (95% confidence interval, 0.0883–0.0929). Sheep distribution density, road density, goat distribution density, minimum temperature of the coldest month, port density, and precipitation of the driest quarter were identified as important factors affecting the occurrence of classical scrapie. The risk map showed that potential high-risk areas in China were mainly located in Northeast China, North China, and Northwest China. This study can provide a valuable reference for the prevention of classical scrapie in China. According to the environmental variables and risk areas of classical scrapie, implementing monitoring and early warning measures in these areas is recommended to reduce the possibility of classical scrapie occurrence and transmission.
Is It Overkill? Analyzing Feature-Space Concept Drift in Malware Detectors

Concept drift is a major challenge faced by machine learning-based malware detectors when deployed in practice. While existing works have investigated methods to detect concept drift, it is not yet well understood regarding the main causes behind the drift. In this paper, we design experiments to empirically analyze the impact of feature-space drift (new features introduced by new samples) and compare it with data-space drift (data distribution shift over existing features). Surprisingly, we find that data-space drift is the dominating contributor to the model degradation over time while feature-space drift has little to no impact. This is consistently observed over both Android and PE malware detectors, with different feature types and feature engineering methods, across different settings. We further validate this observation with recent online learning based malware detectors that incrementally update the feature space. Our result indicates the possibility of handling concept drift without frequent feature updating, and we further discuss the open questions for future research.