Miracle-dz/UniFAR-FTUs huggingface dataset:
Benign Underfitting of Stochastic Gradient Descent

We study to what extent may stochastic gradient descent (SGD) be understood as a"conventional"learning rule that achieves generalization performance by obtaining a good fit to training data. We consider the fundamental stochastic convex optimization framework, where (one pass, without-replacement) SGD is classically known to minimize the population risk at rate $O(1/\sqrt n)$, and prove that, surprisingly, there exist problem instances where the SGD solution exhibits both empirical risk and generalization gap of $\Omega(1)$. Consequently, it turns out that SGD is not algorithmically stable in any sense, and its generalization ability cannot be explained by uniform convergence or any other currently known generalization bound technique for that matter (other than that of its classical analysis). We then continue to analyze the closely related with-replacement SGD, for which we show that an analogous phenomenon does not occur and prove that its population risk does in fact converge at the optimal rate. Finally, we interpret our main results in the context of without-replacement SGD for finite-sum convex optimization problems, and derive upper and lower bounds for the multi-epoch regime that significantly improve upon previously known results.
Ergodic theorems for algorithmically random points

This paper is a survey of applications of the theory of algorithmic randomness to ergodic theory. We establish various degrees of constructivity for asymptotic laws of probability theory. In the framework of the Kolmogorov approach to the substantiation of the probability theory and information theory on the base of the theory of algorithms, we formulate probabilistic laws, i.e. statements which hold almost surely, in a pointwise form, i.e., for Martin-Lof random points. It is shown in this paper that the main statement of ergodic theory - Birkhoff ergodic theorem, is non-constructive in the strong (classical) sense, but it is constructive in some weaker sense - in terms of Martin-Lof randomness.
WSLRec: Weakly Supervised Learning for Neural Sequential Recommendation Models

Learning the user-item relevance hidden in implicit feedback data plays an important role in modern recommender systems. Neural sequential recommendation models, which formulates learning the user-item relevance as a sequential classification problem to distinguish items in future behaviors from others based on the user's historical behaviors, have attracted a lot of interest in both industry and academic due to their substantial practical value. Though achieving many practical successes, we argue that the intrinsic {\bf incompleteness} and {\bf inaccuracy} of user behaviors in implicit feedback data is ignored and conduct preliminary experiments for supporting our claims. Motivated by the observation that model-free methods like behavioral retargeting (BR) and item-based collaborative filtering (ItemCF) hit different parts of the user-item relevance compared to neural sequential recommendation models, we propose a novel model-agnostic training approach called WSLRec, which adopts a three-stage framework: pre-training, top-$k$ mining, and fine-tuning. WSLRec resolves the incompleteness problem by pre-training models on extra weak supervisions from model-free methods like BR and ItemCF, while resolves the inaccuracy problem by leveraging the top-$k$ mining to screen out reliable user-item relevance from weak supervisions for fine-tuning. Experiments on two benchmark datasets and online A/B tests verify the rationality of our claims and demonstrate the effectiveness of WSLRec.
Neural Adaptive SCEne Tracing

Neural rendering with implicit neural networks has recently emerged as an attractive proposition for scene reconstruction, achieving excellent quality albeit at high computational cost. While the most recent generation of such methods has made progress on the rendering (inference) times, very little progress has been made on improving the reconstruction (training) times. In this work, we present Neural Adaptive Scene Tracing (NAScenT), the first neural rendering method based on directly training a hybrid explicit-implicit neural representation. NAScenT uses a hierarchical octree representation with one neural network per leaf node and combines this representation with a two-stage sampling process that concentrates ray samples where they matter most near object surfaces. As a result, NAScenT is capable of reconstructing challenging scenes including both large, sparsely populated volumes like UAV captured outdoor environments, as well as small scenes with high geometric complexity. NAScenT outperforms existing neural rendering approaches in terms of both quality and training time.
Forecasting US Inflation Using Bayesian Nonparametric Models

The relationship between inflation and predictors such as unemployment is potentially nonlinear with a strength that varies over time, and prediction errors error may be subject to large, asymmetric shocks. Inspired by these concerns, we develop a model for inflation forecasting that is nonparametric both in the conditional mean and in the error using Gaussian and Dirichlet processes, respectively. We discuss how both these features may be important in producing accurate forecasts of inflation. In a forecasting exercise involving CPI inflation, we find that our approach has substantial benefits, both overall and in the left tail, with nonparametric modeling of the conditional mean being of particular importance.
Path-Aware Graph Attention for HD Maps in Motion Prediction

The success of motion prediction for autonomous driving relies on integration of information from the HD maps. As maps are naturally graph-structured, investigation on graph neural networks (GNNs) for encoding HD maps is burgeoning in recent years. However, unlike many other applications where GNNs have been straightforwardly deployed, HD maps are heterogeneous graphs where vertices (lanes) are connected by edges (lane-lane interaction relationships) of various nature, and most graph-based models are not designed to understand the variety of edge types which provide crucial cues for predicting how the agents would travel the lanes. To overcome this challenge, we propose Path-Aware Graph Attention, a novel attention architecture that infers the attention between two vertices by parsing the sequence of edges forming the paths that connect them. Our analysis illustrates how the proposed attention mechanism can facilitate learning in a didactic problem where existing graph networks like GCN struggle. By improving map encoding, the proposed model surpasses previous state of the art on the Argoverse Motion Forecasting dataset, and won the first place in the 2021 Argoverse Motion Forecasting Competition.
A reconstruction of quantum theory for nonspinning particles

This work is based on the idea that the classical counterpart of quantum theory (QT) is not mechanics but probabilistic mechanics. We therefore choose the theory of statistical ensembles in phase space as starting point for a reconstruction of QT. These ensembles are described by a probability density $\rho (q, p, t)$ and an action variable $S (q, p, t)$. Since the state variables of QT only depend on $q$ and $t$, our first step is to carry out a projection $p \Rightarrow M (q, t)$ from phase space to configuration space. We next show that instead of the momentum components $M_{k}$ one must introduce suitable potentials as dynamic variables. The quasi-quantal theory resulting from the projection is only locally valid. To correct this failure, we have to perform as a second step a linearization or randomization, which ultimately leads to QT. In this work we represent $M$ as an irrotational field, where all components $M_{k}$ may be derived from a single function $S (q, t)$. We obtain the usual Schr\"odinger equation for a nonspinning particle. However, space is three-dimensional and $M$ must be described by $3$ independent functions. In the fourth work of this series, a complete representation of $M$ will be used, which explains the origin of spin. We discuss several fundamental questions that do not depend on the form of $M$ and compare our theory with other recent reconstructions of QT.
Scalable Byzantine Fault Tolerance via Partial Decentralization

Byzantine consensus is a critical component in many permissioned Blockchains and distributed ledgers. We propose a new paradigm for designing BFT protocols called DQBFT that addresses three major performance and scalability challenges that plague past protocols: (i) high communication costs to reach geo-distributed agreement, (ii) uneven resource utilization hampering performance, and (iii) performance degradation under varying node and network conditions and high-contention workloads. Specifically, DQBFT divides consensus into two parts: 1) durable command replication without a global order, and 2) consistent global ordering of commands across all replicas. DQBFT achieves this by decentralizing the heavy task of replicating commands while centralizing the ordering process. Under the new paradigm, we develop a new protocol, Destiny that uses a combination of three techniques to achieve high performance and scalability: using a trusted subsystem to decrease consensus's quorum size, using threshold signatures to attain linear communication costs, reducing client communication. Our evaluations on 300-replica geo-distributed deployment reveal that DQBFT protocols achieve significant performance gains over prior art: ≈3x better throughput and ≈50% better latency.
Combining Modular Skills in Multitask Learning

A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to new tasks. In this work, we assume that each task is associated with a subset of latent discrete skills from a (potentially small) inventory. In turn, skills correspond to parameter-efficient (sparse / low-rank) model parameterisations. By jointly learning these and a task-skill allocation matrix, the network for each task is instantiated as the average of the parameters of active skills. To favour non-trivial soft partitions of skills across tasks, we experiment with a series of inductive biases, such as an Indian Buffet Process prior and a two-speed learning rate. We evaluate our latent-skill model on two main settings: 1) multitask reinforcement learning for grounded instruction following on 8 levels of the BabyAI platform; and 2) few-shot adaptation of pre-trained text-to-text generative models on CrossFit, a benchmark comprising 160 NLP tasks. We find that the modular design of a network significantly increases sample efficiency in reinforcement learning and few-shot generalisation in supervised learning, compared to baselines with fully shared, task-specific, or conditionally generated parameters where knowledge is entangled across tasks. In addition, we show how discrete skills help interpretability, as they yield an explicit hierarchy of tasks.
Robust resource-efficient quantum variational ansatz through an evolutionary algorithm

Variational quantum algorithms (VQAs) are promising methods to demonstrate quantum advantage on near-term devices as the required resources are divided between a quantum simulator and a classical optimizer. As such, designing a VQA which is resource-efficient and robust against noise is a key factor to achieve potential advantage with the existing noisy quantum simulators. It turns out that a fixed VQA circuit design, such as the widely-used hardware efficient ansatz, is not necessarily robust against imperfections. In this work, we propose a genome-length-adjustable evolutionary algorithm to design a robust VQA circuit that is optimized over variations of both circuit ansatz and gate parameters, without any prior assumptions on circuit structure or depth. Remarkably, our method not only generates a noise-effect-minimized circuit with shallow depth, but also accelerates the classical optimization by substantially reducing the number of parameters. In this regard, the optimized circuit is far more resource-efficient with respect to both quantum and classical resources. As applications, based on two typical error models in VQA, we apply our method to calculate the ground energy of the hydrogen and the water molecules as well as the Heisenberg model. Simulations suggest that compared with conventional hardware efficient ansatz, our circuit-structure-tunable method can generate circuits apparently more robust against both coherent and incoherent noise, and hence is more likely to be implemented on near-term devices.
Theoretical Error Analysis of Entropy Approximation for Gaussian Mixture

Gaussian mixture distributions are commonly employed to represent general probability distributions. Despite the importance of using Gaussian mixtures for uncertainty estimation, the entropy of a Gaussian mixture cannot be analytically calculated. Notably, Gal and Ghahramani [2016] proposed the approximate entropy that is the sum of the entropies of unimodal Gaussian distributions. This approximation is easy to analytically calculate regardless of dimension, but there lack theoretical guarantees. In this paper, we theoretically analyze the approximation error between the true entropy and the approximate one to reveal when this approximation works effectively. This error is controlled by how far apart each Gaussian component of the Gaussian mixture. To measure such separation, we introduce the ratios of the distances between the means to the sum of the variances of each Gaussian component of the Gaussian mixture, and we reveal that the error converges to zero as the ratios tend to infinity. This convergence situation is more likely to occur in higher dimensional spaces. Therefore, our results provide a guarantee that this approximation works well in higher dimension problems, particularly in scenarios such as neural networks that involve a large number of weights.
Off-Policy Evaluation with Policy-Dependent Optimization Response

The intersection of causal inference and machine learning for decision-making is rapidly expanding, but the default decision criterion remains an \textit{average} of individual causal outcomes across a population. In practice, various operational restrictions ensure that a decision-maker's utility is not realized as an \textit{average} but rather as an \textit{output} of a downstream decision-making problem (such as matching, assignment, network flow, minimizing predictive risk). In this work, we develop a new framework for off-policy evaluation with \textit{policy-dependent} linear optimization responses: causal outcomes introduce stochasticity in objective function coefficients. Under this framework, a decision-maker's utility depends on the policy-dependent optimization, which introduces a fundamental challenge of \textit{optimization} bias even for the case of policy evaluation. We construct unbiased estimators for the policy-dependent estimand by a perturbation method, and discuss asymptotic variance properties for a set of adjusted plug-in estimators. Lastly, attaining unbiased policy evaluation allows for policy optimization: we provide a general algorithm for optimizing causal interventions. We corroborate our theoretical results with numerical simulations.
A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning

Large datasets as required for deep learning of lip reading do not exist in many languages. In this paper we present the dataset GLips (German Lips) consisting of 250,000 publicly available videos of the faces of speakers of the Hessian Parliament, which was processed for word-level lip reading using an automatic pipeline. The format is similar to that of the English language LRW (Lip Reading in the Wild) dataset, with each video encoding one word of interest in a context of 1.16 seconds duration, which yields compatibility for studying transfer learning between both datasets. By training a deep neural network, we investigate whether lip reading has language-independent features, so that datasets of different languages can be used to improve lip reading models. We demonstrate learning from scratch and show that transfer learning from LRW to GLips and vice versa improves learning speed and performance, in particular for the validation set.
Conservative random walk

Recently, in ["The coin-turning walk and its scaling limit", Electronic Journal of Probability, 25 (2020)], the ``coin-turning walk'' was introduced on ${\mathbb Z}$. It is a non-Markovian process where the steps form a (possibly) time-inhomogeneous Markov chain. In this article, we follow up the investigation by introducing analogous processes in ${\mathbb Z}^d$, $d\ge 2$: at time $n$ the direction of the process is ``updated'' with probability $p_n$; otherwise the next step repeats the previous one. We study some of the fundamental properties of these walks, such as transience/recurrence and scaling limits. Our results complement previous ones in the literature about ``correlated'' (or ``Newtonian'') and ``persistent'' random walks.
Enhancing Legal Argument Mining with Domain Pre-training and Neural Networks

The contextual word embedding model, BERT, has proved its ability on downstream tasks with limited quantities of annotated data. BERT and its variants help to reduce the burden of complex annotation work in many interdisciplinary research areas, for example, legal argument mining in digital humanities. Argument mining aims to develop text analysis tools that can automatically retrieve arguments and identify relationships between argumentation clauses. Since argumentation is one of the key aspects of case law, argument mining tools for legal texts are applicable to both academic and non-academic legal research. Domain-specific BERT variants (pre-trained with corpora from a particular background) have also achieved strong performance in many tasks. To our knowledge, previous machine learning studies of argument mining on judicial case law still heavily rely on statistical models. In this paper, we provide a broad study of both classic and contextual embedding models and their performance on practical case law from the European Court of Human Rights (ECHR). During our study, we also explore a number of neural networks when being combined with different embeddings. Our experiments provide a comprehensive overview of a variety of approaches to the legal argument mining task. We conclude that domain pre-trained transformer models have great potential in this area, although traditional embeddings can also achieve strong performance when combined with additional neural network layers.
Measurement and Analysis of Lobar Lung Deformation After a Change of Patient Position During Video-Assisted Thoracoscopic Surgery

Video-assisted thoracoscopic surgery (VATS) is a minimally invasive surgical technique for the diagnosis and treatment of early-stage lung cancer. During VATS, large lung deformation occurs as a result of a change of patient position and a pneumothorax (lung deflation), which hinders the intraoperative localization of pulmonary nodules. Modeling lung deformation during VATS for surgical navigation is desirable, but the mechanisms causing such deformation are yet not well-understood. In this study, we estimate, quantify and analyze the lung deformation occurring after a change of patient position during VATS. We used deformable image registration to estimate the lung deformation between a preoperative CT (in supine position) and an intraoperative CBCT (in lateral decubitus position) of six VATS clinical cases. We accounted for sliding motion between lobes and against the thoracic wall and obtained consistently low average target registration errors (under 1 mm). We observed large lung displacement (up to 40 mm); considerable sliding motion between lobes and against the thoracic wall (up to 30 mm); and localized volume changes indicating deformation. These findings demonstrate the complexity of the change of patient position phenomenon, which should necessarily be taken into account to model lung deformation for intraoperative guidance during VATS.
ASSIST: Towards Label Noise-Robust Dialogue State Tracking

The MultiWOZ 2.0 dataset has greatly boosted the research on dialogue state tracking (DST). However, substantial noise has been discovered in its state annotations. Such noise brings about huge challenges for training DST models robustly. Although several refined versions, including MultiWOZ 2.1-2.4, have been published recently, there are still lots of noisy labels, especially in the training set. Besides, it is costly to rectify all the problematic annotations. In this paper, instead of improving the annotation quality further, we propose a general framework, named ASSIST (lAbel noiSe-robuSt dIalogue State Tracking), to train DST models robustly from noisy labels. ASSIST first generates pseudo labels for each sample in the training set by using an auxiliary model trained on a small clean dataset, then puts the generated pseudo labels and vanilla noisy labels together to train the primary model. We show the validity of ASSIST theoretically. Experimental results also demonstrate that ASSIST improves the joint goal accuracy of DST by up to 28.16% on MultiWOZ 2.0 and 8.41% on MultiWOZ 2.4, compared to using only the vanilla noisy labels.
Do We Need Anisotropic Graph Neural Networks?

Graph Benchmark (Hu et al., 2020). These datasets cover a wide range of domains, cover both transductive and inductive tasks, and are larger than datasets which are typically used in GNN works. We use evaluation metrics and splits speciﬁed by these papers. Baseline architectures chosen reﬂect popular general-purpose choices (Kipf & Welling, 2017; Xu et al., 2019; Hamilton et al., 2017; Veli ˇ ckovi ´ c et al., 2018; Gilmer et al., 2017), along with the state-of-the-art PNA (Corso et al., 2020) and GATv2 (Brody et al., 2022) architectures.
LobsDICE: Offline Imitation Learning from Observation via Stationary Distribution Correction Estimation

We consider the problem of imitation from observation (IfO), in which the agent aims to mimic the expert’s behavior from the state-only demonstrations by experts. We additionally assume that the agent cannot interact with the environment but has access to the action-labeled transition data collected by some agent with unknown quality. This ofﬂine setting for IfO is appealing in many real-world scenarios where the ground-truth expert actions are inaccessible and the arbitrary environment interactions are costly or risky. In this paper, we present LobsDICE, an ofﬂine IfO algorithm that learns to imitate the expert policy via optimization in the space of stationary distributions. Our algorithm solves a single convex minimization problem, which minimizes the divergence between the two state-transition distributions induced by the expert and the agent policy. On an extensive set of ofﬂine IfO tasks, LobsDICE shows promising results, outperforming strong baseline algorithms.