Miracle-dz/UniFAR-FTUs huggingface dataset:
Closed-form discovery of structural errors in models of chaotic systems by integrating Bayesian sparse regression and data assimilation

Models used for many important engineering and natural systems are imperfect. The discrepancy between the mathematical representations of a true physical system and its imperfect model is called the model error. These model errors can lead to substantial difference between the numerical solutions of the model and the observations of the system, particularly in those involving nonlinear, multi-scale phenomena. Thus, there is substantial interest in reducing model errors, particularly through understanding their physics and sources and leveraging the rapid growth of observational data. Here we introduce a framework named MEDIDA: Model Error Discovery with Interpretability and Data Assimilation. MEDIDA only requires a working numerical solver of the model and a small number of noise-free or noisy sporadic observations of the system. In MEDIDA, first the model error is estimated from differences between the observed states and model-predicted states (the latter are obtained from a number of one-time-step numerical integrations from the previous observed states). If observations are noisy, a data assimilation (DA) technique such as ensemble Kalman filter (EnKF) is first used to provide a noise-free analysis state of the system, which is then used in estimating the model error. Finally, an equation-discovery technique, such as the relevance vector machine (RVM), a sparsity-promoting Bayesian method, is used to identify an interpretable, parsimonious, closedform representation of the model error. Using the chaotic Kuramoto-Sivashinsky (KS) system as the test case, we demonstrate the excellent performance of MEDIDA in discovering different types of structural/parametric model errors, representing different types of missing physics, using noise-free and noisy observations.
Thermodynamics of a dilute Bose gas: A path-integral Monte Carlo study

We present precise path-integral Monte-Carlo results for the thermodynamics of a homogeneous dilute Bose gas. Pressure and energy are calculated as a function of temperature both below and above the Bose-Einstein transition. Specifically, we address interaction effects, focusing on deviations from the ideal gas law in the thermodynamic limit. We also calculate the isothermal compressibility and the contact parameter, which provide a clear signature of the role played by interactions. In particular, we show that the compressibility exhibits a discontinuity at the transition point. To gain physical insight, numerical results are systematically compared with the predictions of first-order Hartree-Fock and second-order Popov theories, both giving an approximate description of the gas thermodynamics. The comparison shows the extension of the critical region around the transition point, where the inaccuracies of the perturbative expansions are more pronounced.
Contraction-Based Methods for Stable Identification and Robust Machine Learning: a Tutorial

This tutorial paper provides an introduction to recently developed tools for machine learning, especially learning dynamical systems (system identification), with stability and robustness constraints. The main ideas are drawn from contraction analysis and robust control, but adapted to problems in which large-scale models can be learnt with behavioural guarantees. We illustrate the methods with applications in robust image recognition and system identification.
Dip coating of bidisperse particulate suspensions

Abstract Dip coating consists of withdrawing a substrate from a bath to coat it with a thin liquid layer. This process is well understood for homogeneous fluids, but heterogeneities, such as particles dispersed in liquid, lead to more complex situations. Indeed, particles introduce a new length scale, their size, in addition to the thickness of the coating film. Recent studies have shown that, at first order, the thickness of the coating film for monodisperse particles can be captured by an effective capillary number based on the viscosity of the suspension, providing that the film is thicker than the particle diameter. However, suspensions involved in most practical applications are polydisperse, characterized by a wide range of particle sizes, introducing additional length scales. In this study, we investigate the dip coating of suspensions having a bimodal size distribution of particles. We show that the effective viscosity approach is still valid in the regime where the coating film is thicker than the diameter of the largest particles, although bidisperse suspensions are less viscous than monodisperse suspensions of the same solid fraction. We also characterize the intermediate regime that consists of a heterogeneous coating layer and where the composition of the film is different from the composition of the bath. A model to predict the probability of entraining the particles in the liquid film depending on their sizes is proposed and captures our measurements. In this regime, corresponding to a specific range of withdrawal velocities, capillarity filters the large particles out of the film.
Powerpropagation: A sparsity inducing weight reparameterisation

The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling the effective scaling up of models. Whereas much work over the years has been dedicated to specialised pruning techniques, little attention has been paid to the inherent effect of gradient based training on model sparsity. In this work, we introduce Powerpropagation, a new weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, our method gives rise to weight updates exhibiting a"rich get richer"dynamic, leaving low-magnitude parameters largely unaffected by learning. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Powerpropagation is general, intuitive, cheap and straight-forward to implement and can readily be combined with various other techniques. To highlight its versatility, we explore it in two very different settings: Firstly, following a recent line of work, we investigate its effect on sparse training for resource-constrained settings. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark. Secondly, we advocate the use of sparsity in overcoming catastrophic forgetting, where compressed representations allow accommodating a large number of tasks at fixed model capacity. In all cases our reparameterisation considerably increases the efficacy of the off-the-shelf methods.
Score-Based Generative Classifiers

The tremendous success of generative models in recent years raises the question whether they can also be used to perform classification. Generative models have been used as adversarially robust classifiers on simple datasets such as MNIST, but this robustness has not been observed on more complex datasets like CIFAR-10. Additionally, on natural image datasets, previous results have suggested a trade-off between the likelihood of the data and classification accuracy. In this work, we investigate score-based generative models as classifiers for natural images. We show that these models not only obtain competitive likelihood values but simultaneously achieve state-of-the-art classification accuracy for generative classifiers on CIFAR-10. Nevertheless, we find that these models are only slightly, if at all, more robust than discriminative baseline models on out-of-distribution tasks based on common image corruptions. Similarly and contrary to prior results, we find that score-based are prone to worst-case distribution shifts in the form of adversarial perturbations. Our work highlights that score-based generative models are closing the gap in classification accuracy compared to standard discriminative models. While they do not yet deliver on the promise of adversarial and out-of-domain robustness, they provide a different approach to classification that warrants further research.
Reconfigurable Intelligent Surfaces Based on Single, Group, and Fully Connected Discrete-Value Impedance Networks

Reconﬁgurable intelligent surfaces (RISs) allow to control the propagation environment in wireless networks by properly tuning multiple reﬂecting elements. Traditionally, RISs have been realized through a single connected architecture, where each RIS element is controlled by an impedance connected to ground. In a recent work, this architecture has been generalized by realizing RISs through group and fully connected impedance networks. However, impedance networks reconﬁgurable with arbitrary precision are hard to realize in practice. In addition, it is still unexplored how to group together the RIS elements in group connected architectures. These two problems are addressed in this paper. Firstly, we propose a RIS design strategy based on reconﬁgurable impedance networks with discrete values. Secondly, we present three approaches to design the grouping strategy in group connected RISs. Numerical results show that fewer resolution bits are necessary to achieve the performance upper bound as the group size increases. While four resolution bits are needed in single connected architectures, only a single resolution bit is sufﬁcient in fully connected ones. In addition, we show that by dynamically optimizing the grouping strategy, RISs with group size 4 nearly achieve the same performance as fully connected RISs, with reduced hardware complexity.
Divergence-Regularized Multi-Agent Actor-Critic

Entropy regularization is a popular method in reinforcement learning (RL). Although it has many advantages, it alters the RL objective of the original Markov Decision Process (MDP). Though divergence regularization has been proposed to settle this problem, it cannot be trivially applied to cooperative multi-agent reinforcement learning (MARL). In this paper, we investigate divergence regularization in cooperative MARL and propose a novel off-policy cooperative MARL framework, divergence-regularized multi-agent actor-critic (DMAC). Theoretically, we derive the update rule of DMAC which is naturally off-policy and guarantees monotonic policy improvement and convergence in both the original MDP and divergence-regularized MDP. We also give a bound of the discrepancy between the converged policy and optimal policy in the original MDP. DMAC is a flexible framework and can be combined with many existing MARL algorithms. Empirically, we evaluate DMAC in a didactic stochastic game and StarCraft Multi-Agent Challenge and show that DMAC substantially improves the performance of existing MARL algorithms.
ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception

We present ASH, a modern and high-performance framework for parallel spatial hashing on GPU. Compared to existing GPU hash map implementations, ASH achieves higher performance, supports richer functionality, and requires fewer lines of code (LoC) when used for implementing spatially varying operations from volumetric geometry reconstruction to differentiable appearance reconstruction. Unlike existing GPU hash maps, the ASH framework provides a versatile tensor interface, hiding low-level details from the users. In addition, by decoupling the internal hashing data structures and key-value data in buffers, we offer direct access to spatially varying data via indices, enabling seamless integration to modern libraries such as PyTorch. To achieve this, we 1) detach stored key-value data from the low-level hash map implementation; 2) bridge the pointer-first low level data structures to index-first high-level tensor interfaces via an index heap; 3) adapt both generic and non-generic integer-only hash map implementations as backends to operate on multi-dimensional keys. We first profile our hash map against state-of-the-art hash maps on synthetic data to show the performance gain from this architecture. We then show that ASH can consistently achieve higher performance on various large-scale 3D perception tasks with fewer LoC by showcasing several applications, including 1) point cloud voxelization, 2) retargetable volumetric scene reconstruction, 3) non-rigid point cloud registration and volumetric deformation, and 4) spatially varying geometry and appearance refinement. ASH and its example applications are open sourced in Open3D (http://www.open3d.org).
Stochastic Modeling for Learnable Human Pose Triangulation

We propose a stochastic modeling framework for 3D human pose triangulation and evaluate its performance across different datasets and spatial camera arrangements. The common approach to 3D pose estimation is to ﬁrst detect 2D keypoints in images and then apply the triangulation from multiple views. However, the majority of existing triangulation models are limited to a single dataset, i.e. camera arrangement and their number. Moreover, they require known camera parameters. The proposed stochastic pose triangulation model successfully generalizes to different camera arrangements and between two public datasets. In each step, we generate a set of 3D pose hypotheses obtained by triangulation from a random subset of views. The hypotheses are evaluated by a neural network and the expectation of the tri-angulation error is minimized. The key novelty is that the network learns to evaluate the poses without taking into account the spatial camera arrangement, thus improving generalization. Additionally, we demonstrate that the proposed stochastic framework can also be used for fundamental matrix estimation, showing promising results towards relative camera pose estimation from noisy keypoint correspondences.
Learning to Predict Trustworthiness with Steep Slope Loss

Understanding the trustworthiness of a prediction yielded by a classifier is critical for the safe and effective use of AI models. Prior efforts have been proven to be reliable on small-scale datasets. In this work, we study the problem of predicting trustworthiness on real-world large-scale datasets, where the task is more challenging due to high-dimensional features, diverse visual concepts, and large-scale samples. In such a setting, we observe that the trustworthiness predictors trained with prior-art loss functions, i.e., the cross entropy loss, focal loss, and true class probability confidence loss, are prone to view both correct predictions and incorrect predictions to be trustworthy. The reasons are two-fold. Firstly, correct predictions are generally dominant over incorrect predictions. Secondly, due to the data complexity, it is challenging to differentiate the incorrect predictions from the correct ones on real-world large-scale datasets. To improve the generalizability of trustworthiness predictors, we propose a novel steep slope loss to separate the features w.r. t. correct predictions from the ones w.r. t. incorrect predictions by two slide-like curves that oppose each other. The proposed loss is evaluated with two representative deep learning models, i.e., Vision Transformer and ResNet, as trustworthiness predictors. We conduct comprehensive experiments and analyses on ImageNet, which show that the proposed loss effectively improves the generalizability of trustworthiness predictors. The code and pre-trained trustworthiness predictors for reproducibility are available at https://github.com/luoyan407/predict_trustworthiness.
Learning Reward Functions from Scale Feedback

Today's robots are increasingly interacting with people and need to efficiently learn inexperienced user's preferences. A common framework is to iteratively query the user about which of two presented robot trajectories they prefer. While this minimizes the users effort, a strict choice does not yield any information on how much one trajectory is preferred. We propose scale feedback, where the user utilizes a slider to give more nuanced information. We introduce a probabilistic model on how users would provide feedback and derive a learning framework for the robot. We demonstrate the performance benefit of slider feedback in simulations, and validate our approach in two user studies suggesting that scale feedback enables more effective learning in practice.
Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation

We address the non-convex optimisation problem of finding a sparse matrix on the Stiefel manifold (matrices with mutually orthogonal columns of unit length) that maximises (or minimises) a quadratic objective function. Optimisation problems on the Stiefel manifold occur for example in spectral relaxations of various combinatorial problems, such as graph matching, clustering, or permutation synchronisation. Although sparsity is a desirable property in such settings, it is mostly neglected in spectral formulations since existing solvers, e.g. based on eigenvalue decomposition, are unable to account for sparsity while at the same time maintaining global optimality guarantees. We fill this gap and propose a simple yet effective sparsity-promoting modification of the Orthogonal Iteration algorithm for finding the dominant eigenspace of a matrix. By doing so, we can guarantee that our method finds a Stiefel matrix that is globally optimal with respect to the quadratic objective function, while in addition being sparse. As a motivating application we consider the task of permutation synchronisation, which can be understood as a constrained clustering problem that has particular relevance for matching multiple images or 3D shapes in computer vision, computer graphics, and beyond. We demonstrate that the proposed approach outperforms previous methods in this domain.
A Cramér Distance perspective on Non-crossing Quantile Regression in Distributional Reinforcement Learning

Distributional reinforcement learning (DRL) extends the value-based approach by using a deep convolutional network to approximate the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performances. Quantile-based methods like QR-DQN project arbitrary distributions onto a parametric subset of staircase distributions by minimizing the 1-Wasserstein distance, however, due to biases in the gradients, the quantile regression loss is used instead for training, guaranteeing the same minimizer and enjoying unbiased gradients. Recently, monotonicity constraints on the quantiles have been shown to improve the performance of QR-DQN for uncertainty-based exploration strategies. The contribution of this work is in the setting of ﬁxed quantile levels and is twofold. First, we prove that the Cramér distance yields a projection that coincides with the 1-Wasserstein one and that, under monotonicity constraints, the squared Cramér and the quantile regression losses yield collinear gradients, shedding light on the connection between these important elements of DRL. Second, we propose a novel non-crossing neural architecture that allows a good training performance using a novel algorithm to compute the Cramér distance, yielding signiﬁcant improvements over QR-DQN in a number of games of the standard Atari 2600 benchmark.
Mask or Non-Mask? Robust Face Mask Detector via Triplet-Consistency Representation Learning

In the absence of vaccines or medicines to stop COVID-19, one of the effective methods to slow the spread of the coronavirus and reduce the overloading of healthcare is to wear a face mask. Nevertheless, to mandate the use of face masks or coverings in public areas, additional human resources are required, which is tedious and attention-intensive. To automate the monitoring process, one of the promising solutions is to leverage existing object detection models to detect the faces with or without masks. As such, security officers do not have to stare at the monitoring devices or crowds, and only have to deal with the alerts triggered by the detection of faces without masks. Existing object detection models usually focus on designing the CNN-based network architectures for extracting discriminative features. However, the size of training datasets of face mask detection is small, while the difference between faces with and without masks is subtle. Therefore, in this article, we propose a face mask detection framework that uses the context attention module to enable the effective attention of the feed-forward convolution neural network by adapting their attention maps’ feature refinement. Moreover, we further propose an anchor-free detector with Triplet-Consistency Representation Learning by integrating the consistency loss and the triplet loss to deal with the small-scale training data and the similarity between masks and occlusions. Extensive experimental results show that our method outperforms the other state-of-the-art methods. The source code is released as a public download to improve public health at https://github.com/wei-1006/MaskFaceDetection.
Development and External Validation of a Nomogram for Predicting Overall Survival in Stomach Cancer: A Population-Based Study

Objective The study was to develop and externally validate a prognostic nomogram to effectively predict the overall survival of patients with stomach cancer. Methods Demographic and clinical variables of patients with stomach cancer in the Surveillance, Epidemiology, and End Results (SEER) database from 2007–2016 were retrospectively collected. Patients were then divided into the Training Group (n = 4,456) for model development and the Testing Group (n = 4,541) for external validation. Univariate and multivariate Cox regressions were used to explore prognostic factors. The concordance index (C-index) and the Kolmogorov–Smirnov (KS) value were used to measure the discrimination, and the calibration curve was used to assess the calibration of the nomogram. Results Prognostic factors including age, race, marital status, TNM stage, surgery, chemotherapy, grade, and the number of regional nodes positive were used to construct a nomogram. The C-index was 0.790 and the KS value was 0.45 for the Training Group, and the C-index was 0.789 for the Testing Group, all suggesting the good performance of the nomogram. Conclusion We have developed an effective nomogram with ten easily acquired prognostic factors. The nomogram could accurately predict the overall survival of patients with stomach cancer and performed well on external validation, which would help improve the individualized survival prediction and decision-making, thereby improving the outcome and survival of stomach cancer.