Miracle-dz/UniFAR-FTUs huggingface dataset:
A hybrid discrete-continuum approach to model hydro-mechanical behaviour of soil during desiccation

Desiccation cracking in clayey soils occurs when they lose moisture, leading to an increase in their compressibility and hydraulic conductivity and hence significant reduction of soil strength. The prediction of desiccation cracking in soils is challenging due to the lack of insights into the complex coupled hydro-mechanical process at the grain scale. In this paper, a new hybrid discrete-continuum numerical framework, capable of capturing hydro-mechanical behaviour of soil at both grain and macro scales, is proposed for predicting desiccation cracking in clayey soil. In this framework, a soil layer is represented by an assembly of DEM particles, each occupies an equivalent continuum space and carries physical properties governing unsaturated flow. These particles move freely in the computational space following the discrete element method (DEM), while their contact network and the continuum mixture theory are used to model the unsaturated flow. The dependence of particle-to-particle contact behaviour on water content is represented by a cohesive-frictional contact model, whose material properties are governed by the water content. In parallel with the theoretical development is a series of experiments on 3D soil desiccation cracking to determine essential properties and provide data for the validation of mechanical and physical behaviour. Very good agreement in both physical behaviour (e.g. evolution of water content) and mechanical behaviour (e.g. occurrence and development of cracks, and distribution of compressive and tensile strains) demonstrates that the proposed framework is capable of capturing the hydro-mechanical behaviour of soil during desiccation. The capability of the proposed framework facilitates numerical experiments for insights into the hydro-mechanical behaviour of unsaturated soils that have not been possible before.
CoAtNet: Marrying Convolution and Attention for All Data Sizes

Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced"coat"nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.
Congestion Control in the Cellular-V2X Sidelink

This paper presents a detailed quantitative evaluation of standardised Decentralised Congestion Control (DCC) and packet dropping mechanisms for Cellular V2X (C-V2X). Based on the identified shortcomings, an Access layer DCC scheme, RRI adaptive, is then proposed. RRI adaptive accommodates the sidelink scheduling mechanism Sensing Based Semi-Persistent Scheduling (SB-SPS), eliminating incompatibilities between current standards and the scheduling mechanism, to avoid unnecessary and reoccurring collisions. Two variants are proposed; one is an evolution of the ETSI Reactive DCC mechanism and the other aligns with the 3GPP approach based on channel occupancy ratio (CR). Both approaches are compared with current ETSI and 3GPP standards and exhibit improved performance. An evaluation of existing DCC standards and RRI Adaptive to meet the Quality of Service (QoS) requirements of vehicular cooperative awareness applications is also conducted.
Practical Machine Learning Safety: A Survey and Primer

The open-world deployment of Machine Learning (ML) algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities such as interpretability, verifiability, and performance limitations. Research explores different approaches to improve ML dependability by proposing new models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. In this paper, we review and organize practical ML techniques that can improve the safety and dependability of ML algorithms and therefore ML-based software. Our organization maps state-of-the-art ML techniques to safety strategies in order to enhance the dependability of the ML algorithm from different aspects, and discuss research gaps as well as promising solutions
Mixed dark matter: matter power spectrum and halo mass function

We investigate and quantify the impact of mixed (cold and warm) dark matter models on large-scale structure observables. In this scenario, dark matter comes in two phases, a cold one (CDM) and a warm one (WDM): the presence of the latter causes a suppression in the matter power spectrum which is allowed by current constraints and may be detected in present-day and upcoming surveys. We run a large set of N-body simulations in order to build an efficient and accurate emulator to predict the aforementioned suppression with percent precision over a wide range of values for the WDM mass, Mwdm, and its fraction with respect to the totality of dark matter, fwdm. The suppression in the matter power spectrum is found to be independent of changes in the cosmological parameters at the 2% level for k≲ 10 h/Mpc and z≤ 3.5. In the same ranges, by applying a baryonification procedure on both ΛCDM and CWDM simulations to account for the effect of feedback, we find a similar level of agreement between the two scenarios. We examine the impact that such suppression has on weak lensing and angular galaxy clustering power spectra. Finally, we discuss the impact of mixed dark matter on the shape of the halo mass function and which analytical prescription yields the best agreement with simulations. We provide the reader with an application to galaxy cluster number counts.
Attacking Adversarial Attacks as A Defense

It is well known that adversarial attacks can fool deep neural networks with imperceptible perturbations. Although adversarial training significantly improves model robustness, failure cases of defense still broadly exist. In this work, we find that the adversarial attacks can also be vulnerable to small perturbations. Namely, on adversarially-trained models, perturbing adversarial examples with a small random noise may invalidate their misled predictions. After carefully examining state-of-the-art attacks of various kinds, we find that all these attacks have this deficiency to different extents. Enlightened by this finding, we propose to counter attacks by crafting more effective defensive perturbations. Our defensive perturbations leverage the advantage that adversarial training endows the ground-truth class with smaller local Lipschitzness. By simultaneously attacking all the classes, the misled predictions with larger Lipschitzness can be flipped into correct ones. We verify our defensive perturbation with both empirical experiments and theoretical analyses on a linear model. On CIFAR10, it boosts the state-of-the-art model from 66.16% to 72.66% against the four attacks of AutoAttack, including 71.76% to 83.30% against the Square attack. On ImageNet, the top-1 robust accuracy of FastAT is improved from 33.18% to 38.54% under the 100-step PGD attack.
On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty

Inducing point Gaussian process approximations are often considered a gold standard in uncertainty estimation since they retain many of the properties of the exact GP and scale to large datasets. A major drawback is that they have difficulty scaling to high dimensional inputs. Deep Kernel Learning (DKL) promises a solution: a deep feature extractor transforms the inputs over which an inducing point Gaussian process is defined. However, DKL has been shown to provide unreliable uncertainty estimates in practice. We study why, and show that with no constraints, the DKL objective pushes"far-away"data points to be mapped to the same features as those of training-set points. With this insight we propose to constrain DKL's feature extractor to approximately preserve distances through a bi-Lipschitz constraint, resulting in a feature space favorable to DKL. We obtain a model, DUE, which demonstrates uncertainty quality outperforming previous DKL and other single forward pass uncertainty methods, while maintaining the speed and accuracy of standard neural networks.
Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift. \emph{Deep ensembles}, a state-of-the-art method for uncertainty estimation, only ensemble random initializations of a \emph{fixed} architecture. Instead, we propose two methods for automatically constructing ensembles with \emph{varying} architectures, which implicitly trade-off individual architectures' strengths against the ensemble's diversity and exploit architectural variation as a source of diversity. On a variety of classification tasks and modern architecture search spaces, we show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift. Our further analysis and ablation studies provide evidence of higher ensemble diversity due to architectural variation, resulting in ensembles that can outperform deep ensembles, even when having weaker average base learners. To foster reproducibility, our code is available: \url{https://github.com/automl/nes}
Bias-Robust Bayesian Optimization via Dueling Bandit

We consider Bayesian optimization in settings where observations can be adversarially biased, for example by an uncontrolled hidden confounder. Our first contribution is a reduction of the confounded setting to the dueling bandit model. Then we propose a novel approach for dueling bandits based on information-directed sampling (IDS). Thereby, we obtain the first efficient kernelized algorithm for dueling bandits that comes with cumulative regret guarantees. Our analysis further generalizes a previously proposed semi-parametric linear bandit model to non-linear reward functions, and uncovers interesting links to doubly-robust estimation.
Linear Transformers Are Secretly Fast Weight Programmers

We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow"neural net learns by gradient descent to program the ``fast weights"of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values). Such Fast Weight Programmers (FWPs) learn to manipulate the contents of a finite memory and dynamically interact with it. We infer a memory capacity limitation of recent linearised softmax attention variants, and replace the purely additive outer products by a delta rule-like programming instruction, such that the FWP can more easily learn to correct the current mapping from keys to values. The FWP also learns to compute dynamically changing learning rates. We also propose a new kernel function to linearise attention which balances simplicity and effectiveness. We conduct experiments on synthetic retrieval problems as well as standard machine translation and language modelling tasks which demonstrate the benefits of our methods.
Towards Open-World Recommendation: An Inductive Model-based Collaborative Filtering Approach

Recommendation models can effectively estimate underlying user interests and predict one's future behaviors by factorizing an observed user-item rating matrix into products of two sets of latent factors. However, the user-specific embedding factors can only be learned in a transductive way, making it difficult to handle new users on-the-fly. In this paper, we propose an inductive collaborative filtering framework that contains two representation models. The first model follows conventional matrix factorization which factorizes a group of key users' rating matrix to obtain meta latents. The second model resorts to attention-based structure learning that estimates hidden relations from query to key users and learns to leverage meta latents to inductively compute embeddings for query users via neural message passing. Our model enables inductive representation learning for users and meanwhile guarantees equivalent representation capacity as matrix factorization. Experiments demonstrate that our model achieves promising results for recommendation on few-shot users with limited training ratings and new unseen users which are commonly encountered in open-world recommender systems.
All Tokens Matter: Token Labeling for Training Better Vision Transformers

In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViTs that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to compute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specific supervision generated by a machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous models (250M+) reaching 86%. We also show that token labeling can clearly improve the generalization capability of the pre-trained models on downstream tasks with dense prediction, such as semantic segmentation. Our code and all the training details will be made publicly available at https://github.com/zihangJiang/TokenLabeling.
Dynamics for a two phases free boundaries system in an epidemiological model with nonlocal dispersals

The present paper is devoted to the investigation of the long time dynamics for a double free boundary system with nonlocal diﬀusions, which models the infectious diseases transmitted via digestive system such as fecal-oral diseases, cholera, hand-foot and mouth, etc.We start by proving the existence and uniqueness of the Cauchy problem, which is not a trivial step due to presence of couple dispersals and new types of nonlinear reaction terms. Next, we provide simple conditions on comparing the basic reproduction numbers R 0 and R ∗ with some certain numbers to characterize the global dynamics, as t → ∞ . We further obtain the sharp criteria for the spreading and vanishing in term of the initial data. This is also called the vanishing-spreading phenomena. The couple dispersals yield signiﬁcant obstacle that we cannot employ the approach of Zhao, Zhang, Li, Du [47] and Du-Ni [17]. To overcome this, we must prove the existence and the variational characterization for the principal eigenvalue of a linear system with nonlocal dispersals, then use it to obtain the right limits as the dispersal rates and domain tend to zero or inﬁnity. The maximum principle and sliding method for the nonlocal operator are ingeniously employed to achieve the desired results.
Spatial modelling of COVID-19 incident cases using Richards' curve: an application to the Italian regions

We introduce an extended generalised logistic growth model for discrete outcomes, in which a network structure can be specified to deal with spatial dependence and time dependence is dealt with using an Auto-Regressive approach. A major challenge concerns the specification of the network structure, crucial to consistently estimate the canonical parameters of the generalised logistic curve, e.g. peak time and height. Parameters are estimated under the Bayesian framework, using the {\texttt{ Stan}} probabilistic programming language. The proposed approach is motivated by the analysis of the first and second wave of COVID-19 in Italy, i.e. from February 2020 to July 2020 and from July 2020 to December 2020, respectively. We analyse data at the regional level and, interestingly enough, prove that substantial spatial and temporal dependence occurred in both waves, although strong restrictive measures were implemented during the first wave. Accurate predictions are obtained, improving those of the model where independence across regions is assumed.
The dilemma of quantum neural networks

The core of quantum machine learning is to devise quantum models with good trainability and low generalization error bounds than their classical counterparts to ensure better reliability and interpretability. Recent studies confirmed that quantum neural networks (QNNs) have the ability to achieve this goal on specific datasets. In this regard, it is of great importance to understand whether these advantages are still preserved on real-world tasks. Through systematic numerical experiments, we empirically observe that current QNNs fail to provide any benefit over classical learning models. Concretely, our results deliver two key messages. First, QNNs suffer from the severely limited effective model capacity, which incurs poor generalization on real-world datasets. Second, the trainability of QNNs is insensitive to regularization techniques, which sharply contrasts with the classical scenario. These empirical results force us to rethink the role of current QNNs and to design novel protocols for solving real-world problems with quantum advantages.
AdaptOver : Adaptive Overshadowing of LTE signals

We introduce AdaptOver, a new LTE signal overshadowing attack that allows an adversary to reactively and adaptively overshadow any downlink message between the network and the user equipment (UE). We demonstrate the impact of AdaptOver by using it to launch targeted Denial-of-Service (DoS) attacks on UEs. We implement AdaptOver using a commercially available software-deﬁned radio. Our experiments demonstrate that our DoS attacks cause persistent connection loss lasting more than 12 hours for a wide range of smartphones. DoS attacks based on AdaptOver are stealthier than attacks that relied on the use of fake base stations, and more persistent than existing overshadowing attacks, which caused connection loss of only up to 9 minutes. Given that AdaptOver can reactively overshadow any downlink message, its use is not limited to DoS attacks — it can be used for a wide range of other attacks, e.g., to extract the IMSI from a UE in a stealthier manner than traditional IMSI catchers. We consider AdaptOver to be an essential building block for many attacks against real-world LTE networks. In particular, any fake base station attack that makes use of spoofed downlink messages can be ported to the presented attack method, causing a much more reliable, persistent, and stealthy effect.
No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data

A central challenge in training classification models in the real-world federated system is learning with non-IID data. To cope with this, most of the existing works involve enforcing regularization in local optimization or improving the model aggregation scheme at the server. Other works also share public datasets or synthesized samples to supplement the training of under-represented classes or introduce a certain level of personalization. Though effective, they lack a deep understanding of how the data heterogeneity affects each layer of a deep classification model. In this paper, we bridge this gap by performing an experimental analysis of the representations learned by different layers. Our observations are surprising: (1) there exists a greater bias in the classifier than other layers, and (2) the classification performance can be significantly improved by post-calibrating the classifier after federated training. Motivated by the above findings, we propose a novel and simple algorithm called Classifier Calibration with Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated gaussian mixture model. Experimental results demonstrate that CCVR achieves state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10. We hope that our simple yet effective method can shed some light on the future research of federated learning with non-IID data.
Bayesian estimation of the specific shear and bulk viscosity of the quark-gluon plasma with additional flow harmonic observables

The transport properties of the strongly-coupled quark-gluon plasma created in ultra-relativistic heavy-ion collisions are extracted by Bayesian parameter estimate methods with the latest collision beam energy data from the CERN Large Hadron Collider. This Bayesian analysis includes sophisticated flow harmonic observables for the first time. We found that the temperature dependence of specific shear viscosity appears weaker than in the previous studies. The results prefer a lower value of specific bulk viscosity and a higher switching temperature to reproduce additional observables. However, the improved statistical uncertainties both on the experimental data and hydrodynamic calculations with additional observables do not help to reduce the final credibility ranges much, indicating a need for improving the dynamical collision model before the hydrodynamic takes place. In addition, the sensitivities of experimental observables to the parameters in hydrodynamic model calculations are quantified. It is found that the analysis benefits most from the symmetric cumulants and non-linear flow modes, which mostly reflect non-linear hydrodynamic responses, in constraining the temperature dependence of the specific shear and bulk viscosity in addition to the previously used flow coefficients.