julilili42/papers huggingface dataset:
Microstructure, micro-inclusions, and mineralogy along the EGRIP (East Greenland Ice Core Project) ice core – Part 2: Implications for palaeo-mineralogy
2022-02-23

Abstract. Impurities in polar ice do not only allow the reconstruction of past atmospheric aerosol concentrations but also influence the physical properties of the ice. However, the localisation of impurities inside the microstructure is still under debate and little is known about the mineralogy of solid inclusions. In particular, the general mineralogical diversity throughout an ice core and the specific distribution inside the microstructure is poorly investigated; the impact of the mineralogy on the localisation of inclusions and other processes is thus hardly known. We use dust particle concentration, optical microscopy, and cryo-Raman spectroscopy to systematically locate and analyse the mineralogy of micro-inclusions in situ inside 11 solid ice samples from the upper 1340 m of the East Greenland Ice Core Project ice core. Micro-inclusions are more variable in mineralogy than previously observed and are mainly composed of mineral dust (quartz, mica, and feldspar) and sulfates (mainly gypsum). Inclusions of the same composition tend to cluster, but clustering frequency and mineralogy changes with depth. A variety of sulfates dominate the upper 900 m, while gypsum is the only sulfate in deeper samples, which however contain more mineral dust, nitrates, and dolomite. The analysed part of the core can thus be divided into two depth regimes of different mineralogy, and to a lesser degree of spatial distribution, which could originate from different chemical reactions in the ice or large-scale changes in ice cover in northeast Greenland during the mid-Holocene. The complexity of impurity mineralogy on the metre scale and centimetre scale in polar ice is still underestimated, and new methodological approaches are necessary to establish a comprehensive understanding of the role of impurities. Our results show that applying new methods to the mineralogy in ice cores and recognising its complexity, as well as the importance for localisation studies, open new avenues for understanding the role of impurities in ice cores.
Exploring Counterfactual Explanations Through the Lens of Adversarial Examples: A Theoretical and Empirical Analysis
2022-02-23

As machine learning (ML) models become more widely deployed in high-stakes applications, counterfactual explanations have emerged as key tools for providing actionable model explanations in practice. Despite the growing popularity of counterfactual explanations, a deeper understanding of these explanations is still lacking. In this work, we systematically analyze counterfactual explanations through the lens of adversarial examples. We do so by formalizing the similarities between popular counterfactual explanation and adversarial example generation methods identifying conditions when they are equivalent. We then derive the upper bounds on the distances between the solutions output by counterfactual explanation and adversarial example generation methods, which we validate on several real-world data sets. By establishing these theoretical and empirical similarities between counterfactual explanations and adversarial examples, our work raises fundamental questions about the design and development of existing counterfactual explanation algorithms.
The Gaia-ESO Survey: Calibrating the lithium-age relation with open clusters and associations. I. Cluster age range and initial membership selections
2022-02-23

Previous studies of open clusters have shown that lithium depletion is not only strongly age dependent but also shows a complex pattern with other parameters that is not yet understood. For pre- and main-sequence late-type stars, these parameters include metallicity, mixing mechanisms, convection structure, rotation, and magnetic activity. We perform a thorough membership analysis for a large number of stars observed within the Gaia-ESO survey (GES) in the field of 20 open clusters, ranging in age from young clusters and associations, to intermediate-age and old open clusters. Based on the parameters derived from the GES spectroscopic observations, we obtained lists of candidate members for each of the clusters in the sample by deriving RV distributions and studying the position of the kinematic selections in the EW(Li) versus Teff plane to obtain lithium members. We used gravity indicators to discard field contaminants and studied [Fe/H] metallicity to further confirm the membership of the candidates. We also made use of studies using recent data from the Gaia DR1 and DR2 releases to assess our member selections. We identified likely member candidates for the sample of 20 clusters observed in GES (iDR4) with UVES and GIRAFFE, and conducted a comparative study that allowed us to characterize the properties of these members, as well as identify field contaminant stars, both lithium-rich giants and non-giant outliers. This work is the first step towards the calibration of the lithium-age relation and its dependence on other GES parameters. During this project we aim to use this relation to infer the ages of GES field stars, and identify their potential membership to young associations and stellar kinematic groups of different ages.
Partial success in closing the gap between human and machine vision
2022-02-23

A few years ago, the first CNN surpassed human performance on ImageNet. However, it soon became clear that machines lack robustness on more challenging test cases, a major obstacle towards deploying machines "in the wild" and towards obtaining better computational models of human visual perception. Here we ask: Are we making progress in closing the gap between human and machine vision? To answer this question, we tested human observers on a broad range of out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants. We then investigated a range of promising machine learning developments that crucially deviate from standard supervised CNNs along three axes: objective function (self-supervised, adversarially trained, CLIP language-image training), architecture (e.g. vision transformers), and dataset size (ranging from 1M to 1B). Our findings are threefold. (1.) The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets. (2.) There is still a substantial image-level consistency gap, meaning that humans make different errors than models. In contrast, most models systematically agree in their categorisation errors, even substantially different ones like contrastive self-supervised vs. standard supervised models. (3.) In many cases, human-to-model consistency improves when training dataset size is increased by one to three orders of magnitude. Our results give reason for cautious optimism: While there is still much room for improvement, the behavioural difference between human and machine vision is narrowing. In order to measure future progress, 17 OOD datasets with image-level human behavioural data and evaluation code are provided as a toolbox and benchmark at: https://github.com/bethgelab/model-vs-human/
Contrastive Learning Inverts the Data Generating Process
2022-02-23

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.
Laplace Redux -- Effortless Bayesian Deep Learning
2022-02-23

Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection. The Laplace approximation (LA) is a classic, and arguably the simplest family of approximations for the intractable posteriors of deep neural networks. Yet, despite its simplicity, the LA is not as popular as alternatives like variational Bayes or deep ensembles. This may be due to assumptions that the LA is expensive due to the involved Hessian computation, that it is difficult to implement, or that it yields inferior results. In this work we show that these are misconceptions: we (i) review the range of variants of the LA including versions with minimal cost overhead; (ii) introduce "laplace", an easy-to-use software library for PyTorch offering user-friendly access to all major flavors of the LA; and (iii) demonstrate through extensive experiments that the LA is competitive with more popular alternatives in terms of performance, while excelling in terms of computational cost. We hope that this work will serve as a catalyst to a wider adoption of the LA in practical deep learning, including in domains where Bayesian approaches are not typically considered at the moment.
How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
2022-02-23

A precise understanding of why units in an artificial network respond to certain stimuli would constitute a big step towards explainable artificial intelligence. One widely used approach towards this goal is to visualize unit responses via activation maximization. These synthetic feature visualizations are purported to provide humans with precise information about the image features that cause a unit to be activated - an advantage over other alternatives like strongly activating natural dataset samples. If humans indeed gain causal insight from visualizations, this should enable them to predict the effect of an intervention, such as how occluding a certain patch of the image (say, a dog's head) changes a unit's activation. Here, we test this hypothesis by asking humans to decide which of two square occlusions causes a larger change to a unit's activation. Both a large-scale crowdsourced experiment and measurements with experts show that on average the extremely activating feature visualizations by Olah et al. (2017) indeed help humans on this task ( % accuracy; baseline performance without any visualizations is %). However, they do not provide any substantial advantage over other visualizations (such as e.g. dataset samples), which yield similar performance ( % to % accuracy). Taken together, we propose an objective psychophysical task to quantify the benefit of unit-level interpretability methods for humans, and find no evidence that a widely-used feature visualization method provides humans with better "causal understanding" of unit activations than simple alternative visualizations.
Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints
2022-02-23

Evaluating adversarial robustness amounts to finding the minimum perturbation needed to have an input sample misclassified. The inherent complexity of the underlying optimization requires current gradient-based attacks to be carefully tuned, initialized, and possibly executed for many computationally-demanding iterations, even if specialized to a given perturbation model. In this work, we overcome these limitations by proposing a fast minimum-norm (FMN) attack that works with different -norm perturbation models ( ), is robust to hyperparameter choices, does not require adversarial starting points, and converges within few lightweight steps. It works by iteratively finding the sample misclassified with maximum confidence within an -norm constraint of size , while adapting to minimize the distance of the current sample to the decision boundary. Extensive experiments show that FMN significantly outperforms existing attacks in terms of convergence speed and computation time, while reporting comparable or even smaller perturbation sizes.
Shape As Points: A Differentiable Poisson Solver
2022-02-23

In recent years, neural implicit representations gained popularity in 3D reconstruction due to their expressiveness and flexibility. However, the implicit nature of neural implicit representations results in slow inference time and requires careful initialization. In this paper, we revisit the classic yet ubiquitous point cloud representation and introduce a differentiable point-to-mesh layer using a differentiable formulation of Poisson Surface Reconstruction (PSR) that allows for a GPU-accelerated fast solution of the indicator function given an oriented point cloud. The differentiable PSR layer allows us to efficiently and differentiably bridge the explicit 3D point representation with the 3D mesh via the implicit indicator field, enabling end-to-end optimization of surface reconstruction metrics such as Chamfer distance. This duality between points and meshes hence allows us to represent shapes as oriented point clouds, which are explicit, lightweight and expressive. Compared to neural implicit representations, our Shape-As-Points (SAP) model is more interpretable, lightweight, and accelerates inference time by one order of magnitude. Compared to other explicit representations such as points, patches, and meshes, SAP produces topology-agnostic, watertight manifold surfaces. We demonstrate the effectiveness of SAP on the task of surface reconstruction from unoriented point clouds and learning-based reconstruction.
Mind the box: $l_1$-APGD for sparse adversarial attacks on image classifiers
2022-02-23

We show that when taking into account also the image domain , established -projected gradient descent (PGD) attacks are suboptimal as they do not consider that the effective threat model is the intersection of the -ball and . We study the expected sparsity of the steepest descent step for this effective threat model and show that the exact projection onto this set is computationally feasible and yields better performance. Moreover, we propose an adaptive form of PGD which is highly effective even with a small budget of iterations. Our resulting -APGD is a strong white-box attack showing that prior works overestimated their -robustness. Using -APGD for adversarial training we get a robust classifier with SOTA -robustness. Finally, we combine -APGD and an adaptation of the Square Attack to into -AutoAttack, an ensemble of attacks which reliably assesses adversarial robustness for the threat model of -ball intersected with .
Artificial Intelligence Methods in In-Cabin Use Cases: A Survey
2022-02-23

As interest in autonomous driving increases, efforts are being made to meet requirements for the high-level automation of vehicles. In this context, the functionality inside the vehicle cabin plays a key role in ensuring a safe and pleasant journey for driver and passenger alike. At the same time, recent advances in the field of artificial intelligence (AI) have enabled a whole range of new applications and assistance systems to solve automated problems in the vehicle cabin. This paper presents a thorough survey on existing work that utilizes AI methods for use-cases inside the driving cabin, focusing, in particular, on application scenarios related to (1) driving safety and (2) driving comfort. Results from the surveyed works show that AI technology has a promising future in tackling in-cabin tasks within the autonomous driving aspect.
Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
2022-02-23

When engineers train deep learning models, they are very much 'flying blind'. Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing a network's training process solely through these performance indicators is akin to debugging software without access to internal states through a debugger. To address this, we present Cockpit, a collection of instruments that enable a closer look into the inner workings of a learning machine, and a more informative and meaningful status report for practitioners. It facilitates the identification of learning phases and failure modes, like ill-chosen hyperparameters. These instruments leverage novel higher-order information about the gradient distribution and curvature, which has only recently become efficiently accessible. We believe that such a debugging tool, which we open-source for PyTorch, is a valuable help in troubleshooting the training process. By revealing new insights, it also more generally contributes to explainability and interpretability of deep nets.
Synthesis and Thin Films of Thermally Robust Quartet (S = 3/2) Ground State Triradical
2022-02-23

High spin (S=3/2) organic triradicals may offer enhanced properties with respect to several emerging technologies, but those synthesized to date typically exhibit small doublet quartet energy gaps and/or possess limited thermal stability and processability. We report a quartet ground state triradical 3, synthesized by a Pd(0)-catalyzed radical-radical cross-coupling reaction, which possesses two doublet-quartet energy gaps, DeltaEDQ ~ 0.2-0.3 kcal mol-1 and DealtaEDQ2 ~ 1.2-1.8 kcal mol-1. The triradical has a 70+% population of the quartet ground state at room temperature, and good thermal stability with onset of decomposition at &gt; 160 °C under inert atmosphere. Magnetic properties of 3 are characterized by SQUID magnetometry in polystyrene glass and by quantitative EPR spectroscopy. Triradical 3 is evaporated under ultra-high vacuum to form thin films of intact triradicals on silicon substrate, as confirmed by high resolution X-ray photoelectron spectroscopy. AFM and SEM images of the around 1 nm thick films indicate that the triradical molecules form islands on the substrate. The films are stable under ultra-high vacuum for at least 17 h but show onset of decomposition after 4 h at ambient conditions. The drop-cast films are less prone to degradation in air and have longer lifetime.
A Probabilistic State Space Model for Joint Inference from Differential Equations and Data
2022-02-23

Mechanistic models with differential equations are a key component of scientific applications of machine learning. Inference in such models is usually computationally demanding, because it involves repeatedly solving the differential equation. The main problem here is that the numerical solver is hard to combine with standard inference techniques. Recent work in probabilistic numerics has developed a new class of solvers for ordinary differential equations (ODEs) that phrase the solution process directly in terms of Bayesian filtering. We here show that this allows such methods to be combined very directly, with conceptual and numerical ease, with latent force models in the ODE itself. It then becomes possible to perform approximate Bayesian inference on the latent force as well as the ODE solution in a single, linear complexity pass of an extended Kalman filter / smoother - that is, at the cost of computing a single ODE solution. We demonstrate the expressiveness and performance of the algorithm by training, among others, a non-parametric SIRD model on data from the COVID-19 outbreak.
High-Dimensional Gaussian Process Inference with Derivatives
2022-02-23

Although it is widely known that Gaussian processes can be conditioned on observations of the gradient, this functionality is of limited use due to the prohibitive computational cost of in data points and dimension . The dilemma of gradient observations is that a single one of them comes at the same cost as independent function evaluations, so the latter are often preferred. Careful scrutiny reveals, however, that derivative observations give rise to highly structured kernel Gram matrices for very general classes of kernels (inter alia, stationary kernels). We show that in the low-data regime $N
Being a Bit Frequentist Improves Bayesian Neural Networks
2022-02-23

Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this "OOD training" technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN training by exploring four different ways of incorporating OOD data into Bayesian inference. We show in extensive experiments that OOD-trained BNNs are competitive to recent frequentist baselines. This work thus provides strong baselines for future work in Bayesian UQ.
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
2022-02-23

Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present ViViT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first- and second-order directional derivatives. The representation is computed in parallel with gradients in one backward pass and offers a fine-grained cost-accuracy trade-off, which allows it to scale. We demonstrate this by conducting performance benchmarks and substantiate ViViT's usefulness by studying the impact of noise on the GGN's structural properties during neural network training.
Linear-Time Probabilistic Solutions of Boundary Value Problems
2022-02-23

We propose a fast algorithm for the probabilistic solution of boundary value problems (BVPs), which are ordinary differential equations subject to boundary conditions. In contrast to previous work, we introduce a Gauss--Markov prior and tailor it specifically to BVPs, which allows computing a posterior distribution over the solution in linear time, at a quality and cost comparable to that of well-established, non-probabilistic methods. Our model further delivers uncertainty quantification, mesh refinement, and hyperparameter adaptation. We demonstrate how these practical considerations positively impact the efficiency of the scheme. Altogether, this results in a practically usable probabilistic BVP solver that is (in contrast to non-probabilistic algorithms) natively compatible with other parts of the statistical modelling tool-chain.
A straightforward line search approach on the expected empirical loss for stochastic deep learning problems
2022-02-23

A fundamental challenge in deep learning is that the optimal step sizes for update steps of stochastic gradient descent are unknown. In traditional optimization, line searches are used to determine good step sizes, however, in deep learning, it is too costly to search for good step sizes on the expected empirical loss due to noisy losses. This empirical work shows that it is possible to approximate the expected empirical loss on vertical cross sections for common deep learning tasks considerably cheaply. This is achieved by applying traditional one-dimensional function fitting to measured noisy losses of such cross sections. The step to a minimum of the resulting approximation is then used as step size for the optimization. This approach leads to a robust and straightforward optimization method which performs well across datasets and architectures without the need of hyperparameter tuning.