bluuebunny/crossref_metadata_2025_split huggingface dataset:
ConViT: improving vision transformers with soft convolutional inductive biases*
Authors: d’Ascoli Stéphane , Touvron Hugo , Leavitt Matthew L , Morcos Ari S , Biroli Giulio , Sagun Levent 
Published: November 2022

Abstract
Convolutional architectures have proven to be extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision transformers rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ‘soft’ convolutional inductive bias. We initialize the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT (Touvron et al 2020 arXiv:2012.12877) on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analyzing how it has escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit.
Ising model selection using ℓ
1-regularized linear regression: a statistical mechanics analysis*
Authors: Meng Xiangming , Obuchi Tomoyuki , Kabashima Yoshiyuki 
Published: November 2022

Abstract
We theoretically analyze the typical learning performance of ℓ
1-regularized linear regression (ℓ
1-LinR) for Ising model selection using the replica method from statistical mechanics. For typical random regular graphs in the paramagnetic phase, an accurate estimate of the typical sample complexity of ℓ
1-LinR is obtained. Remarkably, despite the model misspecification, ℓ
1-LinR is model selection consistent with the same order of sample complexity as ℓ
1-regularized logistic regression (ℓ
1-LogR), i.e.

M
=
O

log

N

, where N is the number of variables of the Ising model. Moreover, we provide an efficient method to accurately predict the non-asymptotic behavior of ℓ
1-LinR for moderate M, N, such as precision and recall. Simulations show a fairly good agreement between theoretical predictions and experimental results, even for graphs with many loops, which supports our findings. Although this paper mainly focuses on ℓ
1-LinR, our method is readily applicable for precisely characterizing the typical learning performances of a wide class of ℓ
1-regularized M-estimators including ℓ
1-LogR and interaction screening.
On the proliferation of support vectors in high dimensions*
Authors: Hsu Daniel , Muthukumar Vidya , Xu Ji 
Published: November 2022

Abstract
The support vector machine (SVM) is a well-established classification method whose name refers to the particular training examples, called support vectors, that determine the maximum margin separating hyperplane. The SVM classifier is known to enjoy good generalization properties when the number of support vectors is small compared to the number of training examples. However, recent research has shown that in sufficiently high-dimensional linear classification problems, the SVM can generalize well despite a proliferation of support vectors where all training examples are support vectors. In this paper, we identify new deterministic equivalences for this phenomenon of support vector proliferation, and use them to (1) substantially broaden the conditions under which the phenomenon occurs in high-dimensional settings, and (2) prove a nearly matching converse result.
An analytical theory of curriculum learning in teacher–student networks*
Authors: Saglietti Luca , Mannelli Stefano Sarao , Saxe Andrew 
Published: November 2022

Abstract
In animals and humans, curriculum learning—presenting data in a curated order—is critical to rapid learning and effective pedagogy. A long history of experiments has demonstrated the impact of curricula in a variety of animals but, despite its ubiquitous presence, a theoretical understanding of the phenomenon is still lacking. Surprisingly, in contrast to animal learning, curricula strategies are not widely used in machine learning and recent simulation studies reach the conclusion that curricula are moderately effective or even ineffective in most cases. This stark difference in the importance of curriculum raises a fundamental theoretical question: when and why does curriculum learning help? In this work, we analyse a prototypical neural network model of curriculum learning in the high-dimensional limit, employing statistical physics methods. We study a task in which a sparse set of informative features are embedded amidst a large set of noisy features. We analytically derive average learning trajectories for simple neural networks on this task, which establish a clear speed benefit for curriculum learning in the online setting. However, when training experiences can be stored and replayed (for instance, during sleep), the advantage of curriculum in standard neural networks disappears, in line with observations from the deep learning literature. Inspired by synaptic consolidation techniques developed to combat catastrophic forgetting, we propose curriculum-aware algorithms that consolidate synapses at curriculum change points and investigate whether this can boost the benefits of curricula. We derive generalisation performance as a function of consolidation strength (implemented as an L
2 regularisation/elastic coupling connecting learning phases), and show that curriculum-aware algorithms can yield a large improvement in test performance. Our reduced analytical descriptions help reconcile apparently conflicting empirical results, trace regimes where curriculum learning yields the largest gains, and provide experimentally-accessible predictions for the impact of task parameters on curriculum benefits. More broadly, our results suggest that fully exploiting a curriculum may require explicit adjustments in the loss.
Disordered systems insights on computational hardness
Authors: Gamarnik David , Moore Cristopher , Zdeborová Lenka 
Published: November 2022

Abstract
In this review article we discuss connections between the physics of disordered systems, phase transitions in inference problems, and computational hardness. We introduce two models representing the behavior of glassy systems, the spiked tensor model and the generalized linear model. We discuss the random (non-planted) versions of these problems as prototypical optimization problems, as well as the planted versions (with a hidden solution) as prototypical problems in statistical inference and learning. Based on ideas from physics, many of these problems have transitions where they are believed to jump from easy (solvable in polynomial time) to hard (requiring exponential time). We discuss several emerging ideas in theoretical computer science and statistics that provide rigorous evidence for hardness by proving that large classes of algorithms fail in the conjectured hard regime. This includes the overlap gap property, a particular mathematization of clustering or dynamical symmetry-breaking, which can be used to show that many algorithms that are local or robust to changes in their input fail. We also discuss the sum-of-squares hierarchy, which places bounds on proofs or algorithms that use low-degree polynomials such as standard spectral methods and semidefinite relaxations, including the Sherrington–Kirkpatrick model. Throughout the manuscript we present connections to the physics of disordered systems and associated replica symmetry breaking properties.
Transport properties of diffusive particles conditioned to survive in trapping environments
Authors: Pozzoli Gaia , De Bruyne Benjamin 
Published: November 2022

Abstract
We consider a one-dimensional Brownian motion with diffusion coefficient D in the presence of n partially absorbing traps with intensity β, separated by a distance L and evenly spaced around the initial position of the particle. We study the transport properties of the process conditioned to survive up to time t. We find that the surviving particle first diffuses normally, before it encounters the traps, then undergoes a period of transient anomalous diffusion, after which it reaches a final diffusive regime. The asymptotic regime is governed by an effective diffusion coefficient D
eff, which is induced by the trapping environment and is typically different from the original one. We show that when the number of traps is finite, the environment enhances diffusion and induces an effective diffusion coefficient that is systematically equal to D
eff = 2D, independently of the number of the traps, the trapping intensity β and the distance L. On the contrary, when the number of traps is infinite, we find that the environment inhibits diffusion with an effective diffusion coefficient that depends on the traps intensity β and the distance L through a non-trivial scaling function

D

eff

=
D
F

(

β
L
/
D

)

, for which we obtain a closed-form. Moreover, we provide a rejection-free algorithm to generate surviving trajectories by deriving an effective Langevin equation with an effective repulsive potential induced by the traps. Finally, we extend our results to other trapping environments.
Low intensity gamma-ray monitor for in-situ solids fraction measurements in liquids
Authors: Kaur Sraw Talwinder , Yu Bo , Sedgwick Andrea , Junaid Abu , Tsui Ying , Gupta Manisha , Fedosejevs Robert 
Published: November 2022

Abstract
A novel detection system has been designed to measure the settling of solids in tailing ponds in order to facilitate the reuse of water in oil sands extraction processing, tailings excavation, transportation and management. The system is based on a weak gamma ray source and an inexpensive scintillator-based detector. The system measures gamma ray photons which are transmitted through the material of interest which are detected by a combined scintillator and Multi Pixel Photon Counter (MPPC) detector. This non-destructive measurement allows the determination of the solids contents profile versus depth in a tailing pond. The attenuation of gamma radiation depends on the density of the solids within the fluid tailings and thus varies with the weight fraction of solids content of the tailings. Modelling of the system was carried out by Geant4 simulations. The system was deployed using a weak 133Ba gamma-ray source together with a simple microprocessor controller readout circuit to analyze the pulse height response of a Cerium doped Lutetium Yttrium Orthosilicate (LYSO (Ce)) scintillator crystal. The detection system is calibrated with the known samples and can measure solids content with a relative precision of within ∼2%.
Signal amplification in multistage charge collection process in Si detectors in situ irradiated at superfluid helium temperature
Authors: Eremin V. , Shepelev A. , Verbitskaya E. 
Published: November 2022

Abstract
The effect of charge multiplication in Si p+-n-n+ detectors observed in the unique conditions of the in situ irradiation with 23 GeV proton beam fragmented into 400 ms spills and the temperature of superfluid helium, 1.9 K, was analyzed. The current pulse responses of the detector irradiated to 5 × 1013 p/cm2 demonstrated a two-stage process of charge collection, in which, in the first stage, the current was induced by holes, and in the second, by electrons arising due to avalanche multiplication produced by holes. In the detector irradiated to 2.7 × 1014 p/cm2, the third stage of charge collection was observed that was related to avalanche multiplication produced by electrons reaching the n+ contact and sequential drift of generated holes back to the p+ contact. The developed procedure of data treatment allowed extracting the full set of carrier transport parameters and internal charge gain rising to 3.7 at the bias voltage of 400 V and the fluence of 5 × 1013 p/cm2. An evolution of charge multiplication at the p+ contact initiated by holes originated from a high rate of the hole trap introduction at 1.9 K described by a fivefold rise in the trapping probability at 1.9 K in comparison with its value at T = -10°C. This increases the electric field near the p+ contact and leads to the avalanche multiplication affecting the following stages of charge collection.
Active Region Extent Assessment with X-rays (AREA-X)
Authors: Poley Luise , Blue Andrew , Fadeyev Vitaliy , Sperlich Dennis 
Published: November 2022

Abstract
The development of semiconductor sensors for new particle tracking detectors places increasing limits on sensor characteristics such as uniformity, size and shape of inefficient areas and size of active compared to inactive sensor areas. Accurately assessing these relatively subtle effects requires either measurements in particle beams or the modification of samples to be used in dedicated laser test setups.
Active Region Extent Assessment with X-rays (AREA-X) has been developed as an alternative method for the fast, efficient and precise study of the active area of a semiconductor sensor. It uses a monochromatic, micro-focused X-ray beam with a 10–20 keV energy range as provided by several synchrotron beam lines and uses the photo current induced in the sensor to measure the depth of the responsive sensor volume. It can be used to study local inhomogeneities or inefficiencies, the overall extent of the active sensor volume and its shape and its localised application, which makes the need to gather statistics over a large area unnecessary, allowing for fast readout, which enables studies of the sensor behaviour at a range of external parameters, e.g. temperature or applied bias voltage.
This paper presents the measurement concept and technical setup of the measurement, results from initial measurements as well as capabilities and limitations of the method.
Automatic transmission optimization of the plane-grating monochromator using collimated light
Authors: Shi Y. , Tang L. , He J. 
Published: November 2022

Abstract
It is essential to perform plane-grating monochromator transmission optimization for synchrotron radiation beamlines. Usually, The optimization is time-consuming. In this contribution, an automatic transmission optimization method for the plane-grating monochromator using collimated light based on the Differential Evolution (DE) algorithm is introduced. This method can quickly find the optimal solution. To test this method, a program developed using C++ is verified by optimizing the efficiency of a plane-grating monochromator with a 2000 lines/mm lamellar grating at soft X-ray region. Furthermore, the transmission optimizations of most of the PGM using lamellar gratings in soft x-ray region are performed.
A novel hypothesis for how albatrosses optimize their flight physics in real-time: an extremum seeking model and control for dynamic soaring
Authors: Pokhrel Sameer , Eisa Sameh A 
Published: November 2022

Abstract
The albatross optimized flight maneuver—known as dynamic soaring—is nothing but a wonder of biology, physics, and engineering. By utilizing dynamic soaring, this fascinating bird can travel in the desired flight direction almost for free by harvesting energy from the wind. This phenomenon has been observed for centuries as evidenced by the writings of Leonardo da Vinci and Lord Rayleigh. Moreover, dynamic soaring biological inspiration has triggered a momentous interest among many communities of science and engineering, particularly aeronautical, control, and robotic engineering communities. That is, if dynamic soaring is mimicked, we will have arrived at a new class of unmanned aerial vehicles that are very energy-efficient during part (or the full) duration of their flight. Studying, modeling, and simulating dynamic soaring have been conducted in literature by mostly configuring dynamic soaring as an optimal control problem. Said configuration requires accurate dynamic system modeling of the albatross/mimicking-object, accurate wind profile models, and a defined mathematical formula of an objective function that aims at conserving energy and minimizing its dissipation; the solution then of such optimal control problem is the dynamic soaring trajectory taken—or to be taken—by the bird/mimicking-object. Furthermore, the decades-long optimal control configuration of the dynamic soaring problem resulted in non-real-time algorithms and control solutions, which may not be aligned well with the biological phenomenon itself; experimental observations of albatrosses indicate their ability to conduct dynamic soaring in real-time. Indeed, a functioning modeling and control framework for dynamic soaring that allows for a meaningful bio-mimicry of the albatross needs to be autonomous, real-time, stable, and capable of tolerating the absence of mathematical expressions of the wind profiles and the objective function—hypothetically similar to what the bird does. The qualifications of such modeling and control framework are the very same characteristics of the so-called extremum seeking systems. In this paper, we show that extremum seeking systems existing in control literature for decades are a natural characterization of the dynamic soaring problem. We propose an extremum seeking modeling and control framework for the dynamic soaring problem hypothesizing that the introduced framework captures more features of the biological phenomenon itself and allows for possible bio-mimicking of it. We provide and discuss the problem setup, design, and stability of the introduced framework. Our results, supported by simulations and comparison with optimal control methods of the literature, provide a proof of concept that the dynamic soaring phenomenon can be a natural expression of extremum seeking. Hence, dynamic soaring has the potential to be performed autonomously and in real-time with stability guarantees.
Socio-technical modelling of UK energy transition under three global SSPs, with implications for IAM scenarios
Authors: Freeman Rachel , Pye Steve 
Published: November 2022

Abstract
The potential for using findings from socio-technical energy transition (STET) models in integrated assessment models (IAMs) has been proposed by several authors. A STET simulation model called TEMPEST, which includes the influence of societal and political factors in the UK’s energy transition, is used to model three of the global shared socioeconomic pathways (SSPs) at the national level. The SSP narratives are interpreted as inputs to TEMPEST, which drive scenario simulations to reflect varying societal preferences for mitigation measures, the level of political support for energy transition, and future economic and population trends. SSP1 and SSP2 come close to meeting UK net zero targets in 2050 but SSP5 does not reach net zero before 2080. An estimate of the total societal, political, and economic cost of scenarios indicates that while SSP1 achieves the best emissions reductions it also has the highest total cost, and SSP2 achieves the best ratio between rate of emissions reductions and total cost. Feasibility appears to be highest for SSP2 since it is the least different to historical precedent. Current UK government energy strategy is closer to the narrative in SSP5, however, which has the highest total cost and exceeds an estimated carbon budget by 32%. Three key TEMPEST findings are recommended for use in IAMs: (i) the uncertainty in emissions savings due to variability in political and societal support for energy transition, (ii) the influence of negative societal pushback to policies in achievement of expected policy outcomes, and (iii) the combined influence on energy service demand of disposable income, public willingness to participate, and user impacts from measures.
Carbon footprint patterns of domestic migrants in China and 1.5 °C mitigation pathways
Authors: Wang Rui , Chen Guangwu , Wang Yafei , Feng Jiahao 
Published: November 2022

Abstract
Rural–urban migration and the growing carbon footprints (CFs) will challenge the world in keeping the global average-temperature increment below 1.5 °C compared to preindustrial levels. We first quantified the CFs of migrant consumption in 30 Chinese provinces in 2013 and 2015. An integrated approach was developed to match the Chinese multiregional input–output table with household survey data covering 200 000 migrant survey samples. Furthermore, we adopted the three shared socioeconomic pathways (SSPs) to explore the carbon budgets and trajectories of the migrants and permanent residents to align with the 1.5 °C climate target. The results suggest that the average migrant CFs ranged from 2.3 to 2.6 tCO2e per capita in 2015. To align with 1.5 °C pathways, the emissions of migrants should be held below 3.4 tCO2e per capita under the 12 SSPs–1.9 scenarios from 2020 and keep decreasing to net-zero before 2050. To ensure citizenry progress in improving well-being without overshooting dual carbon targets, high-emitting migrants need to reduce their emissions substantially.
Is hydrogen production through natural gas pyrolysis compatible with ambitious climate targets in the United States? A location-specific, time-resolved analysis
Authors: Keller Martin 
Published: November 2022

Abstract
Pyrolysis of natural gas to produce H2 and solid carbon through methane cracking can be characterized as a high-CH4, low-CO2 process. It results in low CO2 emissions because no direct CO2 is generated at the point of H2 generation if solid carbon is not combusted further. However, it results in high CH4 emissions because of its higher natural gas consumption compared to the direct use of natural gas and, thus, higher CH4 losses along the natural gas supply chain. Here, I analyzed whether this process can provide climate benefit in comparison to the direct, unabated utilization of natural gas and also in comparison with H2 produced from water electrolysis with grid electricity. To this end, Monte Carlo simulations of time-resolved and US state-specific emission profiles and their impact on mid-century global warming under different CH4 mitigation scenarios were conducted. It was found that the climate benefit of natural gas pyrolysis is highly dependent on plant location and the speed at which CH4 emissions can be abated. New York, Pennsylvania, and Ohio emerged as the most promising locations. This is because of their projected long reliance on natural gas for power generation, which renders electrolysis using grid electricity less attractive, as well as the relatively low estimate of current CH4 emissions from the natural gas supply chain. However, without fast action on CH4 emission mitigation, the climate benefit of natural gas pyrolysis is small or non-existent, irrespective of the plant location. Overall, the uncertainty in the relative climate benefit of natural gas pyrolysis was found to be large; however, this study developed an easy-to-adapt MS Excel/visual basic for applications (VBA) tool that can be updated as soon as more accurate data on CH4 emissions becomes available. Policymakers, businesspeople, and scholars can use this tool to estimate the climate impact within their own scenarios and locations.