dataesr/scientific-paragraphs-categorization huggingface dataset:
Deep networks and complex backpropagation problems require several nested applications of the chain rule: symbolic differentiation methods become unfeasible because the resulting expression becomes too expensive to compute, whereas numerical differentiation incurs in rounding and truncation errors due to reaching machine precision in the representation of floating numbers. Automatic differentiation is a technique to obtain fast and exact numerical derivatives with no overhead: its astounding performance powers up virtually any modern deep learning or, broadly, gradient-based algorithm. We describe automatic differentiation, as well as the other approaches to computer derivatives, in deeper detail in Section 2. The fast progress of computer science in the past twenty years, together with the development of deep neural networks and optimization software based on differentiable programming, offers us an unprecedented opportunity to rethink the foundations of our design strategies, and to identify and investigate novel, possibly revolutionary solutions we have been unable to figure out by ourselves.
We must remark that the optimization of complex detector design typically involves a nonconvex (with respect to the design parameters) loss function landscape, akin to that of deep learning applications. In this case, stochastic gradient descent (SGD)  [72]  is not guaranteed to find the global minimum. Empirically, though, SGD seems to find correctly the global minimum even for nonconvex loss functions. A review of recent literature in mathematics concerning convergence properties of SGD in nonconvex landscapes can be found in Ref. [73]. Many popular optimization algorithms have been implemented in open-source packages such as SciPy  [74] ; Fig. 2 (a) shows an example calling a SciPy optimizer with the gradient of the objective supplied by the PyTorch's AD module. Essentially, the user must provide an initial solution and code to evaluate both  and its derivatives automatically.
Numerical differentiation is usually easy to implement and its time complexity scales linearly with the number of input variables. Nevertheless, the aforementioned presence of truncation and rounding errors makes numerical differentiation an undesirable technique. Analytic differentiation by hand, and symbolic differentiation by a computer algebra system such as Mathematica, provide exact derivatives as mathematical (symbolic) expressions. These are however usually only applicable to conventional closed-form expressions that e. g. cannot easily describe programming-language concepts such as loops and control flow. Symbolic differentiation also involves expression swell, where the derivative expression obtained can be significantly more costly to compute than the original expression in terms of computational complexity.
Alternatively, AD arithmetics can be added by a modified or special compiler  [85, 86]  or through source-to-source transformation tools  [87] [88] [89] [90]. An extensive overview of AD tools can be consulted online  [91]. In some cases, the entire simulation code needs to be rewritten in the AD framework, which easily makes this approach prohibitive. In Ref. [92] , AD was added to the general purpose matrix element generator MadGraph  [93]  by using JAX  [94]. However, some modifications of the original code were necessary for the implementation to work; this is very typical for complex codes, as we outline infra.
Surrogate models Algorithmic differentiation can also be applied to a surrogate model instead of the original objective function  [58, 98]. While the latter can be a very complex code dealing with the very specific design problem at hand, a surrogate model is usually taken from a simple and generic class of functions, such as various neural network architectures commonly used in deep learning. The set of parameters (or weights) specifying the surrogate is determined by means of a fitting (or training) procedure that makes it imitate the original objective. With a surrogate based on a deep-learning architecture, AD is immediately available within the machine learning framework used to train the surrogate. Note that the surrogate can be differentiable even if the original function is not. In addition, evaluation of the surrogate (and also its derivatives) is usually several orders of magnitude faster  [99]  than the evaluation of the ''true'' model, mainly due to vectorization and access to hardware parallelism of GPUs and TPUs available in machine learning libraries.
An example will clarify this point. Consider a set of silicon tracking layers made up of two strip sensors glued back-to-back, such as those that constitute the MUonE detector (described in Section 4. 1. 6). Almost any relevant figure of merit for such a detector is heavily dependent on the resolution of the track parameters, which may be extracted from a fit to the reconstructed hit positions. We define as hit the binary signal given by an electronic module that fires when a particle passes through it. The latter will in turn be subjected to systematic uncertainties due to positioning and alignment of the sensors: there is thus a clear path through which these systematic uncertainties affect the utility function. For silicon strip sensors, the root-mean-square hit resolution on the coordinate orthogonal to the strips equals ∕ √ 12, where  is the pitch, if the particle ionization causes the system to detect a single-strip hit  4  ; however, a significant resolution gain is obtained if particles create multiple-strip hits, as charge sharing then provides added information on the track intercept.
The number of these voxels, usually of the order (100-1000), should be chosen to be as large as possible to have fine-grained description of the detector module at hand. However, deep generative models do not work with an arbitrarily high number of dimensions, so the number of voxels cannot be too large. Current state-of-the-art calorimeter shower simulations uses at most (10 4 ) voxel: Ref. [108]  uses 65k voxels and Ref. [109]  uses 27k voxels. The problem can be mitigated to some extent by using aggressive parallelization and memory offloading based on CUDA kernels,  5  but more radical and effective solutions will require dedicated hardware such as neuromorphic chips  [110]  for energy-efficient processing or quantum computers  [111]  for quantum supremacy (optimizing a problem of a given complexity faster or with less computational resources).
In order to be differentiable in the additional parameters  that describe the physical properties of the detector, its geometry, or other features of interest for the optimization task, the models have to be trained with  as a conditional label  [112]. Examples for  are discussed in Section 4. 1. 2 and include thickness, position, angle, or size of detector modules (as long as the number of voxels remains constant); tuning parameters of GEANT4; or atomic number  of the detector material. Conversely, there are a few parameters that cannot be studied. Since the dimensionality of the feature space is hard-coded into the models and given by the training data, the number of voxels is fixed. The resulting augmented dataset (, ) is then used to train one of the deep generative models described in the next subsection.
Learning the simulated particle interactions with matter through surrogates The models which we will consider for learning the datasets mentioned supra strongly depend on the situation at hand. As an illustration, we focus on modeling the shower produced by an energetic particle in a calorimeter for a collider detector with deep generative models. Based on the similarity of voxelized calorimeter showers and pixelated digital images, all deep generative models used to generate artificial images can also be used to learn calorimeter showers, albeit some modifications improve the quality of the samples. Note that the task is always to reproduce the shower of a single incoming particle in a given submodule of a larger detector. Such showers are statistically independent of each other and several of them can be combined into the simulation of a more complicated, single event.
To further ensure that the Jacobian can be evaluated in linear time, the architectures must be either bipartite  [156]  or autoregressive  [157, 158]. The former run equally fast in both directions (density estimation for training and sampling generation for application), the latter strongly favor one direction over the other. Masked Autoregressive Flows (MAFs)  [157]  are fast in density estimation, but slower in sampling by a factor , given by the dimension of the space to be learned. Inverse Autoregressive Flows (IAFs)  [158]  are fast in sampling, but a factor  slower in estimating the density of data points. Only recently, NFs (based on MAF and IAF architectures) have been applied to calorimeter shower simulation  [159, 160] , surpassing the quality of showers generated by an older GAN trained on the same dataset  [132, 133].
With this approximation it is possible to construct a summary statistic for each batch by computing the Asimov Poisson-count likelihood  : ŝ𝑖 (𝒙; 𝝓) = ∑ 𝑥 𝑒 𝑓 𝑖 (𝒙;𝜙)∕𝜏 ∑ 𝑏 𝑗=0 𝑒 𝑓 𝑗 (𝒙;𝝓)∕𝜏 (13) 𝐴 (𝜽; 𝝓) = 𝑏 ∏ 𝑖=0 Pois ( ŝ𝑖 ( 𝐺 𝑠 ; 𝝓 ) | ŝ𝑖 ( 𝐺 𝑠 ; 𝝓 )). ( 14 ) where with the HEP jargon term Asimov we indicate that the value of  is computed with the expected values based on the simulated samples   , such that the maximum likelihood estimator for the Asimov likelihood is the parameter vector   used to generate the simulated dataset. In Statistics, this is also referred to as a saturated model. From the Asimov likelihood the Fisher information matrix is then calculated via automatic differentiation, according to 𝑰(𝜽) 𝑖𝑗 = 𝜕 2 𝜕𝜃 𝑖 𝜕𝜃 𝑗 ( -log 𝐴 (𝜽; 𝝓) ). ( 15 ) The covariance matrix can be estimated from the inverse of the Fisher information matrix if θ is an unbiased estimator of the values of : cov 𝜽 ( θ) ≥ 𝐼(𝜽) -1.
neos takes advantage of recent progress in the pyhf package, which implements full HistFactory likelihoods and their inference using many different automatic differentiation backends, e. g. TensorFlow, PyTorch, and JAX  [193, 194]. Despite their close similarity, differences clearly exist in the implementations of INFERNO and neos from a software standpoint. To this end, work is being done on a library that serves as a toolbox of drop-in differentiable operations that are designed for use in this kind of workflow, called relaxed  [195]. This will facilitate design of a pipeline with easily exchangeable components, allowing for more flexibility in algorithm design, and easier replicability through a unified implementation. Moreover, both INFERNO and neos have been separately implemented with this library, and one can trivially combine elements of each approach to reach a potentially more optimal analysis pipeline.
g. active components material, photo-detection and light transport techniques. • Global cost (, ) can be expressed as a function of local cost parameters  and a set of parameters  describing the overall detector conception, such as number and size of detector modules and their respective positions. Modeling the dynamics of global cost with respect to local costs may seem unfeasible for large scale detectors such as experiments at the LHC or future colliders (see Section 4. 1), but can surely be done for setups of moderate complexity such as cosmic-muon trackers for muon radiography (see Section 4. 3), and it is indeed one of the features being included in the TomOpt package described infra (Section 4. 3. 3). For most applications, a similar detection performance can be achieved by several different technologies.
This is essential for running end-to-end simulations and optimizations, as different parts of accelerators typically are simulated with different codes depending on the relevant physics effects for each section. Examples include frameworks such as LUME  [224]  to run different simulations and stitch them together, and Xopt  [225]  to drive high-level optimization and save datasets. This also includes standards for describing the beam distribution (such as OpenPMD  [226] [227] [228] ) and other aspects of the simulation and data sets (such as those used in LUME  [224] ). Efforts to develop standards for exchanging information with user experiments is also underway, for example in doing end-to-end simulations of accelerator-based light source user experiments. This provides a solid foundation for beginning to integrate accelerator simulations with particle physics detector simulations and analyses.
Fig. 9  demonstrates the quality of the spatial reconstruction of the calorimeter cluster for the case of the LHCb 4 cm and 12 cm modules  [238]. An automatically trained ML model (based on XGBoost  [239] , in this case), agnostic to particular calorimeter details, produces a slightly better performance than the manually selected parametric model. Importantly, the automatically trained generic regressor provides a performance comparable with that of the manually tuned one. This justifies the use of this surrogate regressor Fig. 9. ML-based reconstruction of the calorimeter cluster position provides spatial resolution similar to the customized reconstruction procedure, but without a priori knowledge about the particular spatial properties of the calorimeter under study. Left -correlation between cluster center and the true track position; middle -correlation corrected using parameterized correction; right -correlation using ML trained regressor.
Second, automatic differentiation techniques produce computationally efficient analysis codes. They are significantly cheaper than resampling techniques (see Ref. [254]  for an explicit comparison). Finally, some of the ideas discussed here, such as how to deal with Monte Carlo data, and how to determine efficiently derivatives of complicated iterative processes, can potentially be applied to other problems. Numerical implementations of these analysis techniques are freely available: Fortran https://gitlab. ift. uam-csic. es/alberto/aderrors Julia https://gitlab. ift. uam-csic. es/alberto/aderrors. jl Python https://github. com/fjosw/pyerrors.
Figures of merit. The development of the NuRadioMC simulation code  [294, 295]  enables a precise estimation of the figures of merits for arbitrary detector designs. The figures of merit are (1) the overall sensitivity of the detector, i. e. , the expected rate of detected neutrinos; (2) the achievable reconstruction resolution of the neutrino energy, direction and flavor; (3) the ability to reject rare backgrounds. These three quantities can be combined into high-level sensitivity estimates, e. g. , diffuse flux sensitivity or the discovery potential for point sources. At the same time, the monetary cost of the detector as well as deployment and engineering constraints need to be considered to formulate a sensible objective function; otherwise, a larger detector with more radio detector stations would always provide better performance and be preferred.
Part (1) can be achieved using surrogate models (e. g. through a generative adversarial neural network). Part (2) can be achieved through a deep-learning-based reconstruction method. The MC data set itself is used to learn to predict the neutrino direction and energy. The performance of the network is validated on an independent test dataset and used as a proxy for the reconstruction performance. Initial work (e. g. Refs. [296, 297] ) shows that deep neural networks are capable of reconstructing the neutrino properties with good precision; however, large training data sets of >1 million events are required. The demand of training data size can be reduced by starting with a pre-trained network; i. e. , a large training data set only needs to be produced once, if an efficient data resampling scheme is developed.
TomOpt: Differential Muon tomography optimization TomOpt is the first concrete step within the Mode collaboration to investigate the practicality and scalability of the optimization pipeline proposed in Section 1. Muon tomography offers a comparatively simple optimization use case among detector systems. As discussed above, the optimization of such detectors can have a significant benefit, while allowing us to simultaneously bring up our own understanding of the requirements, feasibility, and potential limitations of physics-goal-oriented systems optimization. Package overview. While still in the development phase, the TomOpt package is planned to be a highly-modular, python-based, PyTorch backed  [52] , framework providing users with a full suite for implementing and simulating all required aspects of the optimization pipeline in a flexible and extensible manner: • Muon generation through random sampling of literature muon flux models, e.
); • optimization of the detector system based on the loss, using a variety of standard gradient-based optimizers (SGD  [72] , Adam  [309] , etc. ), over a training cycle backed by a stateful callback system. Given TomOpt's emphasis on modularity, PyTorch was chosen as the back-end graph-computation library due to the torch. nn. Module class, which handles computation in the forwards pass and exposes parameters to the optimizer for the backwards pass. With its relatively abstract definition, but nonetheless generally useful functionality, it provides a convenient parent class for developing the various necessary modules in TomOpt. In its current state, the full pipeline is in place, and development has now moved to focus on improvements to generalization, realism, and inference performance.
Electric arc furnaces are used in the processing of several raw minerals such as silicon or manganese. These furnaces are equipped with three large electrodes inserted vertically in the mineral mixture to produce electric discharges and heat the material. A schematic view of a generic electric furnace can be seen in Fig. 27 (left). The knowledge of the exact position of the electrodes is an interesting parameter that could help to understand the efficiency variations observed in this kind of equipment. Scattering muography can be applied to this problem by simply considering the angular and spatial deviation distributions characterized using their n-quantiles. This information can be used to train a Deep Neural Network using simulated datasets with different electrode locations and regressing over the height of the electrode with respect to the bottom of the furnace.
Therefore, the total optimization objective  for the differential programming pipeline is  = 𝐷 KL (𝑄 || 𝑃 ) +  1 +  2. ( 29 ) We keep the model general, so that it can be adapted to specific experiments. We present example results in Fig. 32  with different regularization results for a Gaussian target distribution. The Julia code and pipeline is available on GitHub 8  and uses the ForwardDiff package  [342]. Beyond beam-line optimization, we can also optimize the geometry of scintillation energy detectors for optimal light transport, leading to better energy resolution and detector uniformity. Such detectors are often used for low-energy particle physics, and we propose using a differentiable programming optimization pipeline as described in Section 3 with a surrogate model from simulations to achieve optimal detector performance.
Guiding principles First of all, the developed pipeline should support a variety of simulation packages that are commonly used in various physics branches: e. g. GEANT4  [102] , Pythia  [343] [344] [345] , Genie  [346] , to mention just a few. Each of the cited packages relies on other software components that should be pre-installed on the system to allow for its execution. Also, to support the execution of the main differential engine, the system would require different sets of software components depending on the task at hand. Secondly, the system should support different scales of devices under optimization: as we have seen supra these range from a coarse, high-level comparison of design options for a simple problem (e. g. the one of the MUonE detector, see infra, Section 4. 1. 6) to a detailed estimation of the performance of an LHC-scale detector entailing the simultaneous consideration of several hundred design parameters; in other words, given the breadth of the possible applications and their very different demands and scale, the usage of computing resources is going to be relatively small in some cases, while it can be enormous in others.
Storage should incorporate both persistent structured data like a relational database for different run results and unstructured data volumes for handling temporary files and intermediate results. Computing resources will be provided with the help of Kubernetes and nodes of the cluster. The code for the computations of individual tasks should be aware of the storage capacities available to it as well as of the computational resources it can use. Part of the system responsible for the overall execution should ensure the availability of storage components, individual task tracking, and result sharing components.
Main software components The optimization system should include the following main software components: • structural storage (database) management; • volume storage management; • cloud compute management; • simulation package connection and software environment configuration; • user task management; • optimization monitoring/benchmarking interface; • black-box optimization runtime (Python, Julia, etc. ); • differentiable optimization runtime (PyTorch  [52] , Tensorflow  [53] , JAX  [94] ). The above is a non-exhaustive list, and is only meant to offer a view of the typical deployment needs of the system we imagine.
Acknowledgments We wish to thank all the participants of the first MODE workshop on differentiable programming that took place in Louvain- la-Neuve (Belgium)  from  6  to 8 September  2021 , for the fruitful discussions that influenced the content of this document. We gratefully acknowledge support by  IRIS-HEP (Institute for Research and Innovation in Software for High Energy Physics ,  National Science Foundation  grant  OAC-1836650 , https://iris-hep. org/) and  JENAA (Joint ECFA-NuPECC-APPEC Activities , http: //www. nupecc. org/jenaa/). A. Giammanco 's work was partially supported by the  EU   Horizon 2020 Research and Innovation Programme  under the  Marie Sklodowska-Curie Grant  Agreement No. 822185  (''INTENSE'') and the  Research and Innovation Action for Security Grant  Agreement No.
101021812  (''SilentBorder''), and by the  Fonds de la Recherche Scientifique -FNRS  under Grants No. T. 0099. 19  and  J. 0070. 21. P. Vischia 's work was supported by the  FNRS  under the Grant No. 40000963  and by the  Ramón Cajal program  under the Project No. RYC2021-033305-I. C. Krause 's work is supported by  DOE  grant  DOE-SC0010008. M. Aehle ,  N. Gauger , and  R. Keidel  gratefully acknowledge the funding of the research training group SIVERT by the  German federal state of Rhineland-Palatinate. The work of  T. Dorigo ,  L. Layer  and  N. Simpson  is supported by a  Marie Sklodowska-Curie Innovative Training Network Fellowship  of the European Commissions  Horizon 2020 Programme  under Contract Number  765710 INSIGHTS. L. Heinrich  and  M. Lamparth  are supported by the  Excellence Cluster ORIGINS , which is funded by the  Deutsche Forschungsgemeinschaft (DFG, German Research Foundation ) under Germany's Excellence Strategy -EXC-2094-390783311.