dataesr/scientific-paragraphs-categorization huggingface dataset:
Adversarial nets  [254]  implemented both generator and discriminator as MLPs and demonstrated their applicability to generating images of datasets, such as MNIST  [256]  and CIFAR-10  [257]. GANs have the advantage that Markov chains are never needed, only backprop is used to obtain gradients, no inference is needed during learning, and a wide variety of functions can be incorporated into the model. However, GANs have been known to be unstable to train, and generators often produce nonsensical outputs. DCGANs  [230]  addressed this issue by implementing both generator and discriminator as deep CNNs. The visualization of the convolutional filters learned by DCGANs empirically showed the connections between the filters and specific objects. This was convincing evidence that DCGANs could learn a hierarchy of representations from object parts.
As the objective of SRGANs was to achieve photorealistic single-image SR, the authors proposed a perceptual loss function that consisted of an adversarial loss and a content loss. They also introduced an MOS, which evaluated the qualities of reconstructions by humans. They found out that the SRResNet without the adversarial component sets a new state of the art on public benchmark datasets when evaluated with the widely used PSNR measure, whereas the SRResNet with the adversarial component, i. e. SRGANs, was the best in terms of MOS. More recently, SinGAN  [263]  achieved the unconditional generation of synthetic images by using only one training image. This was achieved by adopting a multiscale approach: the pyramid representation. This work demonstrated that a pyramid of fully convolutional GANs could learn the generative model of the complex structures of a single natural image.
Static Systems: For simulation studies or for small accelerators whose properties do not change significantly over time, surrogate models are very useful examples of ML applications in the accelerator community. NN-based surrogate models can be trained to quickly map between accelerator parameters and beam properties, providing faster estimates than possible with computationally expensive physics models. Surrogate models can also be used to generate datasets for ML training and optimization studies  [281] ,  [282] ,  [283] ,  [284] ,  [285] ,  [286] ,  [287]. An effort has also been made toward developing ML-based accelerator controllers using Bayesian and GP approaches for accelerator tuning  [275] ,  [288] ,  [289] ,  [290] ,  [291] ,  [292] , including various applications at the Large Hadron Collider for optics corrections and detecting faulty beam position monitors  [293] ,  [294] ,  [295] ,  [296] , and PC expansion-based surrogate models for UQ  [297].
Time-Varying Systems: An open problem in ML is the development of tools for quickly time-varying systems and systems with distribution shifts. If a system quickly changes with time, it is no longer accurately represented by the data that were used to trail the ML model. Therefore, the accuracy of the ML methods for accelerators will quickly degrade for systems that change with time, for which previously collected training data are no longer accurate. Transfer Learning for Slowly Changing Systems: For systems that change very slowly with time and for which gathering large amounts of new data is feasible without interrupting operations, it is possible to utilize transfer learning techniques in which a network is modified to be accurate for a new dataset by taking advantage of some learned feature extraction capabilities and fine-tuning others for the particular problem of interest  [302].
The most common transfer learning technique is retraining. For a particle accelerator, a retraining approach may start by using large amounts of simulation-based data to train ML models and then "freeze" most of the weights in the layers that have learned the high-level features of the physical systems for which they were trained and then fine-tune only a few layers, such as input layers that must handle real data rather than simulation-based data as inputs, by using much smaller experimental datasets. Another approach to transfer learning is domain transform in which a much smaller NN, such as a U-Net approach, is developed using a small amount of experimental data and is used as the input layer of our trained NN; the U-Net encodes and decodes data to translate between experimental and simulation domains  [303].
These transfer learning techniques are not limited to NNs. For example, they can be applied to GP-based algorithms in which the prior and parameter correlations are first estimated via simulation studies and then fine-tuned with experimental data. Such transfer learning techniques have been demonstrated to be very successful on a wide range of systems with recent applications including cross-modal implementations  [304] , and both retraining and domain transform were recently demonstrated for mapping electron backscatter diffraction patterns to crystal orientations in which simulation-based data were first used, and then, many orders of magnitude fewer experimental datasets were successfully used for transfer learning to make the networks accurate for experimental data  [305]. Adaptive ML for Time-Varying Systems: For most accelerator applications, repetitive retraining is not feasible because detailed beam measurements are time-consuming and invasive procedures that interrupt regular operations.
For example, surrogate models can provide a cheap-toevaluate, continuous, and noise-free abstraction of the complex plasma interaction allowing for an investigation into the underlying parameter dependencies and how they influence the achieved optima. It has been demonstrated that the GP models generated during Bayesian optimization can naturally serve such a purpose  [275]. ANNs are also gaining traction as tools for exploring complex experimental datasets. For example, they have found use in explaining and quantifying the influence of drive laser fluctuations on electron beam quality  [329]. Such knowledge is vital to improving the shot-to-shot stability of these machines. In a similar fashion, several different supervised learning techniques have been applied in a predictive capacity to compare their performance in determining the charge generated in a laser-plasma accelerator as a function of changes to the laser wavefront  [330].
TB datasets are available from dusty plasma experiments through particle tracking and imaging  [356]. Dusty plasma movies have been recorded at about 1500-5000 frame length, at the rates between 100 and 500 frames/s and each image size of a few MB per frame  [357]. For an experimental campaign consisting of a few hundred runs, more than 1. 5 million movie frames or more than 1 TB of raw data becomes available  [358]. Automated particle tracking through ML is emerging as necessary to process a large number of images and to extract the particle trajectories  [356]. Particle tracking and PIV techniques have wider applications than plasma physics. In addition to the traditional probabilistic algorithms, new PTV and PIV algorithms based on U-Net, CNN, and PIML  [359]  are emerging. Other examples of ML applications may be found in the phase transitions in the dust cloud  [360] , the correlation of current-voltage (I -V ) characteristics given by a Langmuir probe with the main plasma parameters  [361] , to identify the boundary layer between mixed regions of dust particles with different diameters  [362] , and the response of a single dust particle levitated in the plasma sheath, to a nonlinear excitation frequency  [363].
A combination of a deep learning architecture and high-dimensional datasets has shown to be more effective than earlier ML methods, such as SVMs, small MLPs, random forests, and gradient-boosted trees  [376]. High-dimensional data came from multiple plasma apparatuses and different experimental conditions from about 9000 experiments. Physics consideration guided the selection of more than a dozen features, including plasma density, plasma temperature, and so on, as the NN inputs. Physics-motivated dimensionless combinations of the raw measurements were used for input data normalization. Reliable predictions with 82% or better accuracy have been demonstrated on another plasma from the one on which the NN was trained. Construction of Grad-Shafranov equilibria is usually the first step in understanding and control of magnetically confined plasmas.
A five-layer fully connected DNN was reported for solving the Grad-Shafranov equation constrained with measured magnetic signals in real time  [377]. The computing time was approximately 1 ms on a personal computer, potentially allowing applications in real-time plasma control. An encoder-decoder NN model of tokamak discharge is developed based on the experimental dataset alone  [165] , without a direct reference to a physics constraint, such as the Grad-Shafranov equation. Electron density, stored energy, and loop voltage were reproduced with close to 90% fidelity to experimental data from a series of actuator signals using the NN. The method provides an alternative to the physical-driven method for plasma modeling, experimental planning, and model validation. Variations of experimental plasma conditions are usually captured by statistical models.
J. Challenges and Outlook Rapid advances in computing hardware, architecture, and data acquisition instruments present challenges and opportunities for plasma physics and science at large. One challenge lies in the fact that manual and even semimanual data mining methods face increasing difficulty in extracting new information and knowledge from large and multidimensional datasets, see Fig. 13  for data reduction and knowledge extraction hierarchy. Data science and ML offer transformative tools for laboratory plasma experiments and the physics of plasmas in the bigdata era. The classical physics framework, which includes Newton's laws and Maxwell's equations, is the canonical pathway to understand plasmas and guide the designs of plasma experiments and inventions of plasma technologies. Many problems in plasmas rise from the complexity derived from a large number of particles (on the order of 1 mole in some laboratory plasmas) and their interactions with EM fields and material surfaces.
The combination of accumulative computational errors, insufficient knowledge of the initial condition, boundary condition and perturbations, and the long computing time even by using state-of-the-art computers renders the canonical pathway ineffective if possible for reliable predictions and optimization problems in plasmas. There are also NP-hard problems in plasma physics, which may be difficult for both ML and traditional computation. Enabled by heterogenous multidimensional datasets, including experimental and observational data, simulation data, and other metadata, data science and ML have been successfully or can be used to accelerate all aspects of plasma research or the "scientific data flows," i. e. , from observational and experimental data taking to hypothesis generation, model construction, modeling, and model validation.
As a consequence, efforts have been undertaken to find physics-informed data-driven techniques that are capable of mitigating some of the limitations of these more empirical approaches. For example, PINNs  [405] , or PDEs solved and enriched using DNNs  [407] , have been recently developed and explored. These models have generally been used to solve traditional initial-boundary value problems in physics-based PDEs (e. g. , multicomponent reactive MHD) but with the added benefits of: 1) significantly improved numerical regularity features; 2) the ability to readily incorporate large datasets into the "training regime"; and 3) the ability to simultaneously solve for solutions over an entire parameter sweep (e. g. , over not only (t, x) but over (t, x, γ )). The major drawback of using these methods for solving numerical PDEs, however, is the slower overall runtime per forward solve that can render them impractical for high-dimensional systems (e.
D. Diagnostics and Fusion Data Streams In fusion energy plasmas, many disparate diagnostic instruments are simultaneously used in order to cover the multiple physics phenomena covering a range of spatiotemporal scales. In addition, fusion experiments, such as ITER, will run longer pulses, with the goal of eventually running a reactor continuously. The confluence of these facts leads to large, complex datasets with phenomena manifest over long sequences. Fusion scientists have a range of data analysis timescales, from real-time processing for plasma control to between-shot quick processing of data to give insight to adjustments for next shots and to longer term deep analysis for science discovery. Diagnostic data analysis has always been fundamental to progress in MCF energy, and many current and emerging applications of ML are aiding scientists in these many tasks in making sense of diagnostic data.
NPE was used to train a normalizing flow model on 10 000 UEDGE simulations, producing an NN that could then take in profiles of electron/ion density and temperature from diagnostics at the midplane and the outer divertor, and infer the corresponding anomalous transport coefficients that are consistent with UEDGE  [423]. Various works are using other methods for speeding up and broadening the analysis that can be done with experimental diagnostic data for physics parameter extraction. For example, a simple feedforward NN was trained to extract electron temperature from a database of measured spectra from an EUV/VUV spectrometer based on the measurements of electron temperature from Thomson scattering diagnostics  [187]. The above works and techniques aim to improve our physical understanding of fusion plasmas by leveraging ML to extract physics from experimental diagnostics.
The DIII-D experimental data system is integrated with a Microsoft SQL Server [458] relational database that records O(100) metadata fields across O(10) tables for each shot, including a schema for recording disruption information. A typical ML application will often gather a preliminary list of shots to process by first querying the relational database. As a simple example, one might be interested in shots from a particular date range, or one might have search criteria related to shot length, shot start time, experimental logbook entries, maximum plasma current, and so on, all of which can be queried using standard SQL. However, it is worth noting that, while the approach taken by DIII-D might effectively utilize one set of tools, there has not been a community-wide effort at standardization, particularly with regard to metadata management, an issue that a dedicated FDP would address.
I. Challenges and Outlook As for other application science areas, there are numerous challenges in utilizing data-driven sciences in MCF research. Besides the challenges and outlook listed in each of the above subareas, an important aspect to keep in mind in discussing the challenges and outlook is that MCF is different from many other scientific projects in that it is an international-scale mission-oriented project. This means that global collaborations among geographically separated large-scale laboratory facilities and between laboratory experiments and high-performance computations are key to success. A vast amount of data produced (and stored) in different format at different experimental facilities and by different simulation codes over decades of time span (see Section IV-G) may require building a community-wide federated database and workflow system [459], which is based on the metadatabase management system and which honors individual institution's and code's data format transparently and maps multiple autonomous database systems into a single federated database via wide area network without the need for centralized data mirroring.
We envision data-driven methods as a cross-cutting third pillar that both improve HEDP experiments and simulations, and sit at the interface between the two. Data-driven methods provide an opportunity to efficiently feature our complex datasets, reliably combine information from simulations and experiments, and accelerate the rate at which simulations and experiments can be performed. As a result, HEDP and ICF problems are quickly becoming an important driver of data-driven methods for science. In the remainder of this section, we will describe some aspects of the research in these areas. [Brian K. Spears].
C. Transfer Learning for Simulation and Experiment Standard computer simulations for indirect drive ICF, without platform-specific corrections, often show discrepancies with experiments. In the ICF community, a new approach to calibrating simulations to experimental data has been shown to create models that can predict the outcome of ICF experiments better than simulations alone. This approach leverages an ML technique called "transfer learning" to merge simulation data and experimental results into a common model. Transfer learning is when an NN trained on a large dataset to solve a given task is partially retrained to solve a different, but related task, for which little data are available. For example, an NN trained on the ImageNet dataset to label random objects (such as cars, trees, and cats) can be modified by retraining just a few layers of the NN to label very specific images, such as the type of aircraft in a photograph, which has a significantly smaller training dataset.
The model predicted the subsequent four experiments with significantly higher accuracy than the LILAC simulations alone; this is shown in Fig. 21 (left). The AE-based transfer learning technique was developed to overcome challenges associated with indirect drive ICF-the expense of integrated hohlraum simulations and the sparsity of indirect drive ICF data. An AE trained on a large database of capsule-only simulations learns to encode ICF outputs (such as yield, temperature, and density) into a latent space and decode back to the outputs. The model is transfer learned with pairs of integrated hohlraum preshot simulation outputs and corresponding experimental measurements for a database of 50 ICF experiments carried out at the NIF. The resulting model produces an accurate mapping from preshot simulation predictions to expected experimental measurements; resulting predictions from this model are shown in Fig.
From a data science perspective, experimental datasets are rarely complete enough to make purely experimental-datadriven approaches feasible. Instead, the usual approach is to use the available data to make point checks of physics models (benchmarking) or to fit a handful of parameters to observations (tuning). The tuned and benchmarked physics model can then be used to make predictions at a new point of interest, with a limited (or no) understanding of the uncertainty in the prediction. Recently, ICF and HEDP researchers have started to formalize the process by treating physics as a second source of information and build data-driven models that are in some way informed by both sources. A variety of approaches have been explored, for example, using simplified physics models  [465] ,  [466] ,  [467] ,  [468] , by using physics considerations to limit the size of the design space  [469] ,  [470] , or by attempting to combine data from disparate but physically related experiments  [471].
The key point is that the generation of training data is a massively parallel operation that can leverage leadership-class high-performance computing facilities and software tools, while including simulations in the inference directly requires the samples to be run serially. With a good choice of surrogate  [475] , a high-fidelity analysis that would be impossible with the simulation itself can be run in a few hours, opening the door for thorough and realistic UQ studies. The process of building surrogate models for large simulations has motivated many of the developments described elsewhere in this article and has made ICF and HEDP datasets  [476]  a key driver of developments in scientific ML. A recent application of the Bayesian approach aimed to interpret results from a series of so-called "BigFoot" ICF implosions at the NIF  [477] ,  [478]  (see Fig.
E. High-Performance Computing and Simulation Acceleration The vast amounts of data generated by these simulations and required for training some of these models can create a substantial demand for scalable training algorithms and leadership-class HPC resources. In developing models for these multimodal datasets, we have created new techniques for composing data-, model-, and ensemble-level parallelisms and working with 100M sample datasets with 1. 5B scalar fields and 1. 2B images  [482]. Using the LTFB algorithm developed by Jacobs et al. [482] ,  [483]  enabled the entirety of a supercomputer, such as Sierra, to be used when training a single model architecture and was able to produce a single instance of a well-converged model. Some of the techniques that have been developed are a coupled, tournament, training algorithm that intertwines the training of a set of model instances to produce a single, best model that has been trained on a sufficient portion of the training data to generalize across a held-out tournament and validation datasets.
In addition, we developed a scalable, in-memory data store, and data ingestion algorithm that is able to fetch a massive, distributed data set efficiently and use only a single pass over the data for the entire training regime. Finally, we have developed methods for both modeland data-parallel training of each individual instance of the NN architecture and optimized it for the IBM Power9 + Nvidia Volta architecture of the Sierra system. Building upon these capabilities enabled us to produce a demonstration on training a generative molecular model on 1. 6B small samples, which was selected as a finalist for the 2020 Gordon Bell Special Prize for COVID-19 research  [484]. These algorithms have been implemented in the Livermore big ANN (LBANN) scalable deep learning toolkit, which is open-source and is being optimized for the next generation of leadership-class computing systems, Fugaku, Frontier, and EL Capitan.
In this example, an automated optimal design was able to navigate a high-dimensional space and settle on a design template not-unlike one that human subject matter experts have learned via decades of study. While automated design in ICF has shown some early success in being able to discover both intuitive and nonintuitive designs, the solutions that they discover are inherently limited by the simulator used. That is, if a model disagrees with an experiment, the optimal model-based design may be of little interest since it may or not reflect reality. In this case, it could be possible to use techniques such as transfer learning  [269] ,  [487]  to postprocess raw simulation data during the optimization process. That is, the optimization cost function evaluates not the output of the simulation, but rather the output of an ML model that adjusts the simulation output to better match what might occur in an experiment.
H. Challenges and Outlook We have described several elements of ongoing research that aim to make data-driven methods the third pillar of HEDP and ICF research, alongside large-scale experimentation and simulation. While each of these elements is ongoing work, the ultimate aim of the HEDP and ICF community is to tightly couple them into a continuous, iterative process of scientific discovery; high-fidelity simulations inform the design of experiments, and the resulting data are used to update physics models and propose new experiments at very high throughput. Many of the components of this vision are already in place, and ICF and HEDP research is pushing the remaining pieces forward. [Brian K. Spears].