dataesr/scientific-paragraphs-categorization huggingface dataset:
2001-2021 humpback whale dataset with sample size and characteristics presented by region andyear. Encounters: a count of the total number of photo-ID documented encounters of individuals, defined as one encounter per individual whale per day. Individuals: count of unique identified individuals. Research sourced: percent subset of encounters sourced from NPPID collaborators, while all remaining encounters were sourced from community science efforts. Public: percent subset of encounters visible to all users of Happywhale, while all remaining encounters are visible only to NPPID collaborators. Multiple seasons: a count of identified individuals documented in more than one season during the study, overall total for all years in final column. Seasons: north of 32. 55°N = feeding regions (blue) data by calendar year; south of 32.
Data availability The publicly viewable 66% of the full dataset used in this study, with ongoing additions and updates is available for exploration at www. Happy whale. com. All data are available with collaborator agreement to explore at Happywhale and in spreadsheet format. Please contact the corresponding author for discussion and permission. Approximately one-third of the dataset is public domain, but the collaborators believe that providing this partial dataset for open access download would be a disservice to the integrity of the full dataset.
Efficiency with large datasets: To increase efficiency for collaborators with large, well-curated datasets, some encounters were accepted with an individual ID name/number and supporting date, location, and attribute data, without a photo-ID image linked to every encounter. These encounters were linked to known individuals represented in one or more catalog photos. strike, killer whales) were recorded with each encounter when available and without standardization. Data quality was reviewed on import, with an opportunity for review by both data managers and data contributors before entry into a relational database. 3. 2. Supporting attribute data curation: Given the diversity of supporting data formats received, standardiza- tion was necessary for dataset management. Locations were categorized as general (confident of location within 200 km [km]), approximate (confident of location within 20 km), or precise (confident of location within 2 km).
Recent years have seen a surge in deep learning approaches to accelerate numerical solvers, which provide faithful but computationally intensive simulations of the physical world. These deep surrogates are generally trained in a supervised manner from limited amounts of data slowly generated by the same solver they intend to accelerate. We propose an open-source framework that enables the online training of these models from a large ensemble run of simulations. It leverages multiple levels of parallelism to generate rich datasets. The framework avoids I/O bottlenecks and storage issues by directly streaming the generated data. A training reservoir mitigates the inherent bias of streaming while maximizing GPU throughput. Experiment on training a fully connected network as a surrogate for the heat equation shows the proposed approach enables training on 8TB of data in 2 hours with an accuracy improved by 47% and a batch throughput multiplied by 13 compared to a traditional offline procedure.
Besides, being developed with deep learning frameworks that support automatic differentiation, these surrogates directly provide different gradients, including the adjoint, valuable for many applications  [63]. Generally, the training of these deep surrogates is supervised. It requires the generation of a dataset of simulations. The good generalization capabilities of the surrogate model depend on both the neural architecture and the training dataset  [19, 41, 70]. To create a dataset representative of all the richness the process 𝑓 exhibits when 𝑋 varies requires executing the solver many times (i. e. ensemble runs). The training dataset can quickly grow in size to prohibitive extents, making it difficult to store on disks and incurring substantial I/O operations, which hinders the training speed.
Additionally, training cannot adapt dynamically the data generation process, the latter being performed a priori. This paper focuses on the combined data generation and training of deep surrogate models. Classical training procedures rely on a fixed dataset that is stored on disk and read to extract batches. However, because deep surrogates are trained with synthetic data produced by simulation code, the training can be performed online, simultaneously with the data generation. The potential benefits are: • Storage avoiding. The data are never stored on disk, saving storage space. On supercomputers, storage is often limited, while the increasing size of training datasets exacerbates the demand for storage space and i-nodes. • I/O bypass. Directly transferring the data from the simulation to the neural network bypasses storage and circumvents the I/O bottleneck.
I/O slows down both the data generation when writing data to disk, as well as training when reading back data from disk. This is a well-known issue in HPC that led to the in situ data processing paradigm  [12]. • Training diversity. Because the data are produced by a simulation code, the deep surrogate can potentially be trained from an unlimited dataset, exposing the surrogate to more diverse data than with a fixed dataset repeated over several training epochs. At an equivalent batch count, this higher data diversity can enable online training to converge faster and reach better generalization capabilities compared to offline epoch-based training. The contribution of the present paper is an online and large scale training framework for deep surrogate models called Melissa  [71]. It combines: (1) A multi-level parallelism (parallel solver execution, distributed data parallel training, ensemble run execution), in transit data processing bypassing storage, fault-tolerance for resilience, heterogeneous architecture support, and elasticity for adaptive executions, all features required to ensure efficient training at scale.
(2) A training reservoir. A buffer that mitigates the inherent bias in the streamed data caused by the solver internal logic and the availability of computational resources. The reservoir optimizes the throughput of data presented to the GPUs for training while maximizing their diversity. We show through experiments that the resulting framework is capable of leveraging multiple levels of parallelism to efficiently train deep surrogates in an online context. It generates and trains a neural network on 8TB of simulation data in less than 2 hours using 5,000 cores and 4 GPUs. Such a dataset, which could hardly be handled on a cluster in a traditional offline training setting, would require more than 24 hours to be processed by the same number of GPUs. Compared to offline training with multiple epochs on a subset of 100GB, online training increases by 47% the generalization capability of the trained surrogate at an equivalent number of batches.
The framework has not yet been experimented with model parallelism  [35, 85]. Model parallelism is often combined with data parallelism when the NN cannot fit into the memory of one GPU. This would require splitting the NN and the batches to enable pipeline parallelism for instance as well as making parallel all-reduces per group of GPUs managing the same sub-part of the NN  [72]. In the context of our framework, changes are expected to be mainly limited to the code of the training thread. Today, DL frameworks like Pytorch or TensorFlow, on which we rely, provide extensions to ease the deployment of multi-GPU model parallelism. Each server process runs two threads. The data aggregator thread manages connections to clients, receives data and stores these data into the training buffer. The second thread, the training thread, reads data from the training buffer to build a batch, feeds the GPU with it and performs the forward and backward passes through the NN.
A first call is required to connect the client to the server (init_communication). A send is issued to transfer time steps 𝑢 𝑡 𝑋 as soon as computed. Eventually, a client calls finialize_communication to signal the server that no more data will be sent before disconnecting. We also provide a PDI plugin to interface with PDI instrumented simulation codes  [68]. The launcher and server are developed in Python. Transport layer relies on ZMQ  [32]. We are considering adding ADIOS2  [29]  for gaining on data handling flexibility and better use high performance networks. Regarding training, the framework supports the PyTorch and Tensorflow libraries. The training thread embeds a classical training loop where the main difference is the data source that relies on the training buffer instead of files.
Data management 3. 2. 1 Data diversity. In classical offline training, the gradient descent expects batches built by uniformly sampling the fixed dataset, which can easily be done as the full dataset is available upfront  [16]. The training process of our framework being online, it leads to inherent sources of data bias. Bias in the data is known to be detrimental to the quality of training. For instance, catastrophic forgetting characterizes network performance decrease on previously seen data when the training data distribution changes  [39]. The sources of workflow bias caused by online training are of three different categories: • Inter-simulation: The computational resources are finite. At any time, only c different clients can run simultaneously. The data produced by the c simulations may not present all the diversity the process 𝑓 exhibits.
Training buffer. The training buffer is a fixed-size data container that has the dual role of accumulating a certain number of time steps to ensure batches contain well-mixed data to reduce workflow bias, and amortizing (up to a certain limit) discrepancies between data production and consumption to reduce resource idleness. The Reservoir algorithm we propose for managing this buffer is a key contribution of this paper. For comparison purposes, we also present two other strategies, FIFO and FIRO. Prior work only considered FIRO implementation  [53] , which is shown here to fail in optimizing GPU usage. The classical FIFO buffer (First In, First Out) corresponds to the streaming case, where data are batched for training according to the order they are received. Each data produced is seen once, and only once, during training.
Batch extraction is enabled as soon as the buffer can provide one. Compared to pure streaming, buffering provides some slack to keep the consumer, the learner thread, busy as long as batches are available in the buffer, even if data production is stopped or reduced for a while. Data production is suspended when the FIFO buffer is full. The FIFO buffer is thus simple to implement as a queue but does not enable mixing data beyond what the server receives. The FIRO buffer (First In, Random Out) behaves very similarly to FIFO, with data evicted upon reading, except that these data are extracted from random positions to build less biased batches. Additionally, batches can only be extracted if the buffer is filled beyond a given threshold, again to reduce batch bias. The threshold is set to zero once data production is over to enable consuming the last produced data.
FIRO is implemented as a list container. Newly received samples are appended at its end. The Reservoir buffer (Algorithm 1) enables data to be seen more than once to reduce consumer idleness in case of underproduction while giving priority to storing newly produced data over already seen ones. The Reservoir distinguishes the new unseen data from the ones already selected in a previous batch. When receiving new data while the buffer is full, a seen example is randomly evicted to make room for the incoming one. When building a batch, the elements are uniformly selected one by one among the seen and unseen elements in the buffer. Each selected unseen data is then moved with the seen ones. This Reservoir ensures that batches are well mixed; that data production can proceed as long as the buffer is not full of unseen samples, avoiding discarding any unseen data; and that data consumption is never locked, once the threshold is passed, as new batches can be built from already seen data.
Equation and deep surrogate architectures The experiments consider the classical heat equation on a 2D rectangular domain (Equation  2 ):            𝜕𝑇 𝜕𝑡 = 𝛼∇ 2 𝑇 , 𝑇 (𝑥, 𝑦, 0) = 𝑇 IC , 𝑇 (0, 𝑦, 𝑡) = 𝑇 𝑥 1 , 𝑇 (𝐿, 𝑦, 𝑡) = 𝑇 𝑥 2 , 𝑇 (𝑥, 0, 𝑡) = 𝑇 𝑦 1 , 𝑇 (𝑥, 𝐿, 𝑡) = 𝑇 𝑦 2 , (2) where 𝑇 (𝑥, 𝑦, 𝑡) denotes the field temperature, 𝛼 the thermal diffusivity and (𝑇 IC ,𝑇 𝑥 1 ,𝑇 𝑦 1 ,𝑇 𝑥 2 ,𝑇 𝑦 2 ) the initial and 4 boundary temperatures. The solution is approximated with an in-house solver that implements a finite difference method with an implicit Euler scheme. The temperature field 𝑇 (𝑥, 𝑦, 𝑡) is discretized on a 1000×1000 Cartesian grid, and generated for 100 time steps representing Δ𝑡 = 0. 01 second each. The thermal diffusivity is fixed to 𝛼 = 1 m 2. s -1. The solver code is written in Fortran90 and parallelized with MPI according to classical 2D domain partitioning.
Throughput The experiment illustrates how the different buffer implementations impact the throughput during training on 1 GPU. It considers a dataset of 250 runs of the heat equation solver, accounting for a total of 25,000 time steps. These data are generated by the framework running concurrently 100 clients on 50 nodes. Each instance of the solver executed by the clients is parallelized over 20 cores. Due to the limited support for heterogeneous jobs (CPU/GPU) on the machine, the framework adopts the submission process described before in Section 3. 1. It submits three series of clients. First, it launches 100 simulations. Once the clients have terminated their executions, it launches the second series of 100 and finally the third series of the 50 remaining simulations. The throughput assesses the capacity of the framework to quickly provide training data to the GPU.
The throughput is expressed as the number of samples per second processed by the GPU, computed on the learning thread over 10 successive batches every 10 batches. The batch size is fixed to 10 samples, with one sample being the time step 𝑢 𝑡 𝑋 of one simulation associated with its 6 input parameters (𝑋, 𝑡). For this experiment only, as we are solely considering throughput, the performance of the trained network is never evaluated on a validation dataset. Performing the evaluation on the training thread would stall the data consumption of training batches, thus reducing throughput. Evaluation can either be excluded from the throughput measurement (as in the next experiments), or performed by using a dedicated GPU not involved in training. The three training buffers described in Section 3. 2.
3, FIFO, FIRO and Reservoir, are evaluated. FIFO yields samples as they are received from the clients. FIRO and Reservoir have a fixed capacity of 6,000 samples (roughly a fourth of the whole simulated data) with a threshold set to 1,000 samples. These parameters remain the same in all the experiments. Figure  2  shows the evolution of the throughput with respect to time. The Reservoir provides the highest throughput of the three implementations. It appears that both FIFO and FIRO are sensitive to the flow of incoming data. Their throughput drops briefly at 100 and again at 200 seconds. After 200 seconds, their throughput is half what it was before. The times these drops occur coincide with the transitions between series of client submissions performed by the launcher. A bit before 100 seconds, all the running clients have terminated generating the data of the first 100 simulations.
There is a delay before the clients start running the next 100 simulations and streaming the data they generate to the server. Similarly, there is a delay between the execution of the second series of 100 simulations and the execution of the last series of 50 ones. The amount of generated data is also halved during this last series running 50 instead of 100 concurrent simulations. Because FIFO yields data as they arrive these changes in data production result in drops in the throughput. The FIRO shows similar patterns, but it occurs sooner as the data consumption is stopped when reaching the threshold value and not when emptying the buffer as for FIFO. Figure  2  bottom shows the FIFO and FIRO buffer populations stay around their minimum, respectively 0 and 1,000 set by the threshold value.
Because data are consumed faster than produced their throughput is virtually the one of the data generation by the clients. At the end of the experiment the FIRO throughput rockets up. At this time, the data production by the clients has terminated and the blocking threshold has been released. Samples can thus be freely drawn to form batches. The FIRO population, which is necessarily higher than the fixed threshold of 1,000 samples, can quickly yield 100 batches. Because the Reservoir evicts on writing rather than on reading like FIRO, its population increases quickly to its maximum, while FIRO's population remains around the threshold value, even when data consumption equals or overruns production as in this case. We can expect this enables Reservoir batches to present a higher diversity as they are sampled from a larger population.
The Reservoir throughput remains constantly higher than the ones of FIFO and FIRO. The Reservoir manages to provide a high throughput by repeating samples, which erases the gaps between production and consumption rates. The visible high frequency fluctuations are not yet understood. Figure  3  presents a histogram of the number of times samples have been repeated for the Reservoir. Most of the samples have been seen in batches a couple of times, and at most, though rarely, 8 times during training. Thus, a few repetitions are enough to increase the throughput by 50%. By providing higher throughput, the Reservoir maintains the GPU active, whereas for FIFO and implementations it can be idle, waiting for data to come. To be useful, this extra activity in training on more batches with already seen samples must result in increased generalization capabilities, which should appear as a lower validation loss.
Training quality During training, the learning rate, initially set to 1E -3 , is halved every 1000 batches. FIFO presents a low training loss associated with a high validation loss, which indicates overfitting. Both losses experience bursts, occurring when the learning rate is halved, which is symptomatic of unstable training. Although to a lesser extent, FIRO presents the same problems. On the contrary, Reservoir shows a more stable training not subject to overfitting. It achieves a validation loss that is on par with offline training. First, the performance of the FIFO simple data streaming confirms that online data generation indeed leads to a strong data bias that affects the training quality. Second, buffering with random data reads as performed by FIRO and Reservoir is effective in mitigating this bias.
FIFO and FIRO fail to provide higher throughput when the number of GPUs increases. The population size displayed in Figure  2  was already showing FIFO and FIRO were unable to balance the production of data with the higher consumption. Increasing the number of GPUs only worsens this trend, because it increases the consumption and dilutes the generated data between the buffer replicas. Only the Reservoir scales with the number of GPUs at equal data production. It is important to remember however that because its higher throughput is due to data repetitions, this leads to more batches and thus training can eventually be slightly longer. Table  1  summarizes the results obtained for the different implementations and different numbers of GPUs. As in the previous experiment, the same number of unique samples is used for each training.
For Reservoir, increasing the number of GPUs also increases the global buffer size. But as the data production remains the same, the more GPUs, the less new data each local buffer receives. Besides, batches being larger with more GPUs, more data must be extracted at each training step. Thus, increasing the number of GPUs induces more sample repetition (as seen in Figure  3 ), which results in more training batches. There are 6 times more batches generated by Reservoir with 4 GPUs compared to 1. These additional batches trigger more optimization steps. The greater number of optimization steps that characterize Reservoir can explain why it beats the one-epoch offline training benchmark, not unlike how training for multiple epochs would improve the offline setting. However, Reservoir repeats samples at a much faster pace than would an offline multi-epoch training (Table  1 ).
Reservoir consistently outperforms all the other training settings for the same number of GPUs. At the end of each training, the validation loss of Reservoir is significantly lower than the one of FIRO, often more than halved. The Reservoir capability to keep its buffer full with a self-adjusted amount of new and alreadyseen data makes it more amenable to take full benefit of multi-GPU training compared to FIRO. It provides higher throughput and better generalization materialized in a lower validation loss.
Online versus multi-epoch Offline We have seen that properly managed online training, with Reservoir being the best option, is already competitive with offline training on a single epoch. But the true potential of online training comes when enabling working with datasets so large that they cannot be reasonably handled in an offline fashion, due to storage and I/O costs. Offline needs to restrain to a reasonable dataset presented several times through different epochs, while online can lean towards training on a potentially unlimited dataset. In the following, we compare offline and online in such a context. Offline training is performed on the same 250 simulations data as used for previous experiments, except that here we perform 100 epochs (Figure  6 ). The dataset is 450 GB raw with one file per time step, 95.
5 GB when compressed using one binary file per simulation. The data generation is also performed in parallel with the framework using 2000 cores, but instead of streaming the data through the API, they are simply written on disk. Here, the framework reveals itself also useful to quickly generate datasets by leveraging the parallelism of its clients. During training, the data are loaded from SSD disk. The loading relies on mmap to read only the requested time sep without having to load the entire file in memory. Online training is performed with 20,000 simulations managed by the framework. The simulation clients run on 5,120 cores and generate a total of 8TB that are forwarded as soon as produced to the training server running with 4 GPUs. Figure  6  compares the training and validation losses for the two settings.
16€ when repeated (no storage and data generation costs). If offline training would have been performed with 8TB dataset of online training, the sole storage cost would account for 480€. A realistic production workflow will likely combine pre-training (with the necessary repetitions to tune hyperparameters) from a static reduced dataset and few online re-training at scale with complementary data so as to reach the best possible generalization capabilities. This will enable to control the trade-off between storage footprint and the computing cost of re-running simulations for each new training. The present work does not include the use of the surrogate. Thus, the global gain, counting the cost of training and use of the surrogate, versus using only the original solver cannot be evaluated. But higher generalization capabilities mean a surrogate capable of giving higher quality results, likely leading to reduce surrogate runs as well as some of the simulation runs that are often required to bring higher quality data into the process.