dataesr/scientific-paragraphs-categorization huggingface dataset:
As we utilise a variant of stochastic gradient descent for optimisation, the data pairs are drawn from the training dataset D at each iteration. Before summing all nine loss terms up, we average the negative log-likelihood per-variable across all grid points (here simplified denoted as average across M grid points) as the velocity components have fewer data points than all other variables, caused by their spatial discretisation (c. f. Sect. 2), L total = 1 BM 9 i=1 B j=1 M k=1 1 b i |∆x i,j,k -f i,j,k (x, ϕ)| + ln(2b i ). The factor in front of the absolute error, λ i = 1 bi , is the weighting factor; the MAE can be recovered by setting b i = 1 as constant. The additional term, ln(2b i ), origins from the normalisation of the Laplace distribution, is independent of the errors, and counteracts a too small b i.
This approach optimises b i to match the expected MAE  (Norton, 1984)  in the training dataset and can be seen as an uncertainty estimate, e. g. recently used in  Cipolla et al. (2018) ;  Rybkin et al. (2020). Compared to using a fixed climatological value, this approach adaptively weights the loss, depending on the error of the NN for the different variables. This adaptive weighting marginally improves the training of the NN, as shown in Sect. C. Since we learn the scale parameters purely from data, this approach can be seen as type II maximum likelihood or empirical Bayes approach  (Murphy, 2012)  4 Experimental setup We train and test different NNs with twin experiments, using the sea-ice model described in Sect. 2. We compute trajectory simulations with a resolution of 4 km and an integration step of 8 s, which stand as our truth for the twin experiments.
Based on a coarsened 8 km version of the truth grid and an integration step of 16 s, we simulate low-resolution forecasts. The NNs will learn to correct these low-resolution forecasts (hereafter simply called "forecasts") for a lead time of 10 min and 8 s. In other words, the training target is the difference between the projected truth and the forecasts after an integration of 10 min and 8 s, which equals 38 time steps at the low resolution. We train the NN on an ensemble of 100 truth trajectories. The NN hyperparameters, like the depth of the network or the number of channels, are tuned against a distinct validation dataset with 20 trajectories. Finally, the scores are estimated using an independent test dataset with 50 trajectories. All truth trajectories are initialised with a randomly chosen cohesion field; as the cohesion is one of the state variables, the cohesion is advected and diffused.
Description Value Cohesion C U (5 × 10 3 Pa, 1 × 10 4 Pa) Amplitude A U (8 m s -1 , 20 m s -1 ) Wave length λ U (50 km, 200 km) Phase ϕ U (-100 km, 100 km) Advection ν U (-0. 5 m s -1 , 0. 5 m s -1 ) Base velocity u0 max(20 m s -1 -A, U (0 m s -1 , 10 m s -1 )) The truth trajectory simulations are run for three days of simulation time. The first day of simulation, where the forcing is linearly increased to its full strength after the day of simulation as in  Dansereau et al. (2016) , is treated as spin-up and omitted from the evaluation. Consequently, we use two days of truth trajectories to generate the dataset; the process of generating the dataset is depicted in Fig. 4. To make the truth comparable to the forecast simulations, the truth is projected to the 8 km resolution grid using a Lagrange interpolation, as in finite elements' discretisation.
We generate initial conditions for the forecast trajectories by hourly slicing the projected truth. For these forecasts, the cohesion is one of the projected state variables. Since our only difference in the experiments should come from differently integrated processes, we keep the forcing parameters for the forecasts the same as for the truth. We run the forecasts for one hour of simulation time. After 10 min and 8 s, we compute the residual (the difference between the projected truth and the forecasted state) and include it into our training dataset. This dataset contains input-residual pairs (x j , ∆x j ). The input for the NNs consists of 20 fields: nine fields for the initial conditions, nine fields for the forecasted state, one forcing velocity in y-direction at the initial time, and one forcing velocity  with an initial learning of γ = 3 × 10 -4 , β 1 = 0.
9, and β 2 = 0. 999, is used to optimise the NNs. We refrain of learning rate decay or early stopping, as such methods would make the experiments harder to compare. All experiments are performed on the CNRS/IDRIS (French National Centre for Scientific Research) Jean Zay supercomputer, using a single NVIDIA Tesla V100 GPU per experiment. The NNs are implemented in PyTorch  (Paszke et al. , 2019)  with PyTorch lightning  (Falcon et al. , 2020)  and Hydra  (Yadan, 2019). The code is publicly available under https: //github. com/cerea-daml/hybrid_nn_meb_model.
Performance on the test dataset In a first step, we evaluate the performance of our model error correction on the test dataset, without cycling the correction together with the geophysical model, Table  3. A comparison with other NN architectures, other loss functions, and other activation functions can be found in Appendix B, Appendix C, and Appendix D, respectively. The NN corrects the model forecasts across all variables, which results in an averaged gain of the hybrid model over 75 % compared to the geophysical model. For the stress, damage, and area, the persistence forecast performs better than the geophysical model for the lead time of 10 min and 8 s, as the model forecast drifts towards the attractor of the geophysical model, as discussed in Sect. 5. 3. Since the NN uses the initial conditions as input, the hybrid model surpasses the performance of persistence, even for variables where persistence is better than the geophysical model.
In total, the NN consistently improves the forecast on the test dataset. To apply CNNs to the raw data of our finite-elements-based sea-ice model, we project from triangular to Cartesian space, where the features are extracted. The number of elements in the Cartesian space determines its effective resolution and, thus, the finest scale on which the NN can extract features. To demonstrate the effect of different resolutions on the result, we perform three different experiments, where we change the grid size, while keeping the NN architecture the same (Table  4 ). The training loss, here the negative Laplace log-likelihood, measures how well a NN can be fitted towards the training dataset. Although its resolution is higher than the original resolution of 8 km, the back-projection for the 8×32 grid is underdetermined, as the mapping is non-surjective, degrading the performance of the NN.
By permuting an input variable, its information is destroyed, and the output of the NN is changed. The possible change in the prediction error compared to the original, unchanged, output describes the importance of this input variable. Focussing on active regions, we measure the errors with the RMSE, estimated over the whole test dataset. The higher the RMSE for a permuted input variable, the higher the importance for this variable onto the errors, as summarised in Table  6. Table  6. The permutation feature importance of the RMSE for the given output variable with respect to the input variable for "Initial + Difference" as input, estimated over the whole test dataset. The numbers show the multiplicatively RMSE increase of a specific output variable (y-axis) if a given input variable (x-axis) is permuted, a higher number corresponds to a higher feature importance.
By modulating the momentum equation and mechanical parameters, respectively, the area and thickness influence the velocity and stress components. In total, the area and thickness are the most influential and most influenced variables in the whole dataset, as they represent the conditions of the sea ice. Therefore, the NN has learned meaningful relations between input and residuals. As local measure, we move to the sensitivity ∂f (x,θ) ∂x of the NN output to its input fields  (Simonyan et al. , 2013) , again for the "Initial + Difference" experiment. To showcase what the NN has learned in spatial meanings, we concentrate here on a single grid point in a single outputted sea-ice area field. To smooth the sensitivity and reduce its noise, we perturb the input to the NN 128 times with noise drawn from N (0, 0.
We additionally enforce physical bounds on all variables, by limiting the values to physical reasonable bounds after error correction. Related to optimal control theory in dynamical systems, we change the performance metric to be the RMSE. We evaluate the performance across all 2400 hourly time slices on the test dataset (Fig. 7  and Table  7 ). In the hybrid models, the forecast is corrected every 10 min and 8 s, and the performance is averaged over all ten random seeds Overall, the hybrid models surpass the performance of the original geophysical model (Fig. 7 ). The projected initial state lays 395 not on the attractor of the forecast model, the forecast drifts towards the model attractor; a behaviour similar to what is typical in seasonal or decadal climate predictions initialised with observed data.
Additionally, for the velocity, stress, and damage, the drift towards the attractor makes the training dataset less representative for the model forecast. As a consequence, the "Initial + Difference" experiment performs worse than the "Initial + Forecast" experiment in these variables, and averaged over all nine model variables. Although the forecast error generally increases with lead time, the error reduction gets smaller with each update, especially for the sea-ice area. Since the NN correction is imperfect, the error during the next forecast cycle is an interplay between the initial condition error and the model error. The NN is trained with perfect initial conditions to correct the model error only. As the initial condition error increases with each update, the network corrects less and less forecast errors.
Additionally, this argument cannot explain the increased training loss for the finer grid compared to its lowered errors on the test dataset. The gain likely results from an inductive bias in the NN for Cartesian spaces with higher resolutions. We keep the NN architecture and its hyperparameters, like the size of the convolutional kernels, the same, independent of the resolution in the Cartesian space. Consequently, viewed from the original triangular space, the receptive field of the NN is reduced by increasing the resolution. The function space representable by such NN is more restricted, and, as the fitting power is reduced, Cycling the NN correction together with the forecast model improves the forecasts up to one hour. Since the error correction is imperfect, the initial condition errors accumulate for longer forecast horizons.
There, the targetted residuals in the training dataset are less representative, and also the correlation between the predicted error patterns and the true error patterns reduces. These issues would be solved in online training of the NNs  (Rasp, 2020; Farchi et al. , 2021a) , which could be nevertheless too costly for real-world applications. Offline reinforcement learning additionally tackles similar issues  (Levine et al. , 2020; Lee et al. , 2021; Prudencio et al. , 2022)  and can be thus a way to partially solve them. Although the here-learned NNs can make continuous corrections, they represent only deterministic processes. As the evolution of sea ice propagates from the subgrid-scale to larger scales, unresolved processes can appear like stochastic noise from the resolved point of view.
This way, the neural network has access to the model dynamics and can correct them. Consequently, the dynamics of the forecast model variables are the most important predictors in a forecast error correction for sea-ice dynamics. -The deterministic hybrid model leads to a better damage and total deformation representation than the forecast model without parametrisation. Nevertheless, the unresolved subgrid-scale in the sea-ice dynamics can have seemingly stochastic effects on the resolved scales. To properly parametrise such effects, we would need generative neural networks. The simplest approach to correct the model forecast is to estimate a global bias, one for each variable, in the training dataset and to correct the forecast by this constant. As the low-resolution forecast has almost no systematic bias compared to the projected truth, this bias correction has only a negligible effect on the results.
https://doi. org/10. 5194/egusphere-2022-1342 Preprint. Discussion started: 2 January 2023 c Author(s) 2023. CC BY 4. 0 License. Code and data availability. The authors will provide access to the data and weights of the neural networks upon request. The source code for the experiments and the neural networks is publicly available under https://github. com/cerea-daml/hybrid_nn_meb_model. The regional sea-ice model source code will be made available upon request.
at a lead time of 10 min and 8 s. Our training target is the difference between the projected truth and the forecasted state at the same forecast lead time, and consists of nine fields. All the data is normalised by a global per-variable mean and standard deviation, both estimated from the training dataset. The hourly slicing gives us 48 samples per truth trajectory. Consequently, in total, the training, validation, and test dataset have 4800, 960, and 2400 samples, respectively. If not otherwise specified, all NNs are trained for 1000 epochs with a batch size of 64 to minimise the Laplace negative log-likelihood (cf. Sect. 3. 4) in the training data. Adam (Kingma and Ba, 2017).
3. 3), simply called U-NeXt. We have selected the parameters of the U-NeXt architecture after a randomised hyperparameter screening in the validation dataset with 200 different network configurations. We evaluate our trained NNs on the test dataset with the mean absolute error (MAE). To get comparable performances across the nine model variables, we normalise the errors by their expectation from the globally-averaged climatology in the training dataset. Note that this normalisation results into a constant weighting, differing from the adaptive weighting used during the training process, which depends on the training trajectory. In the case of the MAE, the errors are normalised by their expected MAE in the training dataset. Furthermore, this normalisation allows us to estimate the performance of the NNs with a single metric, averaged over all model variables.
The NNs are trained https://doi. org/10. 5194/egusphere-2022-1342 Preprint. Discussion started: 2 January 2023 c Author(s) 2023. CC BY 4. 0 License. ten times with different random seeds (s ∈ [0, 9]), the results in Table 3-8 correspond to the averaged metric over the ten trained networks. In the following, we discuss the results on the test dataset in Sect. 5. 1, what we can learn about the model errors by analysing the sensitivity of the NN to its inputs in Sect. 5. 2, and how we can combine the NN with the geophysical model for lead times up to one hour in Sect. 5. 3.
the training loss increases again. The NN is biased towards more localised features. These localised features help the network https://doi. org/10. 5194/egusphere-2022-1342 Preprint. Discussion started: 2 January 2023 c Author(s) 2023. CC BY 4. 0 License. to represent previously unresolved processes and the underlying partial differential equation of the residuals better. This better representation improves the generalisation of the NN, resulting into lower errors on the test dataset. However, as this study is performed in very specific settings with twin experiments, it remains unknown to us if the projection into a space that has a higher resolution is advantageous for subgrid-scale parametrisations with machine learning in general. The permutation feature importance as global feature importance and sensitivity maps as local importance help us to explain the learned NN by physical reasoning.
General assessment The data and the risk assessment for 13 lakes created from former coal and lignite mines in Europe (Czech Republic, Poland, Romania, France, Fig. 6 ) are analysed and discussed. The quality of the data in each of them with respect to the feared events and thus the mining hazards were then assessed. The following data were analysed: geometric and morphological data, geological data, hydrological data, climatic data, historical operational data, ooding data, chemical quality of the water, geotechnical data, instabilities data. Table  5  summarises the collected data for the 13 lakes. For a qualitative risk assessment, only one case (Most lake) presents a good quality and complete data, 7 cases present a medium quality data, 3 have a weak quality and 2 without any data. The surface of the lakes varies from 8 ha to 495 ha, and the depth is generally unknown.
lake case study Thanks to the quality of the data of Most lake, qualitative and quantitative risk assessment analysis were carried out. Table  6  presents the mining and the natural hazards identi ed after the lake's creation and lling. One noticed that the lake itself and the adjusted region are exposed to 11 mining hazards and 10 natural hazards. A rst qualitative assessment was carried out, based on the collected information and the previous. 2 natural hazards present a high level, 10 hazards present moderate levels (mining hazards) and 8 present low hazards (natural hazards). The erosion, re and pollution have a severe impact on the environment, mitigation measurements are taken to improve the safety and the security. Additionally, potential interaction between mining hazards ( ooding, slope stability, self-combustion) and natural hazards (forest re, ooding and slope stability).
The weights are normalized in order equally process the hazards categories irrespective of their number. Thus, 10000 different values for R were simulated and presented in the following histograms and cumulative histograms to provide the hazards level R and relative uncertainty bounds. The described probabilistic approach was designed and implemented in Matlab® programming environment. Figure  7  presents the mine hazards risk distribution, and Fig. 8  the natural hazards risk distribution. The mine hazards risk is distributed after the simulations closer to the 'Medium' hazard level as well as the natural hazards risk. Although most of the natural hazards belong to the 'low' level, the corresponding risk level distribution has a clear trend towards the 'Medium' level. The risk distribution quanti es the risk level occurrence considering the hazards impact and probability of occurrence.
, 2018 (Baltrušis et al. , , 2020a (Baltrušis et al. , , 2020b)). In this method, the sample DNA (or RNA) in each well is partitioned into 20,000 nanodroplets, wherein a PCR reaction takes place in each droplet. Then, the droplet reader assesses all droplets (i. e. , positive, or negative for the specific target of interest). By the Quantasoft software, results are automatically analysed by Poisson statistics; the number of positive and negative droplets are displayed, and the fractional abundance of the detected target(s) is presented. The high number of reactions in each sample/well (i. e. , in each of the 20,000 droplets) provides a high sensitivity. Moreover, it has certain advantages over other PCR based methods, as no standards or references are required. The ddPCR has been shown to be particularly useful for sensitive detection of rare mutant alleles as it reduces background DNA competition  (Hindson et al.
Selection of flocks Sheep flocks were recruited from the Norwegian sheep recording system, with selection of flocks using data previously collected using a questionnaire  (Gravdal et al. , 2021). The dataset consisted of responses from 1378 sheep farmers (10% of Norwegian flocks) and included information from all counties in Norway. First, farmers were stratified by geographical location (i. e. , western, eastern, central, and northern regions), and then a random selection from each area was made in Stata SE/16. 0 (Stata Statistics/Data Analysis: Release 16. College Station, TX: StataCorp LLC). In addition, some flocks were recruited through communication with farmers (e. g. , project presentations/meetings). The number of flocks selected from each area was based on the general flock population  (Statistics, 2021) , with the proportion of flocks included per region ranging from <1 to 2.
8 μl of nuclease free water per well. RNase free water was used as a non-template control, while DNA from mutant (TAC) and susceptible (TTC) H. contortus were used as positive controls for each of the respective targets. Droplet formation and thermocycling were then performed as per manufacturer's instructions (BioRad). After amplification, the PCR plate was placed into the droplet reader (QX200) (BioRad) and analysed using Quantasoft™ software (BioRad). Oversaturated samples were diluted by diluting 20 times in RNase free water and the sample was analysed again.
Statistical analysis After ddPCR, all results were analysed for the presence of positive droplets using Poisson statistics with the Quantasoft software (BioRad). Thresholds were manually adjusted according to the baselines of the negative and positive controls (FAM: 4000, HEX: 3000). Samples that had >2. 5 copies per μl were classified as positive for the respective targets, in agreement with the limit of detection (LoD) of the ddPCR assay  (Baltrušis et al. , 2018). In detail; for determining the presence of Haemonchus in lambs and ewes, the respective age group were classified as positive if one or both pooled samples were positive (i. e. , > 2. 5 copies per μl). Furthermore, flocks were classified as Haemonchus positive if Haemonchus was detected in either one or both age groups. Similarly, samples from lambs and ewes were classified as positive for the respective target (i.
Discussion In summary, we used two ways of estimating the amount of energy needed to learn a simple association from behavioural data, namely from excess sucrose consumption and from change in lifespan. Encouragingly, the estimates yield comparable numbers on the order of 100mJ, or some 10mJ/bit. It is interesting to compare these costs to memory costs in digital computers. Both data storage and data transmission from CPU to memory cost substantial amounts of energy. In typical personal computers the slowest, most persistent, and most energy costly storage is farthest removed from the processor  (Das et al. , 2015). For instance, a typical modern Solid State Drive (SSD) can write up to 3GByte/s, taking about 10W (Samsung 970). Hence the energy cost of storage on an SSD is about 0. 5 nJ/bit.
If learning two associations would costs double the energy, synaptic processes are likely the main consumer. In that case the energy needed to learn multiple associations could rapidly become enormous. For instance learning the well-known MNIST data set requires at some 10 8 synaptic updates (in preparation). Saving strategies will be needed in that case  (Li and Van Rossum, 2020 ). An alternative is that the major consumers are changes in brain activity by coordinated processes such as replay -which contributes to memory consolidation in mammals, but also in flies  (Cognigni et al. , 2018). Calorimetry during learning could provide insight into such hypotheses. Finally, behavioral or physiological changes resulting from the learning protocol might explain the increased consumption. Control experiments with unpaired stimuli in  Mery and Kawecki (2005)  might not have completely corrected for such effects.