dataesr/scientific-paragraphs-categorization huggingface dataset:
6 of NEMO  (Madec & NEMO Team, 2017 ) and version 5. 1 of the Community Ice CodE (CICE,  Hunke et al. , 2015). In addition, a few different parameter choices may affect the link between hydrographic properties in front of the ice shelf and the melt at the base of the ice shelf. The training ensemble was computed on 121 vertical levels (representing 20 m at 600 m depth), while the testing ensemble was computed on 75 vertical levels (representing 60 m at 600 m depth). In both ensembles, the thickness of the top boundary layer is bound at 20 m but can differ locally due to the different vertical resolutions. In the training ensemble, the thermal Stanton number is set to 7×10 -4 while in the testing ensemble the thermal Stanton number is set to 1. 45×10 -3. In the training ensemble, the top tidal velocity varies locally based on the CATS2008 dataset  (Padman et al.
In this study, we explore different numbers of layers and numbers of neurons per layer. As activation func-tion, we use the rectified linear unit  (ReLU, Fukushima, 1975; Nair & Hinton, 2010). The multilayer perceptron is implemented in Python with the package Keras  (Chollet et al. , 2015). In parallel to the training, the neural network is applied to a validation dataset to monitor its performance on data that has not been used for the training. After training, the final performance of the neural network is estimated by applying it to a previously unseen testing dataset. In this study, to train the neural network, the loss which we reduce is the meansquared-error over all ice-covered points between the predicted (m N N ) and target (m ref ) basal melt rates, M SE = Npts i Nyears t (m NN [i, t] -m ref [i, t]) 2 N pts N years (1) where N pts is the number of ice-covered grid points and N years is the number of years used in the training.
In  Burgard et al. (2022) , we argued that tuning on the grid-cell level would give too much weight to the larger ice shelves, as they cover a larger area. We still agree with this statement for traditional parameterisations because they already intrinsically contain assumptions about the physics of the circulation and the melt before tuning and have only one or two tuneable parameters. In the case of our neural network, the relationship between the properties in front of the ice shelf and the melt is learnt from scratch, and it contains a larger number of parameters to adjust. We therefore argue that training on the grid-cell level is more sensible. The neural network is optimised with Adam  (Kingma & Ba, 2014) , an initial learning rate of 0. 001, β 1 =0. 9 and β 2 =0. 999. We split the training dataset in batches with a size of 512 samples and optimise the neural network for at most 100 epochs.
Training, validation and testing methodology In a first step, we explore different neural network sizes using the method of cross validation on our training ensemble. In a second step, we choose a subsample of the neural networks to explore their performance on the testing dataset. We conduct two variations of leave-one-block-out cross validation to estimate the validation loss (MSE as defined in Eq. 1), one on the ice shelf dimension and one on the time dimension, like in  Burgard et al. (2022). This approach consists of dividing the dataset into N blocks, training the neural network to minimise the training loss on N -1 blocks and using the left-out block to compute the validation loss  (Wilks, 2006; Roberts et al. , 2017). The procedure is re-iterated N times, leaving out each of the N blocks successively, so that, in the end, each N -th block has been left out of training once.
All predictions for the left-out blocks, using the separately trained neural networks, are then concatenated to form a "synthetically independent" evaluation dataset. Applying an evaluation metric on this evaluation dataset, we assess how well the neural network generalises to data "unseen" during training. We use N =35 for the cross validation over ice shelves. For the cross validation over time, we divide the years into blocks of approximately 10 years (ten 10-year blocks and three 9-year blocks) to reduce the effect of au- Before training, we normalise the training sample to put each of the 14 input variables (listed in Fig. 1 ) as well as the output variable on a similar order of magnitude and avoid potential problems of gradient explosion. We do so by subtracting the mean and dividing by the standard deviation of the training sample.
After cross validation, we choose a subsample of these neural networks to do further evaluation on a completely independent dataset. To do so, we reiterate the training of the subsample of neural networks over the whole training dataset and choose to work with a deep ensemble  (Lakshminarayanan et al. , 2017). The final weights and biases of neural networks depend on the initialisation of the weights before the first training iteration  (Goodfellow et al. , 2016). To account for this uncertainty and gain a more robust performance from the neural networks, we reiterate the training of the subsample of neural networks ten times with ten different random initialisations. We then apply this deep ensemble of ten neural networks to the independent testing input and compute an ensemble mean over the ten resulting melt rates.
The RMSE for the cross validation over ice shelves is higher than for the cross validation over time but remains on the lower end of the range of RMSEs given by traditional parameterisations. The RMSE int of the cross validation over time is very similar between neural network sizes and spans between 6 Gt/yr (XL) and 11 Gt/yr (S). It remains well below the mean reference integrated melt on the ice-shelf level of 39 Gt/yr. 4 ), right panel, on the synthetically independent evaluation dataset. cross validation over ice shelves varies more and is higher, between 24 (S) and 45 Gt/yr (M). The performance does not correlate with the neural network size. On the contrary, the lowest RMSE int of the cross validation over ice shelves is found for a comparably small neural network (S). For the melt near the grounding line, the RMSE GL does not vary much in both cross validations between neural network sizes.
For Pine Island and Amery, the XS and S patterns match the reference better than the L pattern, while, for Fimbul and Totten, the L pattern is a little better. The low RMSE in the cross validation over time suggests an overfit on the geometry, which is fixed over time in the training dataset. The patterns very close to the reference in the cross validation over time show that, even if our neural networks are applied on each grid-cell separately, the location of the grid cell is more or less encoded in one or more input variables. However, as our problem is not necessarily well constrained with the input variables given, we suggest that this overfit can be used to our advantage. Our hypothesis is that, if the neural network has seen each ice shelf once, it has captured the variety of geometries and will be able to generalise to future changes in these "known" ice shelves.
Testing on independent simulations We apply our subsample of neural network sizes on two independent datasets, one As a consequence, this evaluation is demanding and permits to evaluate the limits of the neural networks. For evaluation, we divide the 4xCO 2 run into two 30-year blocks to capture potential differences with warming in time. As explained in Sec. 2. 4, we train the XS, S and L neural networks ten times each, with ten different random initialisations. In the following, the results shown are averages over the predictions of the ten ensemble members for each neural network size.
95-1. 10 m/yr in the second period) is close to the reference mean melt near the grounding line (0. 75 m/yr for the first period and 1. 02 m/yr for the second period). In the first period, only the quadratic local parameterisation using an Antarctic slope and the plume parameterisation have lower RMSE GL (0. 62 and 0. 59 m/yr respectively), while in the second period only the quadratic parameterisation using a local slope performs clearly worse than the other parameterisations. For all, the RMSE increases with warmer conditions but the gap between the periods depends on the parameterisation, ranging from a difference of 0. 04 m/yr for the plume parameterisation to a difference of 0. 76 m/yr for the quadratic parameterisation using a local slope. From this demanding application on an independent testing dataset, several conclusions can be drawn.
Possible improvements While the results of our neural networks are encouraging, a variety of further improvements can be conducted in the future. The most obvious conclusion from this study is that predicting warmer conditions, similar to climate change conditions, is challenging for this particular neural network architecture because these conditions were not con-tained during training and neural networks are known to struggle with extrapolation problems. We therefore suggest, when possible, to introduce a set of simulations containing high-end future scenarios in the training dataset to make the neural network more robust for future projections. At the same time, we saw that the traditional parameterisations struggle to represent future conditions as well. How to tune melt parameterisations to be applicable in both present and future conditions is therefore a problem that is not limited to deep learning approaches.
Conclusions In conclusion, we show that a rather simple neural network architecture can be used to emulate a cavity-resolving ocean model. Our multilayer perceptrons are designed to be rather simply usable as a basal melt parametrisation for ice-sheet modellers. They use input properties needed for the traditional parameterisations already and can be applied on the grid-cell level, similarly to most traditional parameterisations. While they struggle nearly as much as traditional parameterisations to generalise to ice shelves unseen during tuning, the neural networks generalise much better on time blocks unseen manuscript submitted to Journal of Advances in Modeling Earth Systems (JAMES) during training and the patterns are clearly better represented. In the demanding testing phase, on a dataset produced with different NEMO parameters, geometry perturbations unseen during training and different forcing, they still perform at least as well or even better than traditional parameterisations, both in historical and much warmer conditions.
(2021). The two test simulations differ in their atmospheric forcing. In the first one, which we will call "RE-PEAT1970", UKESM1. 0-ice was run for several decades under constant 1970 greenhouse gas and other forcings. In the second one, which we will call "4xCO2", UKESM1. 0-ice was run for several decades under instantaneously quadrupled 1970 CO 2 concentrations. In our study, we use 60 years of simulation, from year 10 to year 70, for both runs. The training and the testing dataset result from NEMO simulations. Nevertheless, next to differences in forcing from the atmosphere and the ice and bed geometry, the train- ing 20 m at 600 m depth), while the testing ensemble was computed on 75 vertical levels (representing 60 m at 600 m depth). In both ensembles, the thickness of the top boundary layer is bound at 20 m but can differ locally due to the different vertical resolutions.
In the training ensemble, the thermal Stanton number is set to 7×10 -4 while in the testing ensemble the thermal Stanton number is set to 1. 45×10 -3. In the training ensemble, the top tidal velocity varies locally based on the CATS2008 dataset  (Padman et al. , 2008; Howard et al. , 2019) , while it is fixed to 5 cm/s in the testing ensemble. In conclusion, this means that the testing ensemble is a slightly different model than the model which the neural network is trained to emulate and therefore represents a demanding testing experiment. The training and testing ensembles cover a range of states that do not necessarily match observational estimates of hydrographic properties and basal melt rates. In both standalone and coupled mode, eORCA025 configurations are prone to biases in the ocean circulation around Antarctica  (Smith et al.
The RMSE for the cross validation over ice shelves is higher than for the cross validation over time but remains on the lower end of the range of RMSEs given by traditional parameterisations. The RMSE int of the cross validation over time is very similar between neural network sizes and spans between 6 Gt/yr (XL) and 11 Gt/yr (S). It remains well below the mean reference integrated melt on the ice-shelf level of 39 Gt/yr. The RMSE int of the    2 ), left panel, and Eq. (  4 ), right panel, on the synthetically independent evaluation dataset. cross validation over ice shelves varies more and is higher, between 24 (S) and 45 Gt/yr (M). The performance does not correlate with the neural network size. On the contrary, the lowest RMSE int of the cross validation over ice shelves is found for a comparably small neural network (S).
Acknowledgments We thank  Dani Jones ,  Paul Holland ,  Tom Andersson ,  Alex Bradley ,  Simon Thomas ,  Anna Vaughan , and many others at the  British Antarctic Survey  for interesting discussions and exchange on ocean, ice, and machine learning. Most of the computations presented in this paper were performed using the GRICAD infrastructure (https://gricad. univ-grenoble -alpes. fr), which is supported by  Grenoble  research communities. The NEMO simulations were performed using HPC resources from  GENCI-CINES  ( MISOCS  project, allocations  A0080106035  and  A0100106035 ). This research was mainly conducted through the  DEEP-MELT  project ( IRGA Pack   IA 2021-2022 ), which is supported by  MIAI @ Grenoble Alpes  ( ANR-19-P3IA-0003 ). This research was also supported by the  European Union 's  Horizon 2020 research and innovation programme  under grant agreements no.
869304  ( PROTECT ),  820575  ( TiPACCs ) and  101003536  ( ESM2025 ), as well as by the  French National Research Agency  through the  AIAI  project ( ANR-22-CE01-0014 ). CB and NCJ developed the original idea of this paper. CB carried out all analyses and wrote the manuscript. PM carried out the NEMO simulations used for training and RSS carried out the UKESM simulations. NCJ, RS and JC provided valuable help and code for the definition of the ice-shelf masks when the ice shelves evolve over time. TSF provided methodological input on the training of neural networks and JEJ provided useful input about how to think about machine learning. CB, NCJ, PM, RS, RSS, JC, TSF, JEJ contributed to discussions. -24-manuscript submitted to Journal of Advances in Modeling Earth Systems (JAMES).
used for the training ensemble can be found on Zenodo: https://doi. org/10. 5281/zenodo. 7308352. The simulation data from (Smith et al. , 2021)  used for the testing ensemble will be uploaded on Zenodo as soon as possible. All code to train the neural networks and produce the figures can be found on Github: https://github. com/ClimateClara/basal melt neural network and will be uploaded to Zenodo upon paper acceptance. AbstractMelt rates at the base of Antarctic ice shelves are needed to drive projections of the Antarctic ice sheet mass loss. Current basal melt parameterisations struggle to link open ocean properties to ice-shelf basal melt rates for the range of current sub-shelf cavity geometries around Antarctica. We present a novel parameterisation based on deep learning. manuscript submitted to Journal of Advances in Modeling Earth Systems (JAMES) Predictive Uncertainty Estimation Using Deep Ensembles.
doi: 10 ing at least one hidden layer, this kind of emulator produces reasonable basal melt rates. 1029/2007JC004368 for our training ensemble, closer to the reference simulation than traditional parameter- Madec, G. , & NEMO Team. (2017). Nemo ocean engine (v3. 6-patch). Notes du Pôle isations. For testing, we use an independent ensemble of simulations that was produced de modélisation de l'Institut Pierre-Simon Laplace (IPSL), Zenodo, 27. doi: 10 with the same ocean model but with different model parameters, different cavity geome-. 5281/zenodo. 3248739 tries and different forcing. In this challenging test, traditional and neural network pa- Mathiot, P. , Jenkins, A. , Harris, C. , & Madec, G. (2017). Explicit representation rameterisations yield similar results on present conditions. In much warmer conditions and parametrised impacts of under ice shelf seas in the z ⋆ coordinate ocean than the training ensemble, both traditional parameterisations and neural networks strug- model nemo 3.
doi: 10. 5555/3104322. 3104425 Naughten, K. , De Rydt, J. , Rosier, S. , Jenkins, A. , Holland, P. , & Ridley, J. (2021). Two-timescale response of a large Antarctic ice shelf to climate change. Nature Communication, 12 , 1991. doi: 10. 1038/s41467-021-22259-0 NEMO Team. (2019). Nemo ocean engine. Scientific Notes of Climate Modelling Center , 27. doi: 10. 5281/zenodo. 1464816 Nicholls, K. W. , & Østerhus, S. (2004). Interannual variability and ventilation timescales in the ocean cavity beneath Filchner-Ronne Ice Shelf, Antarc- 10. 1016/j. jcp. 2012. 08. 037 tica. Journal of Geophysical Research: Oceans, 109 (C4), C04014. doi: Dinniman, M. , Asay-Davis, X. , Galton-Fenzi, B. , Holland, P. , Jenkins, A. , & Tim-10. 1029/2003JC002149 mermann, R. (2016). Modeling Ice Shelf/Ocean Interaction in Antarctica: A Padman, L.
1 ). Extensive tests were conducted on the receding horizon parameters offline using the recorded data for the first participant to determine the optimal trade-off between performance and accuracy among: • t mhe : large values lead to higher accuracy but lower solving frequencies; • frame x : the closer the frame is to the end (100% of n mhe ), the less it is filtered by the algorithm. These receding horizon parameters were fine tuned while varying t mhe from 4 ms to 120 ms and frame x amongst 0%, 25%, 50%, 75%, and 100% of n mhe , for three movements (shoulder abduction, flexion, and one that mimics hand cycling (see video in supplementary material)). Six criteria (italicized below) were used to determine the best parameters. As we targeted a delay below 125 ms  [5] , the overall estimation delay was computed as the sum of the solving duration and the delay introduced by the chosen kept frame x.
In addition, as the movements were not executed at a maximal contraction level, muscle activations should not reach saturation. Therefore, we evaluated the total number of frames where muscles were saturated (activation > 95% of the maximal voluntary contraction (MVC)) for each set of receding horizon parameters. To avoid non-physiologic results on the kinematics, the angular velocity's mean power frequency was computed for all joints, representing their inner variation  [27]. Then, the root-mean-square errors (RMSE) were calculated between estimated and experimental data (markers, EMG). Finally, the inverse dynamics based on experimental markers data was computed. The root-mean-square difference (RMSD) was calculated between the computed inverse dynamics and the estimated torques (sum of muscle-induced torques and additional torques).
Results are shown in Section III-A. The problem was implemented using Bioptim  [28] , a Python library dedicated to solving biomechanical OCP, leveraging the MHE preimplemented class. Non-linear programs were solved using Acados  [29] , with its real-time iteration scheme  [30]  to improve the robustness and computational cost of the convergence of the subproblems. In particular, the OSQP  [31]  solver was used to solve the quadratic program. It allows for stricter boundary constraints in contrast to HPIPM  [32]  used in our previous study  [17]. To avoid indefinite Hessian in the OSQP, a regularization term was added through the Levenberg-Marquardt coefficient  [33]. Then, a damped Newton method was used by adjusting the step length to 0. 9 to improve convergence rate  [34]. Also, the Jacobian matrix was reused within Newton iterations to reduce computational cost  [35].
Data were synchronized using Nexus 2. 8 (Vicon, Oxford, UK). Ten MVC were collected following the recommendations of  [37]. Offline EMG signals were band-pass filtered (10-425 Hz), rectified, and lowpass filtered (5 Hz) using a fourth-order Butterworth filter. Then, maximal activations were obtained by computing the mean of the 2000 highest values (corresponding to 1 s) for each muscle during all 10 MVC trials  [37]. A static pose was used to scale the MSK model using OpenSim  [38] , which was then converted into a Biorbd model format  [39] , as Bioptim MSK dynamics relies on Biorbd  [39]. Finally, the participants were asked to perform two arm elevations in the frontal and sagittal planes and movements mimicking hand cycling in the sagittal plane. For further validation, their were asked to perform motions while holding a 2 kg dumbbell which result in different muscle activity levels.
E. Real-time software implementation We developed a fast (C++ core) and open-source Python library to achieve real time muscle force estimation and biofeedback using live data. Two scripts were implemented: (1) sensor data streaming and processing, and (2) muscle force and kinematics estimation and visual biofeedback. To enhance performance, the two scripts were executed on two computers connected through TCP/IP connection, with the Python multiprocessing package used for parallelization. The software architecture is depicted in Fig. 4. 1) Sensor data streaming and processing (1st computer; three tasks): During Task #1a, skin markers and EMG signals were streamed in real time from Nexus using the Vicon DataStream SDK Python interface  [40]. Concurrently, joint kinematics were computed from the marker 3D positions using EKF, and EMG envelopes were generated using a sliding window (2000 frames, 1 s), band-pass filtering (10-425 Hz), rectification, and moving average smoothing (200 frames) during Tasks #1b.
In Task #1c, a TCP/IP server was started to transmit EMG envelopes, joint angles, and marker trajectories to the second computer on demand. 2) Muscle force and kinematics estimation and biofeedback (2nd computer; two tasks in parallel): Task #2a involved implementing the optimal estimator described in Section II-C and updating tracked data at each iteration via the TCP/IP server. EKF-based joint angles were used as an initial guess, and estimated data were made available for transmission to a visual application at the end of each iteration. In Task #2b, real-time biofeedback on optimal joint kinematics and muscle forces was provided through visualization (Fig. 7 ).
B. Real-time performances Data were streamed and processed (Nexus software included) on a Windows 7 computer (Intel Xeon® X5680 @ 3. 33 GHz 6-core processor). The MHE and biofeedback were carried on eight cores of a second computer (AMD Ryzen® 9 5950x 16-core processor) running Ubuntu 20. 04 LTS. The program estimated and displayed the 31 muscle forces, and the 10 DoFs 3D kinematics at 30 Hz with a total delay of 110 ms (Section II-E). The delays introduced by each step are summarized in Tab. I. Main delays come from the second computer for the estimation and visualization tasks (80%). Data streaming and processing on the first computer correspond to 20% of the total latency.
Estimated activations, corresponding EMG (if applicable), and muscle forces are shown in Fig. 6  for the first participant, for an abduction movement, with and without a 2-kg dumbbell. As expected, higher activations and forces were estimated when holding the dumbbell, even for non-tracked muscles. This shows the ability of the solver to consider experimental EMG but also to generate a physiological prediction for untracked muscles. Furthermore, the activation level of the infraspinatus and supraspinatus were of the same order as in  [41]  (Fig. 6 , data were extracted from graphics using Engauge Digitalizer  [42] ). The subscapularis shows the same activation pattern but with an activation almost three times higher. A loss of activation is observed (Fig. 6 ) for the supraspinatus around 90°of abduction which differs from  [41].