dataesr/scientific-paragraphs-categorization huggingface dataset:
The NNs are trained 10 times with different random seeds (s ∈ [0, 9]), and all results are averaged over the 10 trained networks. As a baseline method, we use a persistence forecast with the initial conditions as a constant prediction. We additionally compare the forecasts corrected by the NN to the uncorrected forecasts from our sea-ice model. In the following, we discuss the results on the test dataset in Sect. 5. 1, what we can learn about the residuals by analysing the sensitivity of the NN to its inputs in Sect. 5. 2, and how we can combine the NN with the geophysical model for lead times up to 1 h in Sect. 5. 3.
Performance on the test dataset In the first step, we evaluate the performance of our model error correction on the test dataset, without applying the correction together with the geophysical model, Table  2. For additional results we refer to Sect. C and Sect. D, where we, among other things, compare with other NN architectures, other loss functions, and other activation functions. The NN corrects the model forecasts across all variables. This results in an averaged gain of the hybrid model over 75 % compared to the sea-ice model. For the stress, damage, and area, the persistence forecast performs better than the sea-ice model, as the model forecast drifts towards the attractor of the low-resolution model setup, as discussed in Sect. 5. 3. Since the NN uses the initial conditions as input, the hybrid model surpasses the performance of persistence, even for variables where persistence is better than the sea-ice model.
In Sect. C, we show that the model error of the sea-ice model is mostly driven by a dynamical error such that simply correcting the bias has almost no impact on the performance. In total, the NN consistently improves the forecast on the test dataset. To apply convolutional NNs (CNNs) to the raw data of our finite-elements-based sea-ice model, we project from triangular to Cartesian space, where the features are extracted. The number of elements in the Cartesian space determines its effective resolution and, thus, the finest scale on which the NN can extract features. To demonstrate the effect of different resolutions on the result, we perform three different experiments, where we change the grid size while keeping the NN architecture the same (Table  3 ). The training loss, here the negative Laplace log-likelihood, measures how well an NN can be fitted towards the training dataset.
By shuffling an input variable, its information is destroyed, and the output of the NN is changed. This possibly changes the prediction error compared to the unperturbed original output. Focussing on active regions, we measure the errors with the RMSE, estimated over the whole test dataset. The higher the RMSE for a shuffled input variable, the higher the importance for this variable onto the errors, as summarised in Table  5. Because the information of only single variables are destroyed, the permutation feature importance is sensitive to correlated input variables (Sect. D5). Consequently, the intervariable importance in Table  5  is likely underestimated. All model variables are highly sensitive to their own dynamics. Furthermore, the feature importance reflects the relations inherited by the model equations (see also Sect.
2. 2;  Dansereau et al. , 2017). For instance, caused by the dependence of the thickness upon the sea-ice area, they are linked together in the input-output relation. The wind forcing externally drives and influences the sea-ice velocity in the y direction, v. The v velocity, however, advects and mixes the Table  5. The permutation feature importance of the RMSE for the given output variable with respect to the input variable for "Initial + Difference" as input, estimated over the whole test dataset. The numbers show the multiplicative RMSE increase of a specific output variable (row) if a given input variable (column) is permuted; a higher number corresponds to a higher feature importance. The colours are normalised by the highest feature importance for a given row (output variable) and proportionally to the feature importance.
6g , and is directionally dependent on the velocity dynamics, Fig. 6h. Hence, the NN seems to rely on localised and anisotropic features to predict the residual. Based on these sensitivities, we can interpret what features the NN has learned, guiding us towards a physical meaning of the model errors. The diametral impacts of the thickness and area in absolute values and dynamics indicate that the sea-ice model tends to overestimate the effect of the dynamics, whereas the initial conditions have a stronger persisting influence than predicted by the model. However, the connectivity between grid points is underestimated by the model, as seen in Fig. 6f. In general, the model overestimates the fracturing process, leading to a mean error of 2. 31 × 10 -3 for the damage in the training dataset. This overestimation of fracturing could also explain the very localised impact of the damage dynamics, Fig.
We change the performance metric to be the RMSE, a commonly used metric to evaluate forecast performances. We evaluate the performance across all 2400 hourly time slices on the test dataset. For forecasting purposes, the NN with the initial and forecasted fields as input performs generally better than the NN with initial and difference fields (Sect. D6); for simplification in the following, we use only the NN with the initial and forecasted fields, again calling it the "hybrid model". Overall, the hybrid models surpass the performance of the original geophysical model (Fig. 7 ). However, for the forecast with the sea-ice model and the hybrid model, a strong drift is evident. As correcting the bias has almost no impact on the performance in the test dataset (Sect. C), this drift is not caused by model biases.
We optimise the nine scale parameters together with the NN by minimising the negative log-likelihood, averaged over B data pairs (x j , x j ). As we utilise a variant of stochastic gradient descent for optimisation, the data pairs are drawn from the training dataset D at each iteration. Before summing all nine loss terms up, we average the negative log-likelihood per variable across all grid points (here simplified denoted as average across M grid points), as the velocity components have fewer data points than all other variables, caused by their spatial discretisation (see also Sect. 2. 2) L total = 1 BM 9 i=1 B j =1 M k=1 1 b i | x i,j,k -f i,j,k (x, φ)| + ln(2b i ). The factor in front of the absolute error, λ i = 1 b i , is the weighting factor; the MAE can be recovered by setting b i = 1 as a constant.
The additional term, ln(2b i ), originates from the normalisation of the Laplace distribution, is independent of the errors, and counteracts a too-small b i. This approach optimises b i to match the expected MAE  (Norton, 1984)  in the training dataset and can be seen as an uncertainty estimate, e. g. recently used in  Cipolla et al. (2018) ;  Rybkin et al. (2020). Compared to using a fixed climatological value, this approach adaptively weights the loss, depending on the error of the NN for the different variables. This adaptive weighting marginally improves the training of the NN, as shown in Sect. D3. Since we learn the scale parameters purely from data, this approach can be seen as type II maximum likelihood or empirical Bayes approach  (Murphy, 2012). dataset. We show what would happen if we would use other CNN architectures (Table  C1 ).
Detailed NN configurations can be found in Sect. C1. We have selected the parameters of the U-Net architecture after a randomised hyperparameter screening in the validation dataset with 200 different network configurations per architecture, like for the U-NeXt architecture. The simplest approach to correct the model forecast is to estimate a global bias, one for each variable, in the training dataset and to correct the forecast by this constant. As we measure the MAE, we take as bias the median error instead of the mean error in the training dataset. Correcting the bias has almost no impact on the scores, and, consequently, the model error is dominated by dynamical errors. As a next level of complexity, we introduce a shallow CNN architecture with one layer as feature extractor. Using dilation in the convolutional kernel, this layer can extract shallow multiscale spatial information for each grid point.
Replacing the classical convolutional layers with Con-vNeXt blocks as described in Sect. B2 gives an additional improvement in the performance of the NNs. Although the number of parameters for the U-NeXt is only a third of the number for the U-Net, the ConvNeXt blocks reduce the forecast errors by an additional 14 %. The blocks hereby reduce the errors in the damage and area especially. The ConvNeXt blocks are able to extract more information from existing data than the convolutional layers. Because the U-NeXt is also the best-performing method in the validation dataset, we use this architecture throughout the paper. Training "Conv (×1)" for each model variable independently has only a marginal impact on the scores, although their latencies are much larger than for the shared NN. The convergence issues in the "Conv (×5)" and "Independent Conv (×1, 128)" architecture indicate that the improvement of the bigger neural networks is not only related to an increased number of parameters but also because of their multiscale layout.
For the downsampling in the encoder, we sequentially use one convolutional layer with a 3 × 3 kernel, stride of 2 × 2, and zero padding; batch normalisation; and Gelu as the activation layer. For the upsampling in the decoder, we sequentially use bilinear interpolation; one convolutional layer with a 3 × 3 kernel, stride of 2 × 2, and zero padding; batch normalisation; and Gelu as the activation Table  C1. Normalised MAE on the test dataset for different NN architectures, shown are average and standard deviation across 10 training seeds. Reported are the errors for the velocity component in y direction v, for the stress component σ yy , the damage d, and the area A. The mean is the error averaged over all nine model variables, including the non-shown ones. A score of 1 would correspond to the performance of the raw forecast in the training dataset.
D1 Number of training samples Large NNs have many parameters, in the case of the U-NeXt 1. 2 × 10 6 , and could fit functions with as many degrees of freedom. If the number of degrees of freedom in the training dataset is similar or even lower, the NN might perfectly fit and remember the training dataset. As there is noise and spurious correlations within the dataset, the NN would also learn these "features" and overfit towards the training dataset. In this overfitting case, the NN would fail to generalise and to give good predictions to data unseen during training. In the following, we analyse the training behaviour of the NN and see what happens if we artificially train on a portion of data only (Fig. D1 ). For the validation dataset, over the course of the training, we show the negative log-likelihood (NLL) with a Laplace assumption and scaling parameters fixed to their climatological values.
This loss is proportional to a mean absolute error (MAE) with a fixed weighting, where the weights are given by the inverse scaling parameters. Additionally, we analyse the weighted MAE in the testing dataset for the different NNs after training. For all fractions of training data, the validation NLL smoothly decreases over the course of training. Consequently, we see no overfitting, even with only 10 % of training data. Additionally, the loss and the weighted MAE saturates as a function of fraction in the training data: the gain training on more data is larger for small sample sizes than for larger sample sizes. Such a logarithmic data scaling behaviour is expected and can even be observed for very large language models  (Kaplan et al. , 2020). Given this scaling, we would need much more data to scale the performance further.
D3 Loss functions Optimising the Laplace log-likelihood corresponds to minimising the mean absolute error (MAE), whereas an optimisation of a Gaussian log-likelihood minimises the meansquared error. Thus, we report the averaged root-meansquared error (RMSE) and MAE over all variables to measure the influence of the loss function on the performance of the NNs (Table  D1 ). As the RMSE and MAE are normalised by their climatological values in the training dataset, the weighting between the model variables is fixed to their climatological values, favouring fitting networks with fixed climatological weighting. Compared to a Gaussian log-likelihood with trainable variance parameters, the Laplace log-likelihood as the loss function improves not only the MAE by around 17 % but also the RMSE by around 3 %.
15) is higher than for the Gaussian log-likelihood (0. 93), as can be seen in Fig. D3c. The distribution for the Laplace log-likelihood is additionally more balanced, meaning that less extreme values appear on both ends. We attribute these differences to the different behaviour of the loss function  (Hodson, 2022). As the Gaussian log-likelihood is more sensitive to larger errors in the training dataset, the NN has to learn specialised feature maps for these cases. The Laplace log-likelihood leads to a higher contrast in the feature maps and to more balanced feature maps. Based on its increased contrast, we hypothesise that the Laplace log-likelihood results in better linearly separable feature maps. On their basis, the linear functions can more easily combine the features to predict sharper and more localised residuals, improving the performance of the NN.
D4 Activation functions Another decision that we took in our architecture is to use the Gaussian error linear unit (Gelu) in the blocks and the rectified linear unit activation (relu) function as the activation of the features, before they are projected back into triangular space and linearly combined. The Gelu activation function is recommended for use in a ConvNeXt block  (Liu et al. , 2022) , but its performance seems to us to be on par with the relu activation function. Whereas the Gelu activation function is a smooth function inspired by dropout  (Hendrycks and Gimpel, 2020) , relu is a non-smooth function, which induces sparsity in the feature maps. As similarly found in  Liu et al. (2022) , replacing the Gelu activation function with a relu activation function in the Con-vNeXt blocks leads to almost the same results on the test dataset (Table  D2 ).
D6 Forecasting with differences as network input For forecasting with the model error correction, we only show results for the NNs with the initial conditions and the forecasts as input, although the NNs with initial conditions and the difference between forecast and initial conditions performs better in the testing dataset. Here, we will shortly discuss the forecasting results of these latter NNs with the initial conditions and the differences as input (Table  D4 ). The dynamics are explicitly represented as the difference between the forecast and initial conditions. On the one hand, this helps the NN to extract more information from the dynamics than for the "Initial + Forecast" experiment (see also Table  4 ). On the other hand, this explicit representation introduces an assumption that the dynamics are additive to the initial conditions.
Acknowledgements. The authors would like to thank  Nils Hutter  and  Steffen Tietsche , by reviewing they helped to significantly improve the paper. The authors would additionally like to acknowledge other members of the project  SASIP  for the delightful discussions along the track. Furthermore, the authors received support from the project  SASIP  (grant no. 353), funded by  Schmidt Futures -a philanthropic initiative  that seeks to improve  societal outcomes through the development of emerging science and technologies. This work was additionally granted access to the HPC resources of IDRIS under the allocation  2021-AD011013069  and  2022-AD011013069R1  made by  GENCI. Financial support. This research has been supported by the  Schmidt Family Foundation  (grant no. 353 ) and the  Grand Équipement National de Calcul Intensif  (grant nos.
Previous studies show that the simulated isotopic distributions compare favorably with observation and other models of similar complexity  (Brady et al. , 2019; Nusbaumer et al. , 2017; Wong et al. , 2017). We performed three simulations: (a) a preindustrial control (PI) experiment, (b) a LGM experiment with period appropriate boundary conditions, and (c) a LGM land-ice only (LGM-ice_only) experiment, using the LGM ice sheet configuration with PI CO 2 and orbital conditions. Coastlines and sea levels are implemented by the oceanic and atmospheric/land surface models but are amended by comparing with the geological records in LGM  (Kageyama et al. , 2018b). Initial ocean oxygen isotopic distributions came from the GISS interpolated ocean δ 18 O data set  (LeGrande & Schmidt, 2006). Ocean average δ 18 O was increased by +1‰ for the LGM and LGM-ice_only experiments to account for the large ice sheets  (Duplessy et al.
, 2002). Ice volume and topography came from the ICE-6G data set  (Peltier et al. , 2015). All simulations were initialized from previous experiments and run for an additional 550 years with water isotope tracers, allowing the atmosphere, land, and upper ocean to reach near equilibrium; data analyzed are from the final 48 years of each simulation. To quantitatively gauge climate model performance, we calculated the skill score for temperature anomalies estimated by each of the models considered in this study using the equation: Skill score (SS) = 1 - √ √ √ √ Σ(𝑚𝑚𝑖𝑖 -𝑜𝑜𝑖𝑖) 2 -Σ ( 𝑒𝑒 2 𝑖𝑖 ) Σ(𝑛𝑛𝑖𝑖 -𝑜𝑜𝑖𝑖) 2 -Σ ( 𝑒𝑒 2 𝑖𝑖 ) where m i are the model results, n i are the reference (in our case, n i = 0, with no change from the LGM to modern), o i are the observations (clumped results), and e i are the observation uncertainties  (Hargreaves et al.
88 m with T( 𝐴𝐴 ∆ 47 ) of 24 ± 3°C and 28 ± 3°C, respectively, in the LGM average. Our data also provide further resolution on both the magnitude and timing of variations inferred by pollen and phytolith data sets. The data, as shown in Figures  3a  and 3b  provide quantitative evidence to support the qualitative hydroclimate reconstructions from pollen in nearby Carp Lake, where a transition in vegetation type occurred from the LGM to modern that would reflect a change to a hotter, drier environment  (Whitlock et al. , 2000). The temperature shift in Figures  3a  and 3b  matches the timing of changes in paleoclimate inferred from phytoliths in the Palouse loess  (Blinnikov et al. , 2002) , where colder conditions prevailed throughout the LGM and deglaciation, but a switch to warming occurred during the Holocene.
Paleoceanography and Paleoclimatology LOPEZ-MALDONADO ET AL. 10. 1029/2021PA004266 16 of 22 multimodel average (SS = 0. 37) were similarly skilled when evaluated using a combination of clumped isotope and pollen data but had lower skill scores than when compared with only the clumped isotope reconstructed temperatures. Only three PMIP3 models (MIROC-ESM, MPI-ESM-P, and COSMOS-ASO) exhibit negative skill when evaluated using clumped isotope and pollen data. We also compared our proxy data to the transient evolution of mean and warmest month temperatures from TraCE-21k and iCESM (Figures  4c  and 4d ). Our MAAT and WAMT results follow the same broad pattern of temperature change as the TraCE simulation, with a warming trend through the last deglaciation. However, the proxy temperature estimates for MAAT show significantly larger-magnitude changes than the TraCE simulation results.
Conclusions This study provides novel paleoenvironmental constraints for the Pacific Northwest, a region that was proximal to the ice sheet margin during the LGM, from clumped isotope analysis and water isotope-enabled simulations. We use clumped isotope measurements of paleosols to reconstruct temperatures and water isotope values from soil carbonates ∼36 to 9 ka in age. At the LGM, when mean annual surface temperature changed globally by 4-7°C, T( 𝐴𝐴 ∆ 47 ) suggests that regional MAAT and WAMT changed by around 2. 5 to 3 times this amount in our study area. Calculated soil carbonate formation temperatures, MAAT, and WAMT exhibit a pattern of change from ∼36 ka to modern that correlates with paleoenvironmental constraints from pollen and phytoliths as well as with model simulations. Model-data comparison indicates the magnitude of temperature change is likely explained by the proximity of the study area to the LIS margin, the resulting influence of the glacial anticyclone on the region, and local albedo, while changes in the isotopic composition of precipitation largely reflect depleted column integrated vapor during the LGM, likely due to atmospheric circulation changes and enhanced rainout as moisture moves inland.
Data Availability Statement The iCESM model data presented in this manuscript are available in the Zenodo repository  (Tabor, 2021). Reconstructed climate parameters and proxy data from Tables and Supporting Tables will be archived at the NOAA's NCEI World Data Service for Paleoclimatology on acceptance at  Lopez-Maldonado et al. (2022) , https://doi. org/10. 25921/r09x-m706.
Discussion The presented ostracod, mollusk and paleomagnetic records as well as age estimates are largely in agreement with previous stratigraphic studies (see Section 2. 1), but integrated with the sedimentology and illustrated in greater detail  (Figs. 2, 14). The position of the Dacian Basin Pontian stage and its substage boundaries in the section are revised where necessary, according to their biostratigraphic definitions  (Marinescu and Papaianopol, 1989). The new dataset is compared with previous results in the adjacent Sla ˘nicul de Buza ˘u record of the Dacian stage  (Jorissen et al. , 2018)  and previous results in the nearby Râmnicu Sa ˘rat section, covering the whole Pontian  (Stoica et al. , 2013). We also highlight some new insight that our work offers on the correlation of sedimentary trends, potential local limitations to chronostratigraphy and restriction of the Dacian Basin during the Pontian.
Data and Methods Constructing APWPs from site-level data requires novel tools and data compilations. Historically, paleomagnetic data have been reported at the site-level but compiled at the study-level. Here, we compiled Cenozoic (0-60 Ma) site-level data and associated metadata from North America (Section 2. 1). In Section 2. 2, we present a new framework enabling APWPs to be constructed from such data.
Cenozoic North America at the Site-Level: Compilation and Structure Site-level paleomagnetic data include a wealth of information that is lost when collapsed into summary statistics describing a paleopole. Our compilation retains this information. Here we note a few salient aspects of the compilation, while the full details are presented in Text S1 in Supporting Information S1. First, we retrieve individual site-level paleomagnetic directions and their corresponding VGPs. This approach enables us to work with the actual observational assemblages of directions/VGPs, rather than parametric models of them. Second, each site-level datum is associated with a numeric age and uncertainty that may not be represented in the overall study-level paleopole. Each site-level datum is thus liberated from its study-level collection, enabling it to be independently re-combined with coeval site-level data.