dataesr/scientific-paragraphs-categorization huggingface dataset:
Benefiting from this international collaboration, the dataset comprised a total of 9,143 milk samples from 3,758 cows located in 589 herds across 10 countries and represented 7 breeds. The samples were analyzed by reference chemistry for biomarker contents, whereas the MIR analyses were performed on 30 instruments from different models and brands, with spectra harmonized into a common format. Four quantitative methodologies were evaluated to address the strongly skewed distribution of some biomarkers. Partial least squares regression was used as the reference basis, and compared with a random modification of distribution associated with PLS (random-downsampling-PLS), an optimized modification of distribution associated with PLS (KennardStonedownsampling-PLS), and support vector machine (SVM).
The resulting positively skewed distribution is particularly difficult to model, as it goes beyond the linear relationship of milk molecules and spectral absorbance values. Other authors emphasize the lack of compatibility between the main modeling method with MIR (i. e. , partial least squares [PLS] regression) and these asymmetric distributions of molecules, as well as the need for alternative modeling methodologies  (Soyeurt et al. , 2020; Kostensalo et al. , 2023). Finally, to be used for large-scale phenotyping, MIR models must be robust (i. e. , provide reliable predictions under all conditions), covering as much variability as possible to avoid extrapolation  (Grelet et al. , 2021). Therefore, the objectives of this study were (1) to evaluate the potential of milk MIR spectra to predict key biomarkers of energy deficit (citrate, isocitrate, glucose-6P, free glucose), ketosis (BHB and acetone), mastitis (NAGase and LDH), and fertility (progesterone); (2) to test alternative methodologies to PLS, randomdownsampling-PLS, and KennardStone-downsampling-PLS, specifically designed to better account for the specific asymmetric distribution of biomarkers, and SVM, which is known for its capacity to handle nonlinear relationships; and (3) to create robust models by merging large datasets from 5 international or national projects, reaching a total number of 9,143 samples.
Milk Sample Analysis for MIR Spectra Aliquots for MIR analysis were stored at 4°C with 0. 02% to 0. 03% bronopol until analysis. Analyses were performed locally on a wide range of instruments with a total of 30 spectrometers used to analyze the samples: 24 Foss instruments of models FT2, FT6000, FT+, and FT7 (Foss, Hillerød, Denmark), 5 Bentley FTS (Bentley, Chaska, MN), and 1 Standard Lactoscope FT-MIR automatic (Delta Instruments, Drachten, the Netherlands). All the spectra from the different instruments were standardized to be merged into a common dataset following the procedure described in  Grelet et al. (2015). Morning and evening samples were analyzed separately but merged into a daily spectrum, to be in line with milk recording 24H, in which recombined samples of a. m. and p. m. milking are analyzed.
Data Editing and MIR Model Development To avoid erroneous association between spectra and sample, as well as analytical issues during analysis, a local fat model was applied to the spectra and the generated predictions were compared with the predictions provided by the laboratories. Records with a difference above 0. 3 g/100 mL between local and laboratory predictions were discarded (n = 218 records deleted) to prevent the wrong association between spectra and samples, or other analytical errors. This threshold was highlighted in a study by  Zhang et al. (2021). Spectra with a standardized Mahalanobis distance (GH) greater than 10 were eliminated (n = 54 records deleted). A high GH value was intentionally used as the threshold because previous work showed that keeping large spectral variability in the dataset was beneficial to the robustness of the developed model  (Grelet et al.
, 2021). Additionally, only records with DIM between 5 and 365 were retained (n = 88 records deleted) and reference values under quantification thresholds were discarded. All these aforementioned edits eliminated 9% of the samples, and resulted in a dataset of 8,783 records. In descriptive statistics, skewness of each variable was calculated as skewness mean SD = - ( ) - ( ) ∑ - , N N N x i 1 2 3 with N being the number of samples and x i the ith observation within each variable. The MIR spectra were pretreated by a first derivative with a gap of 5 wavenumbers. The selected spectral area consisted of 212 wavenumbers from 968. 1 to 1,577. 5 cm  -1 , 1,731. 8 to 1,762. 6 cm -1 , 1,781. 9 to 1,808. 9 cm -1 ,  and 2,831. 0 to 2,966. 0 cm -1 to exclude spectral areas not reproducible between instruments  (Grelet et al.
, 2021)  and the absorbance values were mean-centered. For each biomarker, a dataset was constituted by keeping only the records with reference values and MIR spectra. Each dataset was then split to create an external herd validation dataset, randomly removing 20% of the herds (i. e. , 120 herds out of 589) across all countries to be used as the validation dataset. The remaining 80% were used as the calibration dataset. The modifications to the distribution described below were only performed in the calibration dataset and the original distribution for each biomarker in the validation dataset was kept unchanged, to be as close as possible to the real field application conditions. In both the calibration and validation datasets for each biomarker, all samples were retained without removing outliers.
As a basis for comparison, the first models were developed using PLS regression, this being the most commonly used method. The number of latent variables was set according to the breakpoint of the root mean square error (RMSE) slope during the cross-validation step. However, as mentioned by  Soyeurt et al. (2020) , PLS is not adapted to handle asymmetric distributions of molecules due to its linear structure. Therefore, to better take into account the strongly right skewed distributions of some molecules, 2 methodologies of modification of the distribution were specifically developed and tested to move toward a normal distribution. Partial least squares regression was later applied on the modified datasets. The first methodology was the one used in  Grelet et al. (2016) , randomly removing a portion of the low values to reduce the over-representation of low values in the dataset.
The thresholds for discriminating low and high values were obtained from the literature or were optimized during the cross-validation step regarding the model R 2 cv, and the proportion of low values to be removed was calculated to balance equally the proportion of low and high values. The random elimination was performed by cyclic iterations. After downsampling, the proportion of high values, and especially extreme high values, was still under-represented. Therefore, a logarithmic (10) transformation was applied to the reference values to be closer to a normal distribution. Partial least squares regression was then applied to the reduced dataset. This random modification of the distribution associated with PLS is later referred to as random-downsampling-PLS. However, the random elimination of samples, by discarding samples of interest, inevitably leads to loss of variability and robustness of the models.
Therefore, a second methodology was also tested for an optimized modification of the distribution to keep only the most informative samples in the dataset while harmonizing the distribution of samples over the range of reference values. To do this, a 3-step methodology was applied which is schematically represented in Figure  1. The dataset for each biomarker was first divided into 20 subsets of equal Y-interval across the Y range. In a second step, within each subset, a fixed number of samples (e. g. , n = 100) was selected using the Kennard-Stone algorithm, which iteratively selects the 2 most spectrally different samples until the number of samples to be selected was reached. The number of samples to be selected was manually optimized between 50 and 200. When the number of samples in the subset was lower than the number of samples to be selected (i.
e. , at the right end of the distribution containing samples with high content of biomarkers), all the samples were retained. In a third step, the selected samples among the 20 subsets were merged to compose a dataset with a harmonized distribution along the Y range, but keeping the most informative samples from the initial dataset. Finally, a logarithmic (10) transformation was applied to the reference values and a PLS regression was applied to the data as the extreme values were still under-represented. This optimized modification of the distribution associated with PLS is referred to as KennardStone-downsampling-PLS. Support vector machines regression (SVM-R) was tested as an alternative quantitative methodology to overcome the specific distribution. The SVM-R is a linear method adapted to nonlinear relationships due to its capacity to find a linear link in a space of higher dimension.
In regression the support vectors are the ones including a maximum of samples in the regression within an acceptable margin  (Brereton and Lloyd, 2010). Support vector machines regression was used after a PLS compression to reduce the dimension of the dataset to 14 latent variables. The LIBSVM algorithm was used with the epsilon version and a radial basis function kernel. The gamma, cost, and epsilon hyperparameters were optimized with a grid-search to minimize the RMSEcv. Due to the capacity of SVM-R to handle nonlinear data, no log-transformations were applied to biomarker contents. Finally, when the ability of MIR to predict biomarkers was too low to enable rough quantitative screening (R 2 cv < 0. 74;  Grelet et al. , 2021) , a qualitative methodology was tested to assess the possibility of discriminating low versus high values of biomarkers.
Discriminant models were developed with partial least squares discriminant analysis (PLS-DA) using the full dataset, the randomly balanced dataset (randomdownsampling-PLS-DA), or the optimized balanced dataset (KennardStone-downsampling-PLS-DA). Thresholds to discriminate low and high values came from the literature, 200 µmol/L for BHB  (Denis-Robichaud et al. , 2014) , 150 mmol/L for acetone  (de Roos et al. , 2007) , 4. 3 unit/L for LDH  (Chagunda et al. , 2006) , 5 ng/mL for progesterone  (Roelofs et al. , 2006) , or from personal communication for NAGase (8 unit/L; Torben Larsen, Aarhus University, Tjele, Denmark, personal communication). No threshold could be found for 3 biomarkers of energy deficit, isocitrate, free glucose, and glucose-6P; therefore, discriminant values for feed restrictions were visually estimated from published data; 0.
15 mmol/L for isocitrate  (Pires et al. , 2022) , 0. 25 mmol/L for free glucose  (Pires et al. , 2022) , and 0. 3 mmol/L for glucose-6P  (Billa et al. , 2020). The models were evaluated using the external herd validation set. When a logarithmic transformation was used, predictions were back-transformed (10 prediction ) to further evaluate model performance in a usual scale and for graphical representation. Model statistics, both in the calibration and validation steps, were expressed in terms of R 2 (coefficient of determination) and RMSE. Ratio of RMSE/standard deviation of global dataset (RPD) was not calculated as being not relevant for models with asymmetric distribution, for which standard deviation does not correctly describe the spread of the population  (Bellon-Maurel et al. , 2010).
As a last step, final models were developed using all the data, calibration and validation, and internal cross-validation R 2 cv and RMSEcv (10 subsets constituted by cyclic iteration) are shown. Discriminant model statistics were expressed in terms of sensitivity (percentage of good classification in the high content group), specificity (percentage of good classification in the low content group), and global accuracy (global percentage of correct classification). Computations and models were carried out with programs developed in Matlab 2022 (The Mathworks Inc. , Natick, MA) and the PLS toolbox v. 8. 5. 1 (Eigenvector Research Inc. , Wenatchee, WA).
This could be explained by the over-representation of healthy cows in our dataset, as most of the sampling did not focus on comparing healthy and sick cows, or by minor differences in the analytical process. Table 3 also shows that both ketosis (BHB and acetone) and mastitis biomarkers (NAGase and LDH) have asymmetric distributions with strong positive skewness. Figure  2  shows the distribution of each molecule according to sampling countries. It also particularly highlights the positive skewness of distribution of ketosis and mastitis biomarkers and their exponential increase in case of disorder. The Pearson correlation table between biomarkers is shown in Table  4. As different projects analyzed different biomarkers, not all the correlations could be calculated. Isocitrate is derived from citrate during the Krebs cycle and both molecules were found to be positively correlated in  Larsen (2014).
Citrate decreases with the increase of de novo fatty acid synthesis  (Garnsworthy et al. , 2006)  and is mentioned as an early indicator of physiological imbalance of the animal (Bjerre-Harpøth et al. , 2012). In the current dataset the correlation coefficients (r) between isocitrate and citrate could not be calculated, but both are weakly to moderately positively correlated with BHB (r = 0. 24 and r = 0. 45 for isocitrate and citrate, respectively). This moderate correlation with BHB may reflect that they are also associated with physiological imbalance but at a different degree of severity. A decrease of citrate has also been observed in case of mastitis  (Hyvönen et al. , 2010) ; however, in the present study a weak positive correlation was observed between isocitrate and NAGase and LDH, both of which increase in cases of mastitis.
Alternatively, removing low samples inflates the R 2 , which is directly affected by a better distribution over the range  (Davies and Fearn, 2006). Finally, in the calibration step some methodologies may overfit the hyperparameters to this particular dataset. Therefore, the calibration statistics does not allow to compare the methods on a common basis and the results should be considered with caution and in an informative way. Consequently, for the selection of the best model, the external validation statistics are to be considered only. Table  6  shows the performance of models in external herd validation, by applying the models to the 25% excluded herds across countries. Looking at the R 2 v and RMSEv, the best models in calibration were not necessarily the best in the validation step. Indeed, PLS provided the best results for isocitrate, glucose-6P, free glucose, and LDH.
The SVM-R provided the best results for citrate, BHB, acetone, NAGase, and progesterone. The PLS-random-downsampling and PLS-KennardStone-downsampling provided better results than PLS for some biomarkers (glucose-6P, BHB, acetone, LDH, and progesterone for PLS-randomdownsampling and BHB, acetone, and progesterone for PLS-KennardStone-downsampling). These molecules were the ones with highly skewed distributions. However, for these molecules, the best models were obtained using SVM-R. Therefore, although these 2 methodologies might provide some improvement compared with PLS, SVM-R seems to be the best strategy to take into account nonlinearity (i. e. , the exponential increase of minor molecules not associated with exponential changes in the main components). Therefore, with the current dataset, it is worth using a method adapted to nonlinearity rather than modifying the distribution to approach normality.
Additionally, SVM-R provides better results with the entire dataset than when associated with modification of distribution (results not shown), which enables keeping all the variability of the calibration dataset and provides a higher robustness to the models, leading to better results in the external validation. Therefore, classical PLS and SVM-R based on the full dataset provided the best results for the normal and skewed distributions, respectively. The results also indicate that retaining the full variability of the calibration dataset increases the robustness of the model and the quality of predictions when applied to external data. This demonstrates the benefits of expanding datasets through collaborations. As a final step, the calibration and validation datasets were merged to develop final models across the entire dataset, covering a maximum of variability.
94 and a relative error lower than 10%, which means that accurate quantitative screening can be considered with this model. The glucose-6P model shows a R 2 v of 0. 58 but a very high relative error (RMSE/mean) of 41%, indicating that the model is very inaccurate and should only be considered to detect extreme values  (Grelet et al. , 2021). Therefore, it is more reliable to focus on citrate predictions to obtain information on cow status. It is recommended not to use isocitrate and free glucose prediction models, due to low R 2 v of 0. 48 and 0. 28, respectively. Regarding biomarkers of ketosis, BHB models have similar performances in cross-validation and valida-tion, whereas the final acetone model shows a R 2 cv of 0. 6, which is lower than the R 2 in external validation (0. 76). Therefore, the R 2 v is probably too optimistic, due to the random exclusion of herds that were likely well predicted in the validation dataset, and it seems more relevant to consider the R 2 cv (0.
(2021). Furthermore, the inconsistency between validation results and cross-validation based on the whole dataset shows that the acetone model is dependent on a few extreme high points, indicating that the model is not yet sufficiently robust. Regarding mastitis biomarker models, as for acetone, the NAGase R 2 v is probably too optimistic, due to the random exclusion of herds that were likely well predicted in the validation dataset. The NAGase model shows a more realistic R 2 in cross-validation than in validation, and R 2 cv should be considered rather than R 2 v. The NAGase is predicted with a R 2 cv of 0. 42 and a high relative error (42%), which is not precise enough to be used for individual cow monitoring. Performance of the LDH model, with R 2 v of 0. 24 and a high relative error (76%), is also too low to be considered for use.
Qualitative MIR Models When the ability of MIR to predict biomarkers was too low to enable approximate quantitative screening (R 2 cv < 0. 74;  Grelet et al. , 2021) , a qualitative methodology was tested to assess the possibility of discriminating low versus high values for all biomarkers except citrate. Discriminant models were developed using different methodologies associating PLS-DA and under-sampling or not. The best results obtained during the external herd validation are shown in Table  7. The PLS-DA on the full dataset provided the best classification results for isocitrate, free glucose, acetone, and progesterone, whereas PLS-DA associated with random under-sampling provided the best results for BHB and LDH, and PLS-DA associated with Ken-nardStone under-sampling provided the best results for glucose-6P and NAGase.
The lack of consistency in the best methodology for both normally distributed and skewed molecules shows that no general conclusions can be drawn and that different methodologies should be tested for each biomarker. The results also show that under-sampling methods can be useful to balance the dataset before discrimination. Three biomarkers were predicted with a very high percentage of correct classification, with global accuracy around 90% for BHB, acetone, and NAGase. This validates that MIR spectrometry can discriminate high and low values of BHB and acetone. It also indicates that, unlike quantitative models, qualitative discriminant models can discriminate low and high NAGase contents with good accuracy, which could help in the management and detection of mastitis. Glucose-6P is predicted with a global accuracy of 83%.
Perspectives and Limitations The objectives of this study were (1) to evaluate the potential of MIR milk spectra to predict key biomark- (3) to create robust models by merging large datasets from international or national projects. Milk citrate is predicted with sufficient accuracy to allow accurate quantitative screening of energy deficit of cows. Quantitative and qualitative models predicting BHB and acetone are able to discriminate high from low values, which seems sufficient to detect cows suffering from subclinical ketosis. The qualitative discriminant model predicting NAGase, showing a global accuracy of 88% of correct classification, also demonstrates an ability to discriminate high from low values. It could play a role in the detection of mastitis, especially if flow cytometry is not available to provide SCC information (e.
The 3 updated quantitative models should increase the quality of predictions due to the increased variability in the calibration datasets, which has been facilitated by the international collaboration. However, the predicted raw values should not be considered as an end in themselves. Biomarker concentration may be highly dependent on DIM, parity, breed, and potentially other variables. Therefore, their use and the way information is disseminated at the farm level must take these important aspects into account. For example, assessing the dynamics of biomarkers throughout the lactation stage, or comparing with animals of similar characteristics, may be more appropriate than using the predicted raw value. These predicted biomarkers also come in addition to the wide range of already existing parameters, and their complementarity, or marginal effect, should be considered.
CONCLUSIONS Thanks to international collaborations, the working dataset consisted of a total of 9,143 records. Partial least squares and SVM-R based on the full dataset provided the best results for normal and skewed distributions, respectively. Regarding the ability of MIR to predict biomarkers of energy balance, citrate was very well predicted, which allowed for quantitative screening. Qualitative models indicate that MIR spectrometry can discriminate low and high values of BHB and acetone, as biomarkers of ketosis, with accuracy around 90%. A qualitative discriminant model can discriminate low and high NAGase contents with good accuracy (88% good classification), which could help in mastitis management and detection, especially if SCC through flow cytometry is not available (e. g. , in farm measurements).
ACKNOWLEDGMENTS The authors gratefully thank  Petimat Kitaeva ,  Claire Darimont ,  Mohamed El Morabit ,  Matthieu Dubuisson ,  Quentin Vigneron ,  Lora Renault ,  Olivier Genard ,  Maxence Didelez ,  François Rouelle , and  Sophie Mathieux  ( Walloon Agricultural Research Center [CRA-W], Gembloux, Belgium ) for their precious work. The OptiMIR project has received funding from  INTERREG IVB NEW  and the  Walloon Region. The  GplusE  project has received funding from the  European Union 's  Seventh Framework Programme for research, technological development , and demonstration under grant agreement no. 613689. The views expressed in this publication are the sole responsibility of the authors and do not necessarily reflect the views of the European Commission. The IndiKuh project " Evaluation of Animal Welfare in Dairy Farming-Indicators for the Metabolism and Feeding" (IndiKuh , funding code:  2817905815 ) was supported by funds of the  German Federal Ministry of Food and Agriculture (BMEL).
The weight loss percentage (WL%) was calculated as [1-(net weight of withered grapes in kg/net weight of fresh grapes in kg)] using the weight data measured with a ODECA ACS-30Z precision electronic balance with a maximum weight capacity of 30 kg ± 1 g (Odeca Srl, Varese, Italy). For each withering method, the initial weight of eight sample boxes marked with a code was compared with the weight periodically measured for the same crates. The CTR and SUN samples were withered until reaching the 30% WL each.