dataesr/scientific-paragraphs-categorization huggingface dataset:
Typically, stochastic gradient descent, when initiated from generic initial conditions (with small weights), finds a set of weights W Ã not far from the initial condition, which has a small loss. Surprisingly, in this large-dimensional space, the set of weights with small loss is not sparse, as one could have expected from our experience of optimizing in large dimensions. Once the network has been trained, typically using a number N of weights that is of the same order as the number of points in the database, one must study its generalization properties. From a statistics' perspective, what has been done in the training phase is fitting a complicated N-dimensional function using P datapoints. This is possible because N is large, but one should expect to be in a regime of overfitting, and therefore a poor generalization.
In the next section I shall focus on this last point, argue about the relevance of structured data and describe the challenge it poses to statistical physics. The new challenge of spin glass theory: structured disorder Data are highly structured, and a major objective is to develop mathematical models for the datasets on which neural networks are trained. Most theoretical results on neural networks do not model the structure of the training data. Statistical learning theory  [40, 55]  usually provides bounds that hold in the worst case, but are far from describing typical properties seen in experiments. On the other hand, traditional statistical physics approaches use a setup where inputs are either drawn component-wise i. i. d. from some probability distribution, or are Gaussian-distributed  [13, 47].
Labels are either random or given by some random, but fixed function of the input. Despite providing valuable insights, these approaches ignore key structural properties of real-world datasets. In recent years several aspects of data structure have been explored, and the first ensembles of structured data have started to be developed. The challenge is of course to create ensembles which contain some of the essential structure, but are at the same time simple enough to be analyzed. I will mention here three categories of data properties which are being studied: effective dimensionality, correlations, and combinatorial/hierarchical structure.
Effective dimension Let us consider perhaps the simplest canonical problem of supervised machine learning: classifying the handwritten digits in the MNIST database using a neural network  [29]. The input patterns are images with 28 Â 28 pixels, so a priori we work in the high-dimensional space R 784. However, the inputs that may be interpreted as handwritten digits, and hence constitute the ''world'' of our problem, span but a lower-dimensional manifold within R 784. Although this manifold is not easily defined, its dimension can be estimated based on the distance between neighboring points in the dataset  [8, 19, 31, 53]. In fact, if we consider P independent datapoints in a D-dimensional space, we expect that the distance between nearest neighbors scales like P À1=D. Analyzing the MNIST data base, one finds the effective dimension to be around D % 15, much smaller than N ¼ 784.
The ''perceptual submanifold'' associated with each digit also has an effective dimension, ranging from % 7 for the digit 1 to % 13 for the digit 8  [23]. Therefore, the task of identifying a handwritten digit consists in finding these ten perceptual submanifolds, embedded in the 15-dimensional ''world'' manifold of handwritten digits. Of course, the problem is that these manifolds are nonlinear, folded, and it is hard to find them (see  [14]  for algorithmic approaches). The same phenomenon of reduction in effective dimension is found in other datasets. For instance, images in CIFAR10 are defined in dimension N ¼ 1024, but have an effective dimension D % 35. In most machine learning problems, the effective ''world'' on which we train our networks has an effective dimension D ( N (in fact, a good practice would be to train the networks so that they can identify when they see an input which is far from the world in which they were trained, and refuse to give an answer in such cases).
Correlations From the database, one can construct the empirical pair correlation C ij between two components of the input C ij ¼ 1 P X P l¼1 n l i n l j ð24Þ as well as higher-order correlations. (Here we assume that we use centered data, in which the empirical mean of n i has been subtracted. ) A distinguishing property of practical datasets is that correlations are highly structured, and actually some of this structure is already seen at the level of the pair correlation. For instance, if one diagonalizes the matrix of pair correlations, which is of Wishart type, one finds a spectrum of eigenvalues which differs notably from the Marcenko Pastur one that would be obtained if the components n l i were distributed independently and identically. Instead, one typically gets a power-law distribution of the large eigenvalues, and it has been argued that this power-law scaling is actually related to the power-law decay of the loss with respect to either N or P, found in large language models  [33].
Clearly the simplest approach would be to sample the conditional probability distribution: take the previous k tokens before being interrupted, and look in the database at sentences which have exactly this sequence of k tokens, and compute from this database the most probable next token. This approach was started very early on, by Shannon himself. But clearly it is limited to small values of k: beyond k of order a dozen, one does not have the statistics to infer the conditional probability. But it turns out that key tokens, which are crucial for guessing the next one, can be found much earlier in the text. Take, for instance, this sentence written above: ''Instead, one gets typically a power law distribution of the large eigenvalues, and it has been argued that this power-law scaling is actually related to the power-law decay of the''.
Combinatorial and hierarchical structure A third distinctive structure of datasets used in practice is its combinatorial nature. Imagine, for instance, a photo of a lecture hall: it is composed by a group of students, each sitting at his desk. Then each student is ''composed'' of head, chest, arms, and each head is ''composed'' of eyes, nose, mouth, hair, and the eyes are ''composed'' by pigmented epithemial cells, etc. This is actually typical, and most of the images that we want to analyze have this type of combinatorial structure with a hierarchy of features and subfeatures related to the scale at which one looks. This structure is also related to the decoding that happens when learning from images with a deep network: one typically finds that the first layers of the network decode small scales elements like edges, and going further into the network one gradually identifies larger scale properties, until in the final layers one is able to decide the content of an image.
A fascinating new challenge of spin glass theory is to develop new ensembles of correlated disorder, including some of the most relevant ingredients that are found in real databases, like long-range correlations, hierarchy, combinatorial structures and effective dimensions, while being able to keep some analytic control of the problem. Open Access This article is licensed under a Creative Commons Attribution 4. 0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material.
Acknowledgments: We thank all researchers participating in this Special Issue, specially, those who have contributed to the  Experimentation Work Package  in the  Erasmus+ K3  project " Online Pedagogical Resources for European Universities" (OpenU ). We also thank all participants who contributed to the  OpenU International Conference , "Experimenting with  Online Pedagogical Resources in European Universities (OpenU) , held on 13-14 October 2022 in Paris, France.
BACKGROUND: Enrichment strategies from clinical trials for progressive systemic sclerosisassociated interstitial lung disease (SSc-ILD) have not been tested in a real-life cohort. RESEARCH QUESTION: Do enrichment strategies for progressive ILD impact efficacy, representativeness, and feasibility in patients with SSc-ILD from the European Scleroderma Trials and Research (EUSTAR) database? STUDY DESIGN AND METHODS: We applied the inclusion criteria of major recent SSc-ILD trials (Study of the Efficacy and Safety of Tocilizumab in Participants With Systemic Sclerosis [focuSSced], Scleroderma Lung Study II [SLS II], and Safety and Efficacy of Nintedanib in Systemic Sclerosis [SENSCIS] ) and assessed progressive ILD, which was defined as absolute change in FVC and as significant progression (FVC decline $10%).
Data were compared with all patients and with patients who did not fulfill any inclusion criteria. RESULTS: In total, 2,258 patients with SSc-ILD were included: 31. 2% of the patients met SENSCIS criteria; 5. 8% of the patients met SLS II criteria; 1. 6% of the patients met focuSSced criteria, and 67. 7% (1,529) of the patients did not meet any criteria. In the first 12 AE 3 months, the absolute FVC decline in all patients and in patients who fulfilled criteria from SENSCIS was -0. 1%, in patients who fulfilled criteria from focuSSced was -3. 7%, and in patients who fulfilled criteria from SLS II was 2. 3%, with accompanying more progressors in focuSSced. The patient populations that fulfilled the different study inclusion criteria significantly differed in various clinical parameters. In the second 12-month period, SENSCIS-enriched patients had a further absolute FVC% decline as described for the total cohort.
Study Design and Patient Population This was a cohort study with data that were retrieved for post hoc analyses from the multicenter, multinational EUSTAR database, for which all data are collected in a prospective and standardized manner. 10, 17 tients with the following factors at baseline were included in the present study: registered since 2010 in the EUSTAR database (start of the online version), aged $ 18 years, definite SSc,  18  presence of ILD by high-resolution CT scan, recorded disease duration, and measurements of FVC and diffusing capacity of the lungs for carbon monoxide (DLCO). 10, 18.
Application of Enrichment Strategies To determine the feasibility of cohort enrichment for progressive ILD, we assessed the number of available patients by applying single enrichment factors to the baseline visits. 10 Factors for cohort enrichment were chosen based on their published evidence  9, 10  and their availability in the EUSTAR database: antitopoisomerase I antibody, anticentromere antibody, diffuse cutaneous SSc, age > 57 years, FVC < 70% predicted, DLCO < 60% predicted, male sex, disease duration (< 72 months and < 60 months), signs of inflammation (increased levels of C-reactive protein), presence of more severe respiratory symptoms (New York Heart Association functional class 3 and 4), and presence of gastroesophageal reflux disease. In the second step, we combined single enrichment factors as previously used as inclusion criteria in three different clinical trials: focuSSced,  8   Progressive ILD and Long-Term Impact We assessed ILD progression differentially in the enriched populations both by measuring the absolute change between baseline and follow up of FVC% predicted and DLCO% predicted and categorically by events.
Progressive ILD Over the Second Year For the interpretation of open-label extension studies of 1year randomized clinical trials (RCT), it is important to understand the disease course of the recruited study populations in the second year. In the present study, patients who were SENSCIS-enriched had a further absolute FVC% predicted decline in the same range, as described for the whole cohort. In contrast, patients who fulfilled the focuSSced and SLS II inclusion criteria showed an improvement of lung function in the second period (e-Table  2 ; Fig  3A ). Compared with patients who were not eligible for any study, these changes were not significantly different. Notably, in this real-life observational database, 80% of all patients who fulfilled the focuSSced trial criteria were started on immunosuppressive treatment in this second 12-month AE 3-month period compared with only 35% and 28% in the SLS II and SENSCIS populations, respectively (Table  2 ).
Discussion The ability to enrich SSc-ILD clinical trial cohorts with progressive patients is important because it allows detection of clinically meaningful differences between the treated and the placebo group, which is difficult when the placebo group itself is not progressing in a clinically meaningful way. In our study that was derived from the EUSTAR real-life database, selection of patients based on single risk factors did not increase the proportion of patients with progressive ILD. Noteworthy, selection of patients with SSc-ILD with FVC < 70% at baseline resulted in the lowest proportion of patients with progressive ILD over 1 year. ] [22]  However, our observation supports the findings of a a Defined as any of these treatments at any time registered over the follow-up period. chestjournal.
25 other aim of our study was to provide information on the disease course of the enriched populations beyond the initial 12-month study period of clinical trials. In the second 12 It is also interesting to note that compared with the disease course of all patients with SSc-ILD, none of the combined enrichment strategies had a significant effect on event-free outcome over 5 years, including the initially successful enrichment strategy of the focuSSced study. Thus, recruiting an initially rapid progressive study population does not necessarily translate into long-term organ damage and worse outcome. However, longer follow up than the 5 years in this study is needed to assess long-term survival and end-organ disease with the different enrichment strategies. This study has several strengths. Our analysis is based on the largest multinational database of clinically relevant real-world SSc patient data.
Funding/Support This study was partially funded by  Boehringer Ingelheim International GmbH (BI) ,  Ingelheim am Rhein, Germany. The authors meet criteria for authorship as recommended by the  International Committee of Medical Journal Editors (ICMJE). The authors did not receive payment for the development of the manuscript. Editorial support was provided by  Callan L. Attwell, BSc, BN , of  AMICULUM  Limited, which was contracted and funded by  BI. Editorial support and formatting assistance, in part, were provided by  Hannah Cook, PhD , of  MediTech Media, UK , and was contracted and funded by  BI. This analysis, using the EUSTAR database, was funded by  BI.
5ENSCIS,5and (3) SLS II7 (Fig 1). Because not all variables were available in the EUSTAR database, simplified criteria were applied: (1) focuSSced: diffuse cutaneous SSc, disease onset # 60 months, modified Rodnan skin score 10 to 35, C-reactive protein > 6 mg/L, blood sedimentation rate $ 28 mm/h, no potentially disease-modifying (DMARD) treatment, FVC% predicted > 55%, and DLCO% predicted > 45%. Presence of active disease was defined as presence of disease duration < 18 months and/or increasing modified Rodnan skin score prior baseline registration and/or presence of tendon friction rubs. 8 (2) SENSCIS: diffuse or limited cutaneous SSc, FVC% predicted > 40%, DLCO% predicted 30% to 90%, and disease duration < 7 years. DMARD therapy with mycophenolate mofetil and methotrexate was allowed. 5 (3) SLS II: diffuse or limited cutaneous SSc, active alveolitis on high-resolution CT scan (defined as presence of any ground-glass opacity), age < 75 years, disease duration < 7 years, FVC% predicted 45% to 80%, and New York Heart Association grade $ 2 (instead instead od Mahlers index/modified Medical Research Council Dyspnea Scale > 2 in SLS II).
Discussion Experts from various disciplines at Fennovoima have communicated and presented the supply chain network visualisation in various events and meetings, ranging from top management meetings to nuclear safety committees and regulator inspections. The visualisation has been used to communicate the overall structure of Hanhikivi-1 project network, or to illustrate the reasoning behind supply chain assurance decisions. Overall, the reception and feedback has been very positive. The visualisation has been described as providing a good overview, or as offering a holistic picture of the supply chain in a simple way. What does the supply chain network visualisation tell that non-visual data cannot? The application of the visualisation indicated that there are clear benefits to using visualised as opposed to tabular or textual descriptions of the supply chain.
Numerical examples In this section, we present the results obtained from the computational homogenisation simulations for binary media having the three different morphologies shown in Figure  4 : Morphology 1 is a two-layered problem with a rectangular inclusion; Morphology 2 is a binary medium with a circular inclusion; and Morphology 3 is a binary medium with an L-shaped inclusion. All of the computations were carried out using MATLAB R2018a on a DELL desktop with the configuration: Intel(R) Core(TM) i7-6700 CPU3. 40GHz and RAM 16. 0 GB. = [ 3 8 , 5 8 ] × [0, 1], Ω 2 = [0, 1] × [0, 1]\Ω 1. (b) Morphology 2 Ω 1 = {(x, y)|(x -0. 5) 2 + (y -0. 5) 2 < 1 4π }, Ω 2 = [0, 1] × [0, 1]\Ω 1. (c) Morphology 3 Ω 1 = [ 1 2 , 3 4 ] × [0, 1 2 ] ∪ [ 1 4 , 3 4 ] × [ 1 2 , 3 4 ] , Ω 2 = [0, 1] × [0, 1]\Ω 1.
Phylogenetic inference and protein structure analysis Using E. coli HflX as a query, homologous proteins from photosynthetic organisms were identified by BLAST search using public data at the National Center for Biotechnology Information (NCBI) and JGI. The representative bacterial HflX/HflXr and animal homologs were previously identified  (Suwastika et al. , 2014; Koller et al. , 2022). Multiple-sequence alignments were performed using MAFFT v7. 40262 with option -auto  (Katoh et al. , 2019). Phylogenetic reconstructions were created using maximum likelihood with the IQ-TREE web server version 1. 6. 12 using default settings, with LG +F+ I + G4 automatically selected as the best fit evolutionary model based on BIC values by ModelFinder  (Trifinopoulos et al. , 2016). Branch support was tested using two methods: ultrafast bootstrap approximation using 1000 bootstraps, and the non-parametric Shimodaira-Hasegawa-like approximate likelihood-ratio test (aLRT).
Data analysis Data analysis and visualisation was conducted in R using scripts previously described  (Romand et al. , 2022)  with minor modifications. Data generated from ARADEEPOPSIS was analysed using the script provided in https://github. com/cecile-lecampion/Analyse_croissance. For each experiment, plates were considered as independent replicates and individual plants were considered as biological replicates. The experiments were performed at least twice and similar results were obtained. The significance of differences in categorical data (cotyledon death) was calculated using the proportion test as previously described  (Romand et al. , 2022).
The primary endpoint was overall survival, assessed in all randomly assigned patients who received at least one dose of study drug, regardless of which study drug they received, by intention to treat. Secondary endpoints were disease-free survival, site of treatment failure, operative complications, toxicity, pathological response (complete [ypT0N0] and major [tumour regression grade 1 and 2]), margin-free resection (R0), and health-related quality of life. Toxicity and safety data were analysed in the safety population, defined as patients who took at least one dose of study drug, according to treatment actually received. The initial power calculation was based on superiority of trimodality therapy (n=366 patients); it was adjusted after FLOT became an option to a non-inferiority design with a margin of 5% for perioperative chemotherapy (n=540).
This study is registered with ClinicalTrials. gov, NCT01726452. Findings Between Jan 24, 2013, and Dec 23, 2020, 377 patients were randomly assigned, of whom 362 were included in the intention-to treat population (327 [90%] male and 360 [99%] White): 184 in the perioperative chemotherapy group and 178 in the trimodality therapy group. The trial closed prematurely in December, 2020, after the second interim futility analysis (143 deaths), on the basis of similar survival metrics and the impact of the COVID-19 pandemic. At a median follow-up of 38•8 months (IQR 16⋅3-55⋅1), median overall survival was 48•0 months (95% CI 33⋅6-64⋅8) in the perioperative chemotherapy group and 49•2 months (34⋅8-74⋅4) in the trimodality therapy group (3-year overall survival 55% [95% CI 47-62] vs 57% [49-64]; hazard ratio 1⋅03 [95% CI 0⋅77-1⋅38]; log-rank p=0⋅82).
Both trials showed a clear superiority of each approach compared with surgery alone. The only phase 3 trial of more than 100 patients (POET) comparing trimodality therapy and perioperative chemotherapy (n=119) did not complete and used a different regimen to that used in MAGIC and a lower radiation dose (30 Gy) than that used in CROSS (41•4 Gy). With the CROSS trial generating great interest, Neo-AEGIS set out to address this gap in knowledge, seeking out a gold standard, if one existed, and to provide data to inform international practice, which varies widely. Secondary endpoints were carefully chosen to inform clinician and patient choices if conclusive differences were not apparent in overall survival. The unique elements in trial design included the homogeneity of the study population, mandated staging with CT-PET, radiotherapy quality assurance, and reporting of operative complications using strictly defined criteria.
It was concluded by the data safety and monitoring board that more patients and longer follow-up would not changes survival metrics. Survival outcomes with trimodality therapy were similar to those in the CROSS trial; however, there was no evidence of superiority over perioperative chemotherapy. Trimodality therapy, as expected, was superior to perioperative chemotherapy in producing complete or major pathological responses and negative surgical margins. However, this did not translate to survival advantage, suggesting that radical en-bloc surgery was a key equalising factor. Perioperative chemotherapy did not reduce systemic failure rates compared with trimodality therapy, suggesting a modest effect on micrometastatic disease of the predominant modified MAGIC regimen, with no conclusions possible for FLOT.
Implications of all the available evidence Neo-AEGIS set out to compare the trimodality regimen used in the CROSS trial with optimum perioperative chemotherapy. The similar survival metrics that was relevant to trial termination suggest that clinical equipoise between such approaches, specifically trimodality therapy and the modified MAGIC regimen or FLOT, is a reasonable conclusion. The limited FLOT treatment inclusion cannot be interpreted independently, hence the ESOPEC trial will be informative when published. Moreover, data relating to differences in toxicity, tolerance, and HRQOL, might help to inform clinicians and patients in treatment choices. In a dynamic evolving field, as we enter an era in which immunotherapy will be integrated with both these approaches, Neo-AEGIS provides a rich resource of randomised data.