dataesr/scientific-paragraphs-categorization huggingface dataset:
Training procedure of U-net and V-net During the training process, we gave batches composed of randomly drawn sub-volumes (coronal view) as inputs to the Unet and the V-net as well as the corresponding normalized subvolumes' coordinates to the CPPN. The targeted output was the corresponding label sub-volume. The optimization of the network's parameters was performed following the cross-entropy loss during the first 5000 steps to ensure a correct optimization start. It was then performed in accordance with the softDice loss (L) (4) in order to reach a better local minimum  [29]. Dice was eventually retained. All codes were implemented on Python with Tensorflow  [42]  and all networks were trained using NVIDIA Tesla V100 GPU. L(Y, Ŷ) = 1 -2 N i=1 y i ŷi δ+ y i +ŷ i.
Parameters initialization and optimization-parameters setting Before starting the optimization, the network's weights were initialized in accordance with a uniform distribution with Xavier initializer  [43]  and the networks' bias were initialized at 0. 01. The input sub-volumes' size were 1 × 128 × 128 in U-net cases and 64 × 128 × 128 in the V-net cases. Concerning Moeskops et al. [17]  network, input patch of size 75×75, 51×51 and 25×25 were used in the case of the MRI dataset and input patch of size 127 × 127, 101 × 101 and 75 × 75 were used in the case of the US dataset. The optimization was performed using Adam optimizer  [44]  with β 1 = 0. 9, β 2 = 0. 999 and = 10 -8. The learning rates were set to: 10 -4 for the 10000 first iterations, 2 × 10 -5 for the 10000 following iterations and 5 × 10 -6 until the end of training.
Four-fold cross validation To show that the distribution of the volumes between the training set, the validation set and the test set (Table. 1) did not bias our results, a cross validation was performed. This experiment was realized with the four-fold cross validation dataset described in Table. 2. It was performed such that each of the 4 sets was, in turn, considered as the test set and the 3 remaining sets as the training set. In each configuration, the validation set remained unchanged and the U-net was optimized five times. The mean Dice and the mean MAD obtained for each fold are given in Table. 3.
Influence of the number of training volumes Training a CNN for this application efficiently also requires understanding and determining the accurate number of images to be used in this process. That is why a 24 layers (Fig. 5. b) U-net with different training-set size was optimized. For this experiment, the dataset defined in Table. 1 was used, the validation and the test sets' sizes remained the same while several sizes were used for the training sets: 1, 3, 5, 7, 9, 11 and 13. For each    Results showed that mean Dices over the training and test sets decreased and increased respectively as the training set's size increased. Both values seem to evolve slowly with the standard deviation being very low for training-set sizes reaching 7-or-more volumes. To determine whether the improvement was still significant, the results obtained for a training-set size of 13 were used as a baseline.
Comparison of the U-net, the V-net and Moeskops et al. [17] network To evaluate the benefits of FCNs and to determine whether a 2D FCN or a 3D FCN architecture was more suitable for this problem, the segmentation time and the accuracy of the U-net, the V-net and  [17]  were compared. 3. 7. 1. Quantitative results US dataset. According to the results presented in Table. 5, the V-net was significantly better than the U-net for Dice (p = 7. 6 × 10 -5 ) and also better fr MAD but not significantly (p = 0. 7). It was also significantly better than the network proposed by  [17]  for Dice (p = 8. 96 × 10 -5 ) and MAD (p = 3. 36 × 10 -7 ). Considering the accuracy of the best networks (Table. 6), the V-net was more accurate than the U-net for the Dice and as accurate for MAD. Both FCNs were extremely more accurate than  [17]  according to all metrics.
MRI dataset. As reported by the results presented in Table. 5, the V-net was more accurate than the U-net but not significantly. On the other hand, V-net was significantly better than the network proposed by  [17]  according to Dice (p = 5. 34 × 10 -7 ) and MAD (p = 6. 79×10 -6 ). The Dice and the MAD obtained by the  [17]  network were lower than the one reported in their article on the same database (0. 86±0. 05 and 0. 52±0. 25 mm respectively). The accuracy obtained by applying the methodology described in  [17] : training with 7 classes (background, CVS, basal ganglia, white matter, brain stem, cortical grey matter, cerebellum) and application of a brain mask is reported in the Table. 9. A similar value was then obtained for Dice (0. 844 ± 0. 053), the value obtained for MAD remained more important (0.
The CPPN's influence In this experiment, the number of layers of the U-net and the V-net were incrementally increased to determine if the CPPN brings information about the CVS location. This experiments' quantitative results on the US dataset are given in Table. 10 and the evolution of MAD and Dice on US dataset as a function of the number of layers of U-net and V-net are represented in Fig. 11. A similar behaviour was observed in the 2D and 3D cases. As it can be seen in Fig. 11 , with and without CPPN, Dice and MAD improved with the number of layers, although the improvment shrank as the number of layers increased. Nevertheless, it can be observed that the accuracy obtained with CPPN is better than without CPPN when the number of layers is low and equivalent when the number of layers is high.
This result is confirmed by the quantitative values of Dice and MAD, and the p-values given in the Table. 10. In the case of U-net, Dice and MAD were significantly (p < 0. 05) better with CPPN for the 9, 14 and 19-layers U-net. In the case of V-net, Dice and MAD were significantly better with CPPN for the 7-layers V-net and 15-layers V-net. The results also show that the networks without CPPN needed fewer layers to catch up with the performances of the networks with CPPN in the 3D case. The quantitative results obtained on the MRI dataset are in accordance with those obtained on the US dataset. Accuracy is    given for a 24-layer U-net and a 9-layer U-net with and without CPPN in Table. 10. Dice and MAD were better in all the cases where the CPPN was used and were significantly better in the case of the 9-layer U-net.
Four-fold cross validation The results given in Table. 3 show that the maximum differences between two test sets for Dice and MAD are 0. 031 and 0. 05 mm respectively. These differences are small and the accuracy obtained in the worst cases is good, it is 0. 797±0. 005 for Dice and 0. 54 mm for MAD. This characterise the fact that each training set contained enough variability in order to enable the U-net to generalize its learning to an unknown data set. Based on these results, we conclude that the database used to compare the U-net and the V-net did not incorporate biases, such as very similar patients in the training set and the test set, which could have led to an overestimation of the results.
In particular the network seems to have difficulties to determine precisely the edges of the CVS. Similarly to what was shown on the MRI dataset (Table. 9), training the networks with more classes and using a brain mask could reduce the number of false positives and improve the accuracy on the US dataset. Firstly, it could enable the network to learn to recognize other brain structures that may look similar to the CVS. Secondly, it could eliminate obvious false positives based on their location. However, these solutions are not suitable for 3D US because there are no manual annotations for structures other than CVS and the entire brain is not visible on the images. Considering the methodological metrics, the V-net was significantly better than the U-net for Dice and better for MAD. Moreover, the V-net has almost twice as few parameters as the U-net (31 × 10 6 parameters for the U-net and 16 × 10 6 parameters for the V-net).
Metrics Normal CVS Dilated CVS U-net V-net U-net V-net Dice 0. 04 0. 019 0. 005 0. 011 MAD (mm) 0. 06 0. 04 0 0. 02 ∆V a (cm 3 ) 0. 15 0. 15 0. 04 -0. 05 case of normal ventricles, the V-net is globally closer to the IOV than the U-net with differences of 0. 019, 0. 04 mm and 0. 15 cm 3 considering Dice, MAD and ∆V a. The accuracy of these segmentations is therefore also excellent but it is still possible to improve the accuracy in this case. Performing data augmentation, such as horizontal symmetries, as proposed in  [45]  could be a way to bridge this gap. From a clinical point of view, the important question is whether these architectures enable the identification of a dilated ventricle and the monitoring of CVS dilation. The average volumes of normal and dilated ventricles in our database (Table.
The influence of the CPPN The CPPN was used to generate patterns (Fig. 13 ) which could provide location information to the networks. The quantitative results given in the Table. 10 show that the accuracy of U-net and V-net was better with CPPN when networks have few layers. This can be explained by the fact that the patterns generated by the CPPN enabled the networks to locate the characteristics in the common space defined for the database. Thus, when the networks were not able to use much contextual information, a decrease in the number of false positives, due to CVS-looking-like structures, was expected. This assumption was confirmed by Fig. 12. a: many false positives were avoided when using the CPPN. Examples of false-positives from the best 7-layer V-net segmentation are given in Fig.
The second aim of this paper was to enable CNNs to encode CVS location since this information was used as an important features in most methods dedicated to the segmentation of preterm-neonates cerebral structures. Our experiments showed that the use of a CPPN enables the learning of a pattern dictionary encoding CVS location from the training dataset which improves the networks accuracy when they have few layers. Further investigations must be perform to evaluate the benefits of the CPPN for brain multi-structural segmentation. In addition, other CPPN inputs and basic mathematical functions could be tried to build a CPPN so as to better encode the location information into the patterns. To the best of our knowledge, this study is the first to compare the accuracy of 2D and 3D CNNs for the segmentation of preterm neonates' CVS in 3D US data and to distinguish between normal and dilated ventricles in the analysis of the results.
Acknowledgments This work was performed within the framework of the  LABEX CELYA  ( ANR-10-LABX-0060 ) and  PRIMES  ( ANR-11-LABX-0063 ) of  Universite de Lyon , within the program " Investissements d'Avenir"  ( ANR-11-IDEX-0007 ) operated by the  French National Research Agency (ANR). We would like to thank  Dr. Borhane Slama , the innovation and clinical research commission president of  CH Avignon , and the  CH Avignon direction  for their support to this research.
Together, these disease characteristics result in substantial morbidity throughout life and predispose to high mortality in childhood  (8). A therapeutic strategy involves the application of a thyroid hormone analogue that can enter cells independent of MCT8 and stimulate the endogenous thyroid hormone receptor. Preclinical studies showed that the T3 analogue Triac (3,3′,5-tri-iodothyroacetic acid; also known as tiratricol), whose cellular entry is not dependent on MCT8, fully rescued the brain phenotype in animal models of MCT8 deficiency when administered directly after birth  (1, 9, 10). The ongoing Triac Trial II, a phase IIb clinical trial (NCT02396459) in young infants with MCT8 deficiency, will evaluate the effects of Triac on neurodevelopment once treatment is initiated early in life.
Study Design and Participants In this international pragmatic real-life cohort, data were collected retrospectively in patients with MCT8 deficiency, within our consortium of 33 hospitals in 18 countries, who have been treated with the T3 analogue Triac on an offlabel use basis. Eligible subjects were those with MCT8 deficiency (confirmed by the presence of a pathogenic mutation in the SLC16A2 gene), irrespective of age or comorbidities, who were treated with Triac on an off-label use basis following our previously established dose escalation protocol  (11) , and had their biochemical parameters measured in the central laboratory of the Erasmus MC, The Netherlands. This approach allowed for a uniform strategy of Triac dosing and monitoring of effects. The cohort consisted of (1) patients who finished the Triac Trial I protocol and continued on Triac treatment on an off-label use basis; and (2) additional patients who were identified after the recruitment phase of Triac Trial I or could not be enrolled in the Triac Trial I due to the absence of study sites in their country.
Statistical Analyses Analysis on the primary outcome was based on the full analysis dataset, which included all patients who received at least 1 dose of Triac and who had at least 1 follow-up visit after the baseline assessment. Analyses on all prespecified secondary outcomes were done in all subjects who had a treatment duration that exceeded the mean time of dose escalation (defined as time from baseline measurement to the first follow-up visit on optimal dose), to ensure that treatment effects on secondary outcomes could be reliably assessed. Complementary analyses of secondary outcomes were performed on the full analysis dataset. For all endpoints, P values and 95% CIs were calculated for the mean change between baseline and last available follow-up visit using paired t tests. Serum TSH and CK concentrations were first log-transformed to normalize the distribution.
With the assumption that omission of data occurred randomly, and given the broad age range of the participants and small group size, pairwise deletion was used to adjust when the baseline data point was missing; for missing data that were captured throughout the treatment period, the last available measurement was used (if available). All statistical analyses were performed using GraphPad Prism version 5 (GraphPad Software Inc. , San Diego, California, USA). Two-sided P values of less than 0. 05 were considered to denote statistical significance.
2166; Fig. 2E  and 2F  and  Table 2]. Complementary analyses of all secondary endpoints on the full analysis dataset (all patients who received at least 1 dose of Triac and had at least 1 follow-up visit after the baseline assessment) showed similar results (Supplementary Table  3   (12) ). No severe adverse events related to Triac were reported during follow-up. Transient signs of increased thyrotoxicosis (increased irritability and anxiety, reduced sleep, increase in blood pressure, and tachycardia) were reported in a small subset of patients (5 of 67 patients; Supplementary Table  4 ). No hospitalizations related to Triac were reported. During the total treatment period, 3 patients died: a 13-year-old patient died due to sudden death, a 71-year-old patient died after aspiration and a 14-year-old patient died of unknown cause.
Patients have potentially also received care outside of their designated center, which could hamper reporting of adverse events. However, with trial patients not necessarily representing patients who get the drug prescribed, real-world data are of great additive value to determine effectiveness for any new therapy, as recently re-emphasized  (17, 18). Therefore, the observation that Triac is effective in a real-life setting may further substantiate its beneficial effects. Third, this study precludes determining the optimal Triac dose for MCT8 deficiency for both the peripheral and neurocognitive phenotype as data on neurodevelopmental outcomes had not been uniformly collected. The present international phase IIb clinical trial (NCT02396459) will investigate the effects of Triac on neurodevelopment as well as on the peripheral phenotype in patients with MCT8 deficiency who are younger than 2.
Acknowledgments Our study was funded by  Eurostars  (project number  E11337 ; to W. E. V. ), the  Sherman Foundation  (to W. E. V. ). The centers in Rotterdam, Bucharest, Paris, and Angers are part of the European Reference Network on rare endocrine conditions (Endo ERN). The center in Rome is an HCP member of the European Reference Network for Rare Neurological Disorders (ERN  RND ) and the center in Naples is part of the European Reference Network ITHACA (ERN ITHACA). The center in Cambridge (UK) is supported by the  Wellcome Trust  ( Investigator Award   210755/Z/18/Z  to K. C. ) and the  National Institute of Health Research Cambridge Biomedical Research Centre. We thank the patients and their families and caregivers who participated in this study. Financial Support: This work was supported by a grant from  Eurostars  (project number  ESTAR E11337 ) (to W.
2166 All outcomes were assessed in all patients who received Triac treatment longer than the mean time to optimal dose (5. 0 months; N = 64). Data are mean. Body weight-for-age Z-scores were calculated using TNO growth calculator and heart rate-for-age Z-scores were calculated using the Boston Z-score calculator. Abbreviations: bpm, beats per minute; T3, triiodothyronine; T4, thyroxine; TSH, thyroid-stimulating hormone. *TSH and creatine kinase concentrations were log-transformed to ensure a normal distribution before paired t tests were done (nontransformed means [SD] and mean changes [95% CI] are presented for the sake of interpretability).
( 13 ) Specifically, we use Long Short-Term Memory (LSTM) networks for both functions h 1 and h 2. The input to the networks at step k is the concatenated observation y k and the network input u k , to which is also appended the timeindependent one-hot vector c. The subject-specific parameters θ s depend only on the subject identity encoded in the one-hot vector c. They are represented directly in the matrices of means M s and log-variances V s. For a specific subject the relevant values are extracted through the product with the one-hot vector c, (µ s , log σ 2 s ) = h 3 (c) = (M s • c, V s • c) (14) q(θ s |c) = N (θ s |µ s , diag(σ 2 s )) (15) Optimization. The optimization target is the negative dataset ELBO, L dataset = n sub i=1 n j=1 L ij , ( 16 ) where L ij is the ELBO associated with a subject i and region j, defined by Eqs.
Additive Gaussian noise η is included in the equations, with standard deviation β. To generate the synthetic dataset, we use the structural connectome matrices of human subjects as described above. We simulate eight subjects, with increasing coupling coefficient G spaced linearly between 0 and 0. 7. The intrinsic frequency f i of all nodes is sampled randomly from uniform distribution on [0. 03, 0. 07] Hz. The bifurcation parameter a is sampled randomly from uniform distribution  [-1, 1]. The initial conditions of the system for all subjects and both variables are chosen randomly from normal distribution N (0, 0. 3), the system is then simulated for 205 s. First 25 s are then discarded to avoid the influence of the initial conditions, leaving 180 s of data. The system is simulated with Euler method with time step ∆ t = 0.
Results Evaluation workflow. We test the proposed method on two synthetic data sets, where the data are generated by models commonly used in whole-brain modeling. First is the Hopf bifurcation model  (Deco et al. , 2017) , shown on Fig. 2. That is a two-equation neural mass model, where depending on the value of the bifurcation parameter a i the dynamics is either noise-driven around a stable fixed point (for a i < 0) or oscillatory with frequency f i (for a i > 0). In the. (E) Inferred regional parameters for all regions (top left, example nodes highlighted in color) and inferred subject-specific parameter (bottom left, in gray among parameters for all subjects in the dataset). The span of the crosses/lines corresponds to two standard deviations of the inferred Gaussian distribution. In the bottom panel circles are added for visual aid due to the small standard deviations.
The inferred dynamics in state space of the three example nodes are on the right. The vector field is evaluated assuming zero network input and using the inferred region-and subject-specific parameters. (F) Inferred regional parameters colored by the ground truth values of the bifurcation parameter a i (top) and frequency f i (bottom). The bifurcation parameter correlates with inferred θ r 2 , while frequency correlates with θ r 1 , but only for regions in the oscillatory regime, i. e. where a i > 0. (G) Timeseries generated with the trained model and using the inferred parameters. Important features of the data are preserved both the level of single regions (amplitude, frequency) as well as on the network level (functional connectivity). synthetic data set, these two parameters are randomly varied across regions.
Trained model reproduces the features of regional timeseries. A crucial test of the trained model is an evaluation whether the generated data resemble those used for the initial training. This resemblance should not be understood as reproducing the timeseries exactly, since they depend on a specific noise instantiation, rather that the features we consider meaningful should be preserved. For both test cases, we evaluate the similarity of two features. For the Hopf model with its oscillatory dynamics we evaluate the cosine similarity of the spectra of the original and generated timeseries, and the difference between the variance of the timeseries, since the variance differs greatly between the nodes in oscillatory and fixed-point regimes (Fig. 4B ). For the pMFM, we compare the timeseries based on the distribution in the 1D observation space (that is, taking the samples collapsed across time) using the Wasserstein distance (called also Earth mover's distance) of two distributions.
Second feature of pMFM timeseries is the log-scaled number of switches between the up-and down-state, capturing the temporal aspect of the switching dynamics (Fig. 4F ). We evaluate the measures for 50 different noise instantiations, leading to 50 different time series for each region, obtaining a distribution of goodness-of-fit metrics. The same metrics are evaluated also for three surrogates: First is the original computational model, run with different noise instantiations. That provides an optimistic estimate of what can be achieved in terms of goodnessof-fit, considering that the features will necessarily depend on the specific noise instantiation used in the initial simulations. Second surrogate is obtained by randomly reshuffling the original data between regions and subjects. Third surrogate is simply white noise with zero mean and variance equal to one (which, due to the data normalization, is equal to the mean and variance of the original data set taken across all subjects and regions).
Each line represents the 5 to 95 percentile range, with the dot representing the median. These results indicate that while the trained model can discover the existence of the network coupling, it systematically underestimates its strength. Given that in the pMFM the strength of the network input can shift a single neural mass from the monostable down-state to bistable regime and to monostable upstate, the underestimated coupling leads to the necessity of utilizing the regional parameter to compensate for the missing coupling (Fig. 3F ). Large perturbations of the connectome lead to reduced performance. To assess the influence of the inexact structural connectome on the goodness-of-fit, we have trained the model on the pMFM data set with perturbed connectomes. That is, instead of the original connectivity matrix W we have trained the model with W = W + A, where A is matrix with elements drawn from standard normal distribution, and > 0 is the perturbation magnitude.