dataesr/scientific-paragraphs-categorization huggingface dataset:
Diabetes Dataset Disease progression after 1 year was to be derived from 10 measures in 442 patients (Figure  9C ). In modeling for inference, only the body mass index (bmi) was deemed significant at p = 0. 01 among all input variables. This single measure, however, only accounted for 3% of explained disease progression in the population when modeling for prediction. Adding another predictive variable, s5, to the linear model with bmi enhanced the prediction accuracy to R 2 = 0. 42. Adding more and ultimately all input variables into the model led to small additional improvements in prediction performance (R 2 = 0. 46). In fact, s5 showed the highest positive b coefficient (at the beginning of the regularization path, where small sparsity constraint was imposed) but did not turn out as the final variable remaining in the model.
FEV Dataset Finally, the lung capacity captured as forced expiratory volume (FEV) was to be derived from four measures in 654 healthy individuals (Figure  9D ). All input variables easily reached the statistical significance threshold, but a predictive model built from the same data revealed that considering body height alone performed virtually on par with predictions based on all four coefficients (R 2 = 0. 74 versus R 2 = 0. 76). That is, age, sex/gender, and smoking habits all easily reached statistical significance but offered little added value for the purpose of prediction. In the An exploration of the consequences of applying a linear model to datasets that are known to contain non-linear data mechanisms of different types and degrees (cf. Figure  2F ). Certain non-linear effects are likely to influence measurements of various real biological systems.
DISCUSSION Exploring a battery of empirical simulations and several biomedical datasets, our study offers insight into the possibly asymmetric tendencies between seeking accurate predictions in new individuals and identifying statistically significant effects across individuals. 27 Prediction and inference share a first step of estimating linear model coefficients; however, the difference arose in what we decided to do next with the estimated model effect sizes. Charting a broad spectrum of data-analysis scenarios possible in everyday research, statistically significant relationships were not in all examined cases a guarantee to also achieve successful predictions when applying the model to other individuals. To restate, quantitative models varied between accounting for little or almost 100% variance in a target variable in fresh data points, even if a given model's effects were declared robust at the conventional significance level of p < 0.
This ongoing evolution in data-analysis practices can promote across-study and across-method confirmation. 32 oreover, ''importance'' in quantitative research has probably no uniform theoretical basis. 1, 28, 33 The notion of variable relevance can therefore take different forms and shapes even in the family of the canonical linear model. A statistical method that produces importance assessments still requires the informed judgment of the investigator as to how far the Across dataset simulations, we assessed the accuracy at which the predictively relevant coefficients of the Lasso or the significantly relevant coefficients (p < 0. 05) of the OLS captured the known true set of data-generating variables. Circles indicate equal performance and triangles diverging performance. Upward-pointing triangles indicate that the Lasso scored better at variable identification.
Their study and ours concur in trying to expose similarity and differences of significance and predictability through concrete data-analysis scenarios. However, Lo and colleagues focus on what they call three ''artificial examples'' as thought experiments. Our study instead revisited four real-world biomedical datasets and analyzed >100,000 concrete data simulations covering a breadth of practical data-analysis scenarios that scientists and analysts are facing every day. Finally, the previous study concludes by ''[encouraging] exploration away from significance-based methodologies and toward prediction-oriented ones. ''  39 In contrast, the core recommendation from our study is encouraging awareness for ''horses for courses'' rather than ''if all you have is a hammer, everything looks like a nail.
Resource Availability Lead Contact Associate Professor Danilo Bzdok, M. D. , Ph. D. Email: danilo. bzdok@ mcgill. ca. Department of Biomedical Engineering, McConnell Brain Imaging Center (BIC), Montreal Neurological Institute (MNI), Faculty of Medicine, McGill University, Montreal, CanadaCanada CIFAR AI Chair, Mila -Quebec Artificial Intelligence Institute, Montreal, Canada. Email: bzdokdan@mila. quebec Materials Availability Readers wishing to explore in real time >100,000 simulation results on the relationship between inference and prediction can use our WebApp (binderenabled): https://mybinder. org/v2/gh/banilo/inf_vs_pred_2018/r0_bioRxiv. Data and Code Availability Python was selected as the scientific computing engine. Capitalizing on its open-source ecosystem helps enhance replicability, reusability, and provenance tracking.
The statsmodels package was used to estimate OLS regression and corresponding p values (http://statsmodels. github. io). The scikit-learn package  41  provided efficient, unit-tested implementations for handling state-of-the-art machine-learning procedures (http://scikit-learn. org). All analysis scripts and necessary data that reproduce the results of the present study are readily accessible and open for reuse (https://github. com/ banilo/inf_vs_pred_2020). The repository also provides extended Jupyter notebooks with additional analyses and an interactive WebApp.
For display of simulation density, agreement between OLS and Lasso would show a strong diagonal. Instead, the collective findings make apparent that systematic disagreements between Lasso and OLS emerge in all cases and were most pronounced for committed false-positive errors. same model. 34, 42 We have performed least-squares regression to optimize the following objective: min b˛R p ( 1 n X n i = 1 ðy i À x 1 b 1 À x 2 b 2 À. À x p b p Þ 2 ) ; where n is the number of individuals in the dataset, p is the number of input variables x i (i. e. , independent, explanatory, or predictor variables) for each individual, and y is the outcome measure (i. e. , dependent or explained variable) that is to be expressed as a weighted sum of the variables x. Prior to model fitting, the data x were standardized by mean centering to zero and variance scaling to 1, across individuals.
After model fitting, the model was applied to other data points to predict unobserved outputs (or, conceptually, ''shipped'' to other laboratories for repeated application). Within the cross-validation framework (Figure  1. ), the selected model thus automatically chose the minimal subset of predictive variables that served to forecast the outcome. Following model estimation, the practical performance of the candidate predictive model was evaluated based on standard cross-validation. 25 An explicit empirical measure of model performance was thus obtained to answer the question how much the predictive algorithm could be expected to generalize to data that would be seen in the future. To this end, model parameters were estimated on some data points, while the emerging model was explicitly put to the test in some independent, unseen data points.
45 To obtain this quantity of prediction accuracy, the linear model was first built on a larger part of the dataset. Second, emerging candidate models were evaluated and selected on unused data to avoid an overoptimistic evaluation of goodness-of-fit. 25 The outof-sample prediction performance on the testing data samples thus quantified how likely the same pattern could be detected in future. In this way, the crossvalidation scheme quantified the out-of-sample performance as an estimate of a model's capacity to generalize to data points acquired in the future. This assumption enforced that not all input variables were expected to be relevant in the final linear model solution (sparsity property of the Lasso). Instead, we have accommodated the biasing of model parameter estimation toward zero (shrinkage property of the Lasso, cf.
Empirical Data Simulations Our synthetic data-simulation approach was motivated by the realization that formal guarantees for the expected model prediction performance have often been challenging to derive by mathematical theory for a particular number of available data points. 10, 45 Instead, we opted for empirical simulations as an alternative access route for studying the properties of the statistical methods in computational experiments. 43 In this way, we have directly confronted linear modeling for inference and for prediction in a series of synthesized datasets, columns of input variables X, each related or not related to the outcome y. Each dataset was generated from a set-up ground-truth model y = Xb + ˛, where b are fixed random coefficients, X is a data matrix containing n data points and p variables with random entries drawn from a standard Gaussian distribution N ðm = 0;s = 1Þ , and ε denotes the added Gaussian noise.
Each dataset was fed into the linear models with the aim of identifying significant input measures or identifying input measures most useful for accurate predictions on new observations (cf. above). To sharpen the distinction between explanatory and predictive modeling in general, we have systematically varied distinct aspects of the data-generating process as follows. (1) Samples-to-variables ratio. To investigate the relation between the number of data points n relative to the number of variables p, we gradually scaled the number of available observations. We covered the lower range between 50 and 100 data points in steps of 10, which probably well represents a majority of studies in biomedicine. Between 100 and 2,000 data points, we increased the sample size in steps of 100. Moreover, we considered the extreme cases 10,000 and 100,000 data points, which acknowledges recent large-scale datasets such as the UK Biobank.
The total number of input variables was kept constant to preclude secondary effects on the results due to changing model capacity. (2) Proportion of informative variables. To study how the fraction of relevant versus irrelevant variables modulates the inferential and predictive processes, we have varied the proportion of non-zero b coefficients in the ground-truth model used for generating X. We considered 14 proportions ranging from only 1 to all 40 input variables carrying information about the response y. Note that this breadth of experimental scenarios covers the number of considered variables in the analyses of real-world biomedical datasets (cf. below). (A) The body weight is to be derived from eight measures in 189 newborns. Three out of eight measures are statistically significantly associated with birth weight at p < 0.
(D) Lung capacity as quantified by forced expiratory volume (FEV) is to be derived from four measures in 654 healthy individuals. All measures easily exceeded the statistical significance threshold. However, a predictive model incorporating body height alone performed virtually on par with predictions based on all four coefficients (R 2 = 0. 74 versus R 2 = 0. 76). To expose the trade-off between parsimony and prediction performance, the circles (green) show the sets of Lasso coefficients at different sparsity levels. In sum, linear models can show all combinations of predictive versus not and significant versus not in biomedical data analysis. (3) Redundant versus unique sources of information: To elucidate how correlated input measures trade off against each other with respect to the outcome, we have introduced different degrees of pairwise covariation between the variable columns of X (i.
e. , collinearity). Ground-truth models also generated data from a multivariate Gaussian distribution that exposed 50% or 90% of common variation between the relevant variables, complementing datasets that contain only mutually independent variables (i. e. , 0% covariation). (  4 ) Signal-to-noise (SNR) ratio. To assess the role of nuisance variation in the data, such as induced by imperfect measurement techniques, we have systematically manipulated the noise ε in how the real model relates to the response y. The nuisance term-as well as the input data (cf. above)-was generated from N ðm = 0;s = 1Þ and multiplied by 0. 5, 1, 2, 5, 10, or 0 (i. e. , generating data without any noise). Given the definition of SNR as Var(true mean)/Var(noise), our simulations of observed datasets covered a range of SNR scenarios that are practically realistic.
(5) Model violations. Finally, we examined how inference and prediction behave when the linear model cannot fully capture how the data came about. We have therefore introduced alterations on 50% of the relevant variables in X that lead to associations between the input variables and the outcome inconsistent with the linear model specification. In addition to datasets with exclusively linear effects (i. e. , we can find the true model), deviations between the generating and fitting model were introduced by one of several data transformations: taking the absolute value, the natural logarithm, the exponential, the square root, the multiplicative inverse, or polynomial expansion of degree 2-5. The collection of simulated datasets has realized 113,400 different dataanalysis scenarios. For each case, we focused on the best (smallest) p value among all input variables in the model and the highest prediction performance of the overall model as quantified by the (out-of-sample) R 2 score.
Note that the smallest p values fall within the range below the smallest 64-bit floating point number around 2. 2 3 10 À308 that can be represented in Python. Such p values are conceptually plausible and can occur in large simulated datasets such as when many input variables explain the outcome, and noise is absent. For a detailed breakdown of the relationship between inference and prediction according to experimental factors of the simulation, see Figures3 and 4.
ACKNOWLEDGMENTS We are indebted to  Olivier Grisel  and  Gael Varoquaux  for fruitful discussion on the topic (both  INRIA Saclay/France ). We thank several investigators for insightful comments on a previous version of the manuscript:  B. T. Thomas Yeo  ( National University of Singapore ),  Guillaume Dumas  ( Institut Pasteur/ France ),  Nikolaus Kriegeskorte  ( Columbia University/USA ),  Daniele Marinazzo  ( Ghent University/Belgium ),  Benjamin de Haas (University of Giessen/Germany) , and  Joa ˜o Sato  ( Universidade Federal do ABC/Brazil ). D. B. was supported by  National Institutes of Health  ( NIH R01 AG068563A ), the  Canadian Institutes of Health Research  ( CIHR 438531 ), the  Healthy Brains Healthy Lives initiative (Canada First Research Excellence fund) ,  Google  ( Research/Teaching Award ), and by the  CIFAR Artificial Intelligence Chairs program  ( Canada Institute for Advanced Research).
D. B. was also funded by the  Deutsche Forschungsgemeinschaft  ( DFG BZ2/2-1 ,  BZ2/3-1 , and  BZ2/ 4-1 ;  International Research Training Group   IRTG2150 ),  Amazon   AWS Research Grant , and the  German National Merit Foundation , as well as the  START-Program  of the  Faculty of Medicine  ( 126/16 ) and  Exploratory Research Space  ( OPSF449 ),  RWTH Aachen. The research leading to these results has received funding from the  European Union Seventh Framework Programme  ( FP7/2007-2013)  under grant agreement no. 604102  ( Human Brain  Project). D. E. acknowledges support by the  Amazon AWS   Research Grant  ( 2015 ) and the  German National Merit Foundation , as well as the  French National Institute for Informatics and Automation (INRIA)  (Starting Researcher Position SRP 2016).
Particle tracking velocimetry A MATLAB Particle Tracking Velocimetry (PTV) algorithm was adapted from a classical PTV algorithm  (Crocker and Grier, 1996)  and used to obtain instantaneous particle velocities. This algorithm was only used to produce Figure  1B  as it only worked consistently in the best SNR conditions. The first crucial step was to segment the particles inside the region of interest corresponding to the central canal. This region was automatically inferred based on an intensity threshold on the low pass filtered mean image. Particle segmentation was more efficiently performed using a machine learning-based segmentation software (iLastik  Sommer et al. , 2011). Particle fine positioning was performed by fitting a 2D gaussian to each segmented region (particle) and by extracting the center position.
Only pixels above the average signal were kept, and an internal Matlab segmentation algorithm (regionprops) was applied to extract regions of interest in each kymograph. In order to filter only the straight lines corresponding to particle motion, a few additional empirical filtering conditions were applied: Only regions with at least 15 pixels (to avoid random fluctuations), and corresponding to lines with eccentricity above 0. 9 (to prevent from selecting big aggregates or any large objects) were kept. Additionally, to prevent selecting particles stuck in the canal (vertical line in the kymograph), or camera/illumination fluctuations (global change of intensity in the entire image, leading to a horizontal line in the kymograph), we filtered any region whose orientation was close to 0 or 90˚± 180˚(excluding regions whose cosine or sine of orientation were below 0.
1). Note that the intensity threshold was not strong, but most of the particle filtering was made through these four last conditions. A few regions corresponding to individual particle rostro-caudal trajectory were thus obtained. The tangent of these line orientations was calculated, and corrected by pixel size and frame time, to obtain the individual particle velocity. All particles from one kymograph were aggregated to the average velocity and standard error to infer the local flow. Note that we refer to the number of events, or number of lines tracked, rather than the number of particles, as this algorithm cannot guarantee that one particle is not tracked several times if its dorsoventral position changes over time. The full MATLAB script can be found on Github. (Thouvenin, 2019 ; https://github.
This value mixes the instantaneous derivative value and the contraction duration, as they can both play a role in the establishment of different flows. In datasets where embryos did not move, the average flow and displacement were calculated over the entire time series. The average displacement value was calculated in order to avoid getting a large value due to the large time span of the dataset and the use of absolute value. Comparison between methods to measure local flow (PTV/PIV/and kymograph approach) Besides the PTV and kymograph approaches, we also tried to use a particle imaging velocimetry (PIV) algorithm, which is generally used to measure local flow  (Raffel et al. , 2007; Santiago et al. , 1998). This section aims at comparing all these techniques qualitatively with respect to our imaging conditions.
Unfortunately, PTV was not very robust with low SNR, and not very efficient when particle density was quite high. The most critical part was to efficiently detect the particles without error. A first solution is to have a high threshold that selects only the brightest particles, as what was done in Figure  1. Nevertheless, only a few particles could be tracked for a given dataset, and the profile could not be inferred from only a few particle positions. If the threshold were lowered, it would increase the calculation time, and the probability of false detection. Since the PTV algorithm tends to minimize the distance traveled by particles, as the number of false detection increases, the false associations increased even more leading to an important underestimation of particle speed. In contrast, the automatic kymograph seemed more robust in this context because the flow is mostly steady in the rostro-caudal direction, allowing us to perform efficient averaging in this direction.
Manual central canal diameter measurement A Z-Stack of 30 images around the central canal of 30 hpf zebrafish embryos filled with Texas Red was acquired. A custom MATLAB program  (Thouvenin, 2019 , Github, https://github. com/wyartlab/ eLife_2019_OriginAndRole) was used to open, rotate, and crop the image around central canal, then to calculate the maximum intensity projection. The rotation was crucial to fit the central canal projection by a manually selected rectangle, whose width measured the canal average diameter. Even though the canal diameter was not constant along the rostro-caudal direction, the rectangle was placed in order to minimize the total error. The canal diameter was averaged on large rostro-caudal portions, to provide fairly good estimate of its size. Data from wild type and mutant siblings were randomly mixed to perform blind analysis.
For Kurly embryos displayed in Figure  4 , Texas Red was not injected, but we used the time average image of fluorescent beads filling the central canal. The results on WT siblings showed remarkably similar measurements as those obtained with Texas Red. Thus, we decided to combine all these datasets. Live imaging of motile cilia in the central canal 30 hpf embryos from Tg(b-actin:Arl13b-GFP; scospondin icm13/+ ) incrosses were manually dechorionated, laterally mounted in 1. 5% low-melting point agarose, and paralyzed by injecting 1-2 nL of 500 mM alpha-bungarotoxin (TOCRIS) in the caudal muscles of the trunk. A Leica spinning disk confocal microscope equipped with a 40X water immersion objective (N. A. = 0. 8) was used to measure cilia dynamics at a 100 frames per second for 3 s. Note that our estimation of beating frequency were limited to 50 Hz.
Because we found that the average frequency of cilia beating in the central canal was higher than the value of 12-15 Hz previously reported  (Kramer-Zucker et al. , 2005) , we imaged cilia dynamics with three different microscopes (two commercial spinning disk microscopes from 3i and Leica, and one 2-photon scanning microscope using the scanning artifact, as previously described by  Supatto and Vermot (2011) , in order to validate our results). However, all the results displayed in this report were obtained with the same Leica commercial microscope to ensure data homogeneity. We can only postulate that the difference found in beating frequency reflects a bias for long cilia with possible slower kinematics when using differential interference contrast (DIC) to identify cilia.
Quantification of parameters related to cilia structure and dynamics Datasets of cilia position over time were analyzed using custom MATLAB  (Thouvenin, 2019 , Github, https://github. com/wyartlab/eLife_2019_OriginAndRole) scripts based on Fourier transform. First, we opened, rotated, and cropped the image around the central canal. A straight line was manually drawn from the mean intensity image to define the center of the central canal. We also applied a 4 Â 4 spatial sliding-average to increase the local signal-to-noise ratio. The temporal Fourier transform of each averaged pixel was calculated, and the position of its norm extracted and transformed into a frequency value. We could reconstruct 2D maps with the main frequency for each pixel (Figure  2B2 ). It drew distinct regions of interest of constant frequency, which we assumed to be the regions individual cilia could reach within one beating cycle.