dataesr/scientific-paragraphs-categorization huggingface dataset:
Deep neural network implementation The DNNs were implemented using Keras  (Chollet, 2015)  or PyTorch  (Paszke et al. , 2017)  and run on NVIDIA Titan Xp GPU using CUDA. Our implementation of BrainNetCNN and GCNN were based on GitHub code from the original papers  (Kawahara et al. , 2017; Kipf and Welling, 2017). Our implementations achieved similar results as the original implementations when using the toy datasets and hyperparameters provided by the original GitHub implementations. More details about hyperparameter tuning can be found in Appendix A2.
Statistical tests For the HCP dataset, we performed 20-fold cross-validation, yielding a prediction accuracy for each test fold. To compare two algorithms, the corrected resampled t-test was performed  (Nadeau and Bengio, 2003; Bouckaert and Frank, 2004). The corrected resampled t-test corrects for the fact that the accuracies across test folds were not independent. In the case of the UK Biobank, there was only a single test fold, so the corrected resampled ttest could not be applied. Instead, when comparing correlations from two algorithms, the Steiger's Z-test was utilized  (Steiger, 1980). When comparing MAE, a two-tailed paired sample t-test was performed. When comparing prediction accuracies for sex, the McNemar's test was utilized  (McNemar, 1947).
Scaling of prediction performance as a function of sample size The large UK Biobank dataset allowed us to explore the effect of sample size on predicting fluid intelligence. The test set (N = 1000) was the same as before to allow for meaningful comparisons. We considered 100, 500, 1000, 2000, 3000, 4000, 5000 and 6000 and 7868 subjects for training and validation. The case of 7868 subjects was identical to the analysis from the previous sections. In the case of 3000, 4000, 5000 and 6000 subjects, the validation set comprised the same set of 1000 subjects as in the previous sections. In the case of 100, 500, 1000 and 2000 subjects, we split the participants with a 3:1 ratio. For example, in the case of 100 subjects, there were 75 training and 25 validation subjects. Like before, the participants were randomly selected but we ensured the distributions of fluid intelligence in the training and validation sets were similar to the distribution of the test set.
Data and code availability This study utilized publicly available data from the HCP (https://www. humanconnectome. org/) and UK Biobank (https://www. ukbiobank. ac. uk/). The 400 cortical ROIs  (Schaefer et al. , 2018 ) can be found here (https://github. com/ThomasYeoLab/CBIG/tree/master/stable_projects/brain_parcellation/Sch aefer2018_LocalGlobal). The kernel regression and DNNs code utilized in this study can be found here (https://github. com/ThomasYeoLab/CBIG/tree/master/stable_projects/predict_phenotypes/He 2019_KRDNN). The trained models for the UK Biobank dataset can also be found in the above GitHub link. The code was reviewed by one of the co-authors (MN) before merging into the GitHub repository to reduce the chance of coding errors.
Control analysis Tables  S3  and S4  show the performance of the DNNs using hyperparameters from the original publications  (Kawahara et al. , 2017; Parisot et al. , 2017; 2018)  versus our tuned hyperparameters. The performance of our hyperparameters compared favorably to the performance of the original hyperparameters. This is not surprising, since our hyperparameters were obtained by tuning using the datasets shown in this paper.
Discussion In this study, we showed that kernel regression and DNNs achieved similar performance in RSFC-based prediction of a wide range of behavioral and demographic measures across two large-scale datasets totaling almost 10,000 participants. Furthermore, FNN performed as well as the two DNNs that were specifically designed for connectome data. Given comparable performance between kernel regression and the DNNs and the significantly greater computational costs associated with DNNs, our results suggest that kernel regression might be more suitable than DNNs in some neuroimaging applications.
Potential reasons why DNNs did not outperform kernel regression for RSFC-based prediction There are a few potential reasons why DNNs did not outperform kernel regression in our experiments on RSFC-based behavioral prediction. First, while the human brain is nonlinear and hierarchically organized  (Deco et al. , 2011; Breakspear, 2017; Wang et al. , 2019) , such a structure might not be reflected in the RSFC matrix in a way that was exploitable by the DNNs we considered. This could be due to the measurements themselves (Pearson's correlations of rs-fMRI timeseries), the particular representation (N x N connectivity matrices) or particular choices of DNNs, although we again note that BrainNetCNN and GCNN were specifically developed for connectome data. Second, given the much larger datasets used in computer vision and natural language processing  (Chelba et al.
The reason is that within a nested (innerloop) cross-validation framework, information from tuning one fold might leak to another fold (via the person tuning the hyperparameters). To elaborate, recall that we divided the HCP dataset into 20 folds. We tuned the hyperparameters of the DNNs using folds 2 to 20 and applied the trained DNNs to fold 1. Since fold 1 was not used in tuning the hyperparameters, the performance of the DNNs in fold 1 was unbiased. However, when fold 2 became the test fold, we utilized the same hyperparameters to train using folds 1, 3 to 20. This is problematic because fold 2 was originally utilized to tune the hyperparameters, so consequently the performance of the DNNs in test fold 2 was inflated. One could try to independently tune the hyperparameters for each fold independently.
Thus, overfitting was "caught" in the test set, which highlights the benefits of adopting a training-validation-test framework. Finally, we note that there are generally too many DNN hyperparameters (and design choices) to be listed in a paper. In fact, there were hyperparameters too complex to completely specify in this paper. However, we have made our code publicly available, so researchers can refer to the code for the exact hyperparameters. We encourage future neuroimaging DNN studies to also make their code publicly available.
Prediction performance in the literature Comparing our prediction performance with the literature is difficult because of different datasets, sample sizes, cross-validation procedures and de-confounding strategies. For example, we regressed age, sex, and motion (FD) from the behavioral measures, but other studies might not perform any regression or use a different set of regressors. Nevertheless, we believe that our prediction performance is generally consistent with the literature. As mentioned earlier, our sex prediction accuracy of 91. 6% in the UK Biobank is among the best in the literature. For example, Ktena and colleagues (2018) reported a sex prediction accuracy of around 80% when using 55 x 55 functional connectivity matrices from 2500 UK Biobank subjects. On the other hand, Chekroud and colleagues (2016) reported sex prediction accuracy of 93% when using cortical thickness and subcortical volumes of 1566 subjects from the Brain Genomics Superstruct Project  (Holmes et al.
, 2015). In the case of fluid intelligence, our prediction accuracies (Pearson's correlation) ranged from around 0. 257 to 0. 297 (excluding GCNN which performed poorly) in the HCP dataset. Although earlier RSFC-based behavioral prediction studies have reported high fluid intelligence prediction accuracy in the HCP dataset  (Finn et al. , 2015) , newer studies using more subjects reported lower accuracies comparable with our results. For example, Dubois and colleagues (2018) reported a prediction accuracy (Pearson's correlation) of 0. 27 for fluid intelligence in the HCP dataset. On the other hand, Greene and colleagues (2018) reported a prediction accuracy (Pearson's correlation) of 0. 17 for fluid intelligence in the HCP dataset (but only using data from a single resting-fMRI session). Thus, our prediction accuracies for fluid intelligence is consistent with the literature.
In the case of age prediction, we achieved a prediction accuracy (Pearson's correlation) of 0. 6 and an MAE of 4. 8 in the UK Biobank dataset. Comparing these results with the literature is difficult because of sensitivity to age range in the dataset. For example, many studies utilized either lifespan  (Cole et al. , 2017; Liem et al. , 2017)  or developmental  (Sturmfels et al. , 2018; Nielsen et al. , 2019)  cohorts, while the UK Biobank comprised older adults (more than 45 years old). Furthermore, many studies preferred to use structural MRI, instead of RSFC, for predicting age  (Cole et al. , 2017; Sturmfels et al. , 2018; Varikuti et al. , 2018). Liem and colleagues (2017)  achieved MAEs ranging from 5. 25 to 5. 99 when using RSFC for predicting age in a lifespan dataset comprising 2354 subjects, which was worse than our MAE.
On the other hand, their prediction accuracies (Pearson's correlation) ranged from 0. 79 to 0. 93, which was better than our prediction accuracy (Pearson's correlation). Overall, this suggests that our prediction performance is probably comparable with other RSFC studies, although we emphasize that comparing age prediction performance across datasets is non-trivial. It is important to mention that prediction performance was poor for a number of target variables across all four prediction algorithms. For example, in the case of pairs matching in the UK Biobank dataset, predicting the median of the training set yielded lower MAE than all four models, suggesting that pairs matching is not an easily predictable trait using RSFC. Therefore, it might not be meaningful to compare the models for pairs matching.
On the other hand, we note that for both age and fluid intelligence prediction, all four models performed better than predicting the median of the training set. Similarly, sex prediction was a lot better than chance, given that there were roughly equal number of males and females in the dataset. For these three target variables (age, sex and fluid intelligence), all four models exhibited very similar performance. It is also worth noting that the poor average COD in the HCP dataset is consistent with the literature. For example, of the 58 behavioral measures, 48 of them were also utilized in the HCP MegaTrawl (https://db. humanconnectome. org/megatrawl/). For the 300dimensional group-ICA results, HCP MegaTrawl achieved an average COD of -0. 177 (original data space), while kernel regression in the current study achieved an average COD of -0.
Limitations and caveats Although the current study suggests that kernel regression and DNNs achieved similar performance for RSFC-based behavioral prediction, it is possible that other DNNs (we have not considered) might outperform kernel regression. Furthermore, our study focused on the use of N x N static RSFC matrices for behavioral prediction. Other RSFC features, such as dynamic RSFC features  (Calhoun et al. , 2014; Preti et al. , 2017; Liégeois et al. , 2019) , in combination with DNNs might potentially yield better performance  (Hongming Li et al. , 2018; Khosla et al. , 2019). We also note that our evaluation procedure was performed on the HCP and UK Biobank datasets independently. Therefore, we expect the reported prediction performance to be maintained if new participants were recruited in the respective studies under the same experimental conditions (e.
g. , no scanner upgrade, same population, same acquisition protocol and preprocessing, etc). However, the reported prediction performance would likely drop if the trained models (from the UK Biobank or HCP) were applied to other datasets  (Arbabshirani et al. , 2017; Woo et al. , 2017). At this point, it is unclear which approach (kernel regression, FNN, BrainNetCNN or GCNN) would generalize better to a completely new dataset. This is obviously an active area of research given the increasing number of large-scale publicly available brain imaging datasets.
Conclusion By using a combined sample of nearly 10,000 participants, we showed that kernel regression and three types of DNN architectures achieved similar performance for RSFCbased prediction of a wide range of behavioral and demographic measures. Overall, our study suggests that kernel regression might be just as effective as DNNs for certain neuroimaging applications, while incurring significantly less computational costs. each continuous prediction H , the participant was classified as male or female based on whether it was larger or smaller than the threshold. We tuned the threshold to obtain the best accuracy in the UK Biobank validation dataset and used this threshold in the test set.
A2. More details of deep neural networks In this section, we describe further details of our DNN implementation. • For GCNN, we adopted Keras code from the GCNN GitHub repository (https://github. com/tkipf/keras-gcn; Kipf and Welling, 2017). We made some minor modifications to the code, e. g. , the modified code directly loaded the graph adjacency matrix, instead of loading the edges and generating the adjacency matrix. As another example, our graph convolution layer loaded the graph matrix as parameters rather than as an input. However, we emphasized that the core functionalities (e. g. , graph convolution) remained unchanged. As a sanity check, we applied our modified code to the original toy data using the original hyperparameters provided by the original GitHub repository. Our results were comparable to the original implementation (Table  S5 ).
• The original BrainNetCNN implementation used the Caffe framework (https://github. com/jeremykawahara/ann4brains; Kawahara et al. , 2017). We reimplemented BrainNetCNN in Keras and PyTorch following the original Caffe code as closely as possible. The Keras version was applied to the HCP data, while the PyTorch version was applied to the UK Biobank data. The reason for this inconsistency was that after our experiments with the HCP dataset using Keras, we realized that the Keras framework yielded slightly different results each time the code was run. This was apparently a well-known issue of the framework. As such, we decided to implement a second version in PyTorch, which was then applied to the UK Biobank. As a sanity check, we applied both implementations (Keras and PyTorch) to the original toy data using the original hyperparameters provided by the original GitHub repository.
Our implementations achieved comparable results with the original implementation (Table  S6 ). • In the case of the FNN, since this is just a generic feedforward neural network, so we implemented using default libraries in Keras and PyTorch. The Keras version was applied to the HCP data, while the PyTorch version was applied to the UK Biobank data. The reason for this inconsistency is the same as the previous bullet point. • Representative learning curves for the HCP dataset are shown in Figure  S12. Learning curves for the UK Biobank are shown in Figures S13 to S15. The training curves showed good accuracy/error, suggesting that we are not underfitting to the data. The validation curves were plateauing, suggesting that we were not stopping too early in our training. Since the validation and test curves were progressing in almost lockstep (except for certain instances of GCNN), our stopping criterion (based on the peaks of the validation curves) was reasonable.
For most behavioral measures, there were relatively big gaps between the training and validation/test curves, suggesting overfitting. However, we have already deployed several standard strategies to reduce overfitting, including dropout, L 2 -regularization/weight-decay and batchnormalization. In the case of the HCP dataset: • For all three DNNs, all behavioral measures were z-normalized based on training data. The loss function was mean squared error (MSE). Optimizer was stochastic gradient descent (SGD). With the MSE loss, the output layer has 58 nodes (FNN and BrainNetCNN) or filters (GCNN). • In the case of the main results  (Figures 4, 5, S2 and S3) , the hyperparameters were tuned manually by trial-and-error. Since each test fold was of size 47 or 48, we simply set 48 to be the batch size (except GCNN, which utilized the whole dataset in a single mini-batch).
, 2013; Ilievski et al. , 2017 , Eriksson et al. , 2019)  to assist in hyperparameter tuning using the UK Biobank validation dataset. For each DNN, the HORD algorithm automatically tuned the DNN hyperparameters within user-specified ranges of various hyperparameters. Not all hyperparameters were tuned by HORD because the speed and performance of HORD worsened when too many hyperparameters were tuned. Therefore, we determined several hyperparameters based on our previous manual tuning experience, i. e. momentum = 0. 9, batch size = 128 (except GCNN's batch size is 1 as it loads all data at once), weight initialization = Xavier uniform (PyTorch) or Glorot uniform (Keras), Chebyshev polynomial basis filters with degree of 1 for GCNN. • For FNN, we tuned the number of layers (2 to 4 layers), number of nodes for each layer (2 to 512 nodes), dropout rate (0 to 0.
large UK Biobank dataset allowed us the luxury of splitting the 8868 subjects into training (N = 6868), validation (N = 1000) and test (N = 1000) sets, instead of employing an inner-loop cross-validation procedure like in the HCP dataset. Care was taken so that the distributions of various attributes (sex, age, fluid intelligence and pairs matching) were similar across training, validation and test sets. Hyperparameters were tuned using the training and validation sets. The test set was only utilized to evaluate the final prediction performance. A separate DNN was trained for each of the four behavioral and demographic measures. Thus, the hyperparameters were tuned independently for each behavioral/demographic measure. Further details about DNN hyperparameters are found in Appendix A2. Initial experiments using a single neural network.
Acknowledgment We like to thank the anonymous reviewers for the very helpful feedback. We would also like to thank  Christine Annette ,  Taimoor Akhtar ,  Li Zhenhua  for their help on the HORD algorithm. This work was supported by  Singapore   MOE Tier 2  ( MOE2014-T2-2-016 ),  NUS Strategic Research  ( DPRT/944/09/14 ),  NUS SOM Aspiration Fund  ( R185000271720 ), Singapore  NMRC  ( CBRG/0088/2015 ),  NUS YIA  and the  Singapore National Research Foundation (NRF)  Fellowship (Class of 2017). Our research also utilized resources provided by the  Center for Functional Neuroimaging Technologies ,  P41EB015896  and instruments supported by  1S10RR023401 ,  1S10RR019307 , and  1S10RR023043  from the  Athinoula A. Martinos Center for Biomedical Imaging  at the  Massachusetts General Hospital. Our computational work was partially performed on resources of the  National Supercomputing Centre, Singapore  (https://www.
By determining similar parameters for fixed dye molecules, we found that the apparent diffusion coefficient of the chromatin bound state was within our precision limit and so represents molecules that are essentially immobile (Extended Data Figure  8 ). We conclude that we can use 20 ms exposure trajectories to distinguish unconfined freely diffusing molecules from confined chromatin bound proteins. Each dataset was produced by imaging 30 cells, and the parameters were extracted from 5,557 and 15,528 trajectories, respectively (see Extended Data Figure  5  and the Online Methods for more detail).
This is consistent with the in vitro experiments which show that, in comparison to CHD4, the deacetylase subunit by itself does not bind strongly to nucleosomes (Extended Data Figure  9a ). To investigate the chromatin binding kinetics of the CHD4 remodeller and the MTA2 deacetylase subcomplex in the presence and absence of MBD3, we next determined association times from the time spent freely diffusing between confined chromatin-bound states. We also attempted to determine dissociation times from the time spent bound to chromatin between unconfined freely diffusing states (Figure  2d  and Extended Data Figure  5  -see also the Online Methods). The distribution of association and dissociation times were well approximated by a single exponential, suggesting a Poissonian process. Consistent with our conclusion that CHD4 is primarily responsible for recruitment of NuRD to chromatin, we found no increase in the association time of CHD4 upon removal of MBD3.
Because the background fluorescence from freely diffusing dCas9-GFP prevented 3D tracking of specific genomic loci on our DH-PSF microscope, we tracked the Tbx3 enhancer locus in 2D. Although this meant that we could not directly compare the parameters obtained in the 2D (CARGO-dCas9) and 3D (NuRD single molecule) tracking experiments, classification of the sub-trajectories using our Gaussian mixture model and subsequent Gaussian fitting analysis of the anomalous exponent distributions once again revealed one slow and two fast chromatin states. In particular, as was the case when tracking CHD4, removal of MBD3 significantly decreased the rate of diffusion of the Tbx3 enhancer in the fast-diffusing F1 and F2 states and increased the proportion of sub-trajectories in the fast decondensed F2 state (Figure  4b , Extended Data Figure  11 ).
A) In-nucleus chromosome conformation capture (Hi-C) In-nucleus Hi-C was carried out on E14 wild-type and 7g9 Mbd3-ko ES cells as previously described  16, 29. 50 bp paired-end sequencing was carried out on a HiSeq4000 instrument. Sequences were mapped to the genome using NucProcess (https://github. com/tjs23/nuc_processing) and analysed further using NucTools (https://github. com/tjs23/nuc_tools). Enhancer-promoter interactions were analysed using active enhancers previously defined in mouse ES cells by the presence of H3K27Ac, H3K4me1, p300 and lack of H3K4me3, whilst active promoters were defined by the presence of H3K4me3  16. CTCF/SMC3-mediated looping was analysed using loops previously defined in mouse ES cells by Pekowska et al  55.
F) 3D single-molecule image processing and generation of trajectories Single molecules were localized from 3D movies using the easy-DHPSF software 59 with a relative localization threshold of 100 in all 6 angles for 20 ms data and relative thresholds of 116, 127, 119, 99, 73 and 92 for the 500 ms data. Trajectories of individual molecules were then determined using custom Python code for connecting molecules in subsequent frames if they were within 800 nm for 20 ms trajectories and within 500 nm for 500 ms trajectories (https://github. com/wb104/trajectory-analysis).
Gaussian fitting of 500 ms exposure anomalous exponents Although the algorithm we designed segments 500 ms exposure trajectories into slow-diffusing and fastdiffusing components, the distribution of the anomalous exponents in the fast-diffusing component showed that two peaks can be identified by Gaussian fitting. Indeed, when we fitted 1, 2 or 3 Gaussians to the anomalous exponent distributions for chromatin bound NuRD complex subunits such as CHD4, MBD3 and MTA2 in wildtype ES cells, we found that 2 Gaussians were the best minimal model to account for the data -having the lowest BIC value (Extended Data Figure  9c ). Next, we assessed the reproducibility of analysing the anomalous exponent distributions of the fastdiffusing chromatin bound CHD4. Data was collated from 3 fields of view with around 6 cells in each field of view imaged, leading to a total of around 18 cells per condition.
To ensure enough data was collected to account for cell-to-cell heterogeneity, an additional 3 fields of view containing around 18 cells were collected for chromatin bound CHD4 and shown to have a similar anomalous exponent distribution (data not shown). In contrast, CHD4 in the absence of MBD3 showed a different distribution from CHD4 in wild-type ES cells when compared to either the first or second day CHD4 was imaged (p = 0. 005 and p = 0. 02 respectively), thus showing the reproducibility of the changes observed.