dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2018). Despite these findings, less is known about the fundamental mechanisms which might explain those phenomena of rfMRI in a unified framework. In this study, we propose a novel nonlinear feature extraction framework which extracts hidden nonlinear temporal structures repeatedly appearing in the data, and using this approach we show that rfMRI data are composed of a set of distinctive (nonlinear) temporal structures, which we call temporal primitives. The proposed method is a novel framework of nonlinear spatial independent component analysis (sICA), called local space-contrastive learning (LSCL), which is an extension of the recently proposed nonlinear sICA method referred to as SCL  (Morioka et al. , 2018) , see Fig. 1. The method extracts distinctive local (short time-range) nonlinear temporal structures from data so as to decompose it into independent components (spatial patterns, or networks), by training a nonlinear feature extractor in an unsupervised, data-driven manner.
Applying the method to the human-connectome project (HCP) rfMRI dataset  (Essen et al. , 2013) , LSCL identifies a new kind of primitive, local temporal structures which we call temporal primitives. These primitives appear recurrently and consistently across the whole dataset. Our analyses show that the temporal primitives are strongly temporally modulated at each occurrence, and the nonlinearity of the feature extractor (deep convolutional neural network; CNN) contributes to extract the fundamental temporal structures underlying such modulating patterns. This may be impossible to achieve via linear feature extractors. We also show that the temporal primitives recurrently (repeatedly) appear at different time points and spatial locations, leading to a spatial organization of the co-activations of rfMRI signals, including well-known RSNs.
Dataset We used publicly available rfMRI data from the HCP 1200-subject data release (https://db. humanconnectome. org), collected on a 3 T Siemens Skyra scanner with gradients customized for the HCP  (Essen et al. , 2013). All subjects gave informed consent consistent with policies approved by the Washington University Institutional Review Board. We analyzed 1,003 subjects whose full four 15-min (1,200 data points; TR ¼ 0. 72s) rfMRI runs were available. The data were preprocessed based on the HCP's preprocessing pipeline; briefly, 1) spatial processing was applied using the procedure described in  Glasser et al. (2016) , 2) areal-feature-based surface registration was applied (MSMAll HCP pipeline)  (Glasser et al. , 2016) , and 3) structured artifacts were removed by ICA-FIX (independent component analysis followed by FMRIB's ICA-based X-noiseifier)  (Griffanti et al.
Phase encoding was counterbalanced to have both of right-to-left (RL) and left-to-right (LR) directions for each session. For the evaluation of the model trained from the rfMRI data, we also used task fMRI data (tfMRI) from the same HCP data release. tfMRI data were acquired with identical pulse sequence settings while subjects performed 7 tasks  (Barch et al. , 2013) ; Working Memory (WM; 405 frames), Gambling (253), Relational (232), Emotion (176), Social (274),  Motor (284), and Language (316). Each task was comprised of a pair of runs with different phase encoding directions (RL and LR), both used for the analysis. We only used the subjects who had complete tfMRI runs.
Preprocessing Since our method is based on spatial parcel-wise stationarity, it requires prior information about a functional parcellation which guarantees spatial stationarity of source components within each parcel. For such parcellation, we here simply divided the 59,412 cortical surface vertices into regular similar-sized small regions (1,833 parcels), using the following procedures: 1) We located centers of parcels by creating a new cortical sphere model containing a smaller number of vertices (1,002) separately for each hemisphere, whose vertices represent the centers of parcels, with unique parcel labels ("-surface-create-sphere" command in the HCP's Connectome Workbench software, with the setting of the desired number of vertices of 1,000). 2) Each vertex on the original spherical surface was assigned to one of the new parcel labels by searching for the nearest parcel center.
The other training parameters of LSCL are set as follows: Initial learning rate of 0. 1, momentum of SGD of 0. 9, mini-batch size of 128. The initial weights of each layer were randomly drawn from a uniform distribution. The training was performed only from the session-1 of rfMRI dataset; the session-2 of rfMRI and the task-runs of tfMRI were used to evaluate the generalizability of the trained network. The LSCL toolbox (Python) is available from the authors upon request, which fundamental part is based on that of TCL (https://gith ub. com/hirosm/TCL).
Extraction of whole feature values For evaluating the resulting components, we extracted feature values from the dataset (time series Â runs Â subjects Â parcels) by applying the trained feature extractor to short time series of length 46, slidingwindowed with stride 4 from the time series of each run, subject, and parcel (number of features: number of time points Â runs Â subjects Â parcels Â components ¼ 189 Â 2 Â 1; 003 Â 1; 833 Â 5). The extracted feature values were followed by the linear sICA. Compared to the data matrix, the feature values have the additional dimension of components computed from the original data. Note that the temporal dimension of the feature values now represents the timing (temporal index) of the sliding windows corresponding to the obtained feature values, and its temporal resolution is 4 times smaller than the input because of the stride size of the sliding windows.
Timing of event-related temporal primitives To facilitate interpretation of the temporal primitives, we investigated the timing and location of their realizations during task blocks in the task data. Firstly, we extracted feature values from the tfMRI data by applying the feature extractor and the linear sICA matrix trained from the rfMRI data (Section 2. 5) for each task run, through temporal slidingwindow with stride 1 (see Section 2. 6). We then picked realizations which appeared during the task blocks of all subjects and runs (excluding Motor-CUE condition). Considering the temporal invariance of the feature extractor, for each realization, we estimated the timings of the realizations relative to the representative pattern by temporally matching them. Then, the reference point of the (shifted) representative pattern (0 s in Fig.
The subject-representative value of the component was obtained by taking an average of the whole feature values of the subject (across time, parcels, and runs) for each component. To avoid possible bias across acquisition days, the averaging was performed across session-1 and session-2; the feature values of session-2 were obtained by applying the feature extractor trained from session-1 to the data of session-2. Head motion, gender, and age were regressed out as confounds from both of the averaged feature values and the traits. We then calculated Spearman correlation between the feature values and the behavioral traits, and evaluated the significance (two-sided, permutation test). We also evaluated the spatial distribution of the relationship by calculating the average feature value and evaluating the correlation separately for each parcel.
Linear sICA For comparison, we also analyzed the data by a conventional linear sICA analysis. The dataset preprocessed above (parcel-averaged, 800 time points in each run) was at first processed by group-PCA  (Smith et al. , 2014)  to reduce the dimensionality to 800. The output was then fed into group-ICA using FSL's MELODIC tool  (Hyv€ arinen, 1999; Beckmann and Smith, 2004)  to estimate the spatially independent components. We here set the number of components to 15, considering the similarity of the estimated components to the spatial co-occurrence patterns of the temporal primitives.
Representative patterns of the temporal primitives In the LSCL framework, the feature extractor learns componentspecific, spatiotemporally localized elements which we call temporal primitives, so as to demix the rfMRI data into nonlinear components (see Section 2 for details). The learning is based on logistic regression in a "self-supervised" scheme where class labels of time series fragments are defined based on spatial locations. The training (on session-1 of rfMRI) and testing (on session-2 of rfMRI) accuracies after the learning procedure were 4. 3% and 3. 9% respectively, both of them were much higher than the chance level (0. 055%). The similarity of the training and testing accuracies further implies that the extracted temporal primitives are not the artifacts caused by overfitting to the training dataset.
To understand what kind of temporal structure was learned from data, we first visualized their representative temporal patterns for each component (Fig. 2a ). These patterns were represented by an average of the time fragments from the whole dataset (all subjects, runs, parcels, and timings), whose (unmixed) feature values were very large at each component dimension. That means those representative patterns show the most important characteristics representing the component learned by the feature extractor. We also showed representative frequency spectra of the temporal primitives (Fig. 2b ) (see Inline Supplementary Fig. 6a  for the global average spectra of all realizations, and Inline Supplementary Fig. 6b  for the subject-wise-average spectra). The obtained representative patterns are clearly distinctive across components (Fig.
C4 has a long plateau with a weak negative trend followed by a sharper slope (and a positive rebound), which looks relevant to the task-block BOLD response with an adaptation (negative slope) followed by an undershoot. C5 responded to high frequency (noisy) temporal patterns, which can be also seen from its flat spectrum at high frequency range over 0. 2 Hz (Fig. 2b ), which implies that C5 represents high frequency artifacts in fMRI data. The average spectrum (Inline Supplementary Fig. 6a ) has a small peak around 0. 52 Hz, which may be related to the previously reported head position spectra (0. 55 Hz) specific to this dataset  (Power et al. , 2019). Those distinctive patterns across components support our claim that LSCL makes the feature extractor learn component-specific temporal structures to decompose the data into components.
Occurrence ratios of the temporal primitives Fig. 3a  shows the histogram of the feature values. Comparing with surrogate data (see Section 2. 7 for the surrogate data generation), the feature values had significantly higher values compared to chance level: Proportions of 0:28%; 1:4%; 1:3%; 0:10%, and 0:43%, respectively, exceeded the threshold of α ¼ 0:001 (not corrected). This implies local time series contained temporal structure, related to the temporal primitives, which is rarely observed in the surrogate data. The significantly occurring time series are hereafter called realizations of the temporal primitives. C2 and C3 have especially high proportions of the realizations relative to the others, indicating they comprise a relatively larger part of the rfMRI time series. The difference of the sensitivities to the surrogate data also indicates some distinctive characteristics of the temporal primitives; e.
Nonlinear modulations of the temporal primitives The visualization of the representative patterns (Fig. 2a ) is intuitive, but just the first step in understanding the nonlinear computations in LSCL. In fact such patterns do not in themselves describe the nonlinear nature of the components. We next further analyze the nonlinear modulation observed in the realizations of the temporal primitives. The most fundamental nonlinearity in our framework is invariance to temporal shifts. This is partly because the input samples are obtained from data randomly cropped from the data matrix without considering the actual timing of the brain activities; it is also worth noting that rfMRI data do not have any task design. Shift-invariance is also enforced by construction because we use CNNs which have the well-known property of learning shift-invariant features due to their convolutional nature.
Spatial co-occurrence of the temporal primitives To show that the temporal primitives are fundamental elements underlying the intrinsic time-varying co-activations of the rfMRI signals, we visualized the distributions of their spatial co-occurrence patterns during the resting-state, by embedding them into a two-dimensional space for each component (see Section 2. 11 for more details) (Fig. 6 ). To show their relationship to the well-known RSNs, the embedded patterns were colored based on their similarities to the RSNs obtained by the conventional linear sICA analysis on the same dataset; they represent defaultmode network (DMN), posterior-DMN (PDMN), lateral-DMN (LDMN), dorsal attention network (DAN), fronto-parietal network (FPN), cinguloopercular network (CON), motor (MOT), and visual (VIS) networks (Inline Supplementary Fig.
, 2003; Luczak et al. , 2009; Smith et al. , 2009). We also trained the feature extractor only from the tfMRI data (Inline Supplementary Fig. 10 ). The extracted patterns did not have much variety compared to those from rfMRI, and most of them seemed to be relevant to the task design matrix convolved by the HRF. Considering the similarity of those patterns to the temporal primitives from rfMRI (especially C3 and C4), the temporal primitives would be sufficient to represent the task-induced patterns and their variability to some extent. The distinctive relationship of the temporal primitives to some of the cognitive traits suggest that they may have different biological substrates, and capture important individual variations. The correlation of C2 and C3 with fluid intelligence indicates their contributions to the flexible functional organization of the brain.
The experiments implies that the setting n ¼ 5 was reasonable, considering that the components were properly demixed without being contaminated by different temporal primitives. As we can see from Fig. 5 , the temporal patterns of the realizations looked qualitatively consistent for each component even after the nonlinear modulations. However, the best setting may vary across datasets. For example, different length of TR would lead to a different number of distinctive components; a longer TR would complicate the detection of high frequency patterns such as C1, on the other hand, a shorter TR may allow us to decompose C5, considered as high frequency artifact in this study, into some distinctive physiological components. The best setting would be also dependent on the duration of the temporal pattern of interest.
Training of the feature extractor (CNN) by LSCL took about 25 h (Intel Xeon 3. 5 GHz 8 core CPUs, 128 GB Memory, NVIDIA Tesla P100 GPU). After the training, the feature value extraction from fMRI data through sliding-window took about 2 s for each resting-run. LSCL (and SCL) frameworks can be applied not only to rfMRI data, but to many kinds of multidimensional time series which satisfy the assumption of spatial-parcel-wise stationarity, such as calcium imaging, videos, and so on. Compared to SCL, LSCL has a wide applicability because it treats different timings as different data points in addition to the spatial data points; it could be applied to data with much lower spatial dimension, e. g. , electroencephalography (EEG), magnetoencephalography (MEG), electrocorticography (ECoG), where sICA is usually considered inadequate because of a small number of spatial locations.
Conclusion In this study, we proposed a novel nonlinear feature extraction framework called local space-contrastive learning (LSCL), which extracts distinctive nonlinear temporal structure hidden in brain imaging data, by training a deep temporal convolutional neural network in an unsupervised, data-driven manner. By applying to the HCP's fMRI dataset obtained from over 1,000 subjects, we demonstrate that: 1) LSCL identified certain distinctive local temporal structures, referred to as temporal primitives, which repeatedly appeared at different time points and spatial locations, reflecting dynamic resting-state networks, 2) these temporal primitives were also present in task-evoked spatiotemporal responses, and 3) the temporal primitives captured unique aspects of behavioral traits. In addition to these findings underlying fMRI data, our newly-developed feature extraction framework can provide a novel general tool to find out fundamental information from various kinds of imaging modalities, and give us new insight into the complex dynamics of the brain.
Acknowledgements We thank  Okito Yamashita  for the useful comments on this study. This research was supported in part by  JSPS   KAKENHI 18KK0284 ,  19K20355 , and  19H05000. Dr. Calhoun  was funded in part by  NIH   R01EB020407. A. H. was funded by a Fellow Position from CIFAR as well as the DATAIA convergence institute as part of the  "Agence nationale de la recherche" , ( ANR-17-CONV-0003 ) operated by Inria. Data were provided by the  Human Connectome Project ,  WU-Minn Consortium  (Principal Investigators:  David Van Essen  and  Kamil Ugurbil ;  1U54MH091657 ) funded by the 16  NIH Institutes  and Centers that support the  NIH Blueprint for Neuroscience Research ; and by the  McDonnell Center for Systems Neuroscience at Washington University.
A wind turbine's power curve relates its power production to the wind speed it experiences. The typical shape of a power curve is well known and has been studied extensively; however, the power curves of individual turbine models can vary widely from one another. This is due to both the technical features of the turbine (power density, cut-in and cut-out speeds, limits on rotational speed and aerodynamic efficiency), and environmental factors (turbulence intensity, air density, wind shear and wind veer). Data on individual power curves are often proprietary and only available through commercial databases. We therefore develop an open-source model which can generate the power curve of any turbine, adapted to the specific conditions of any site. This can employ one of six parametric models advanced in the literature, and accounts for the eleven variables mentioned above.
In addition, measurement procedures recommended in the IEC standard  61400-12-1 (IEC, 2005)  are continuously improved  (PCWG, 2015). All these activities jointly contribute to the increased accuracy of assessed power curve and in a reduction of the acquisition time needed to evaluate them. Power curves are assessed and made available by turbine manufacturers after the correction of different issues, such as turbulence intensity, wind shear, wind veer, up-flow angle; following the procedures defined by the IEC standard 61400-12-1  (IEC, 2005). These power curves can be found in the product sheets of wind turbines or in databases which collate numerous power curves, such as thewindpower. net (2018)  or  WindPRO (Jens Villadsen, 2010) , which are used in this study. While convenient, these databases are not freely available.
This paper is structured in six main parts. A comprehensive description is given in section 2 of the different steps necessary to evaluate a power curve from general characteristics of a wind farm, such as the rotor area or the nominal power. This section also includes a discussion on the consideration and influence of external environmental parameters. Owing to the numerous input parameters of the model, a sensitivity analysis and statistical analysis of these parameters are described in section 3 and section 4, respectively. The results of our validation are summarised in section 5, where the model output has been compared to power curve from thewindpower. net (2018) database. Some insights on the limitations and possible improvements to the proposed model are discussed in section 6, as well as on its possible domains of application.
Finally, the last group of parameter includes 6 design characteristics of wind turbines that can in most cases be found in manufacturer's product sheets. As argued in  (Pfenninger et al. , 2017) , energy models should be made open to improve the quality of science and aid the productivity of other researchers. We therefore implement the model outlined in Figure  7  in three widely-used programming languages, Python, R and MATLAB. The power curve for an arbitrary wind turbine can be generated by simply specifying the rotor diameter and nominal power output. Sensible defaults are given for all other parameters, or they can be customised as desired. The model code is available from Github https://github. com/YvesMSaintDrenan/WT_PowerCurveModel. 1. The present sensitivity analysis is limited to a univariate analysis: one parameter is varied at the time.
Statistical analysis of the most sensitive model input parameters While parameters such as the nominal power and the rotor area are readily available for each turbine, this is not the case for other parameters such as the maximum power coefficient or the rotor minimal and maximal speeds. In order to address such situations, a statistical analysis of the model parameters was performed, which can be used as guidance to the choice of unknown parameters. For this, we used the database of wind turbines and power curves provided by thewindpower. net (  2018 ), which includes extended data on the main turbine characteristics. As of May 2019, this commercial database contains about 780 turbines models.
Validation of the parametric power curve model A validation of the model introduced in section 2 has been conducted using the power curve of 91 wind turbines with a nominal power greater than 1 MW as provided by their manufacturers and available in the thewindpower. net (2018) dataset. For this validation, the power curve model has been run with specific information on the nominal power and rotor area, while other model inputs are set to the reference values described in section 4 and the air density is set to 1. 225 kg/m 3. As the level of turbulence intensity corresponding to each power curve is unknown, they are compared to model outputs obtained with values of the turbulence intensity ranging between 0 and 10 %. As a consequence, a quantitative validation could not be conducted, as this unknown parameter would have to be optimised in the model (which would instead be calibration).
Rather, a qualitative validation is performed through a visual comparison of the model output to the database of power curves, whose main outcomes are described in this section. In Figure  12 , the result of the validation is given for three wind turbines which show a close correspondence between model output and power curve from the database. While this does not completely exclude the possibility of a systematic modelling error, the degree of similarity across the plots suggests the model can synthesise realistic power curves for a variety of wind turbines. It can be observed that the best matches between model output and turbine data are obtained for different values of the turbulence intensity: 2. 5, 5 and 7. 5% for the wind turbine 258, 263 and 270. As mentioned above, information on the turbulence intensity is not available in the database, which hinders a proper validation of the model and represents a non-negligible source of uncertainty for power estimation made with these power curves taken from manufacturers and other sources.