dataesr/scientific-paragraphs-categorization huggingface dataset:
R 2 comparison with other predictive models The goodness of fit (R 2 ) has been calculated by regressing measured and predicted scores to assess the neuropsychological variance explained by the proposed prediction model. In the model training phase (dataset 2-training), the average R 2 achieved was 0. 19 ± 0. 09 The axial projection of the labelling of neuropsychological scores corresponds to the strongest white matter associations. Visit https://neurovault. org/collections/11260/ or Supplementary material, Section C to view individual neuropsychological white matter maps. The labelling text font size and curvature reported were manually set to suit the size of the significant clusters and to follow the orientation of the white matter (see Supplementary material, Section D for high-resolution images).
Discussion Applying state-of-the-art data-embedding methods, we succeeded in combining complementary databases of stroke patients. We produced a novel atlas of neuropsychological scores associated with brain disconnections-the NWMA. The proposed atlas is associated with an openly available web application, the DSD, which capitalizes on our methods and provides new anatomical insights into cognitive symptoms for researchers and clinicians. Out-of-sample validation of the DSD model (dataset 2-validation) accurately predicted 65 neuropsychological scores with a small prediction error below 20%. Similar patterns of disconnections in our stroke cohorts were distributed closely in the embedding space, comparable to other research fields using the UMAP method for data clusterization purposes, 56 e.
This information will be a valuable resource in clinical settings, for example for the planning of personalized therapeutics and rehabilitation strategies. This is a step forward in comparison to many previous stroke AI methods that have a merely diagnostic purpose. 70 The DSD model has a prognostic vocation based on cross-modal data (neuroimaging input-neuropsychological outcome prediction). Figure  6  Personalized neuropsychological profile predictions. Predictions of neuropsychological profiles are reported for three representative patients from dataset 2-validation. For each profile, the outside ring indicates the correspondence with neuropsychological domains. Note that the polarity of some scores was inverted for readability so that higher scores always indicate better performance. A-C illustrate patients with a left inferior frontal lesion associated with chronic motor impairment (A); a left temporoparietal lesion suffering from chronic language impairment (B); and a right cerebellar stroke with chronic deficits in visuospatial and verbal memory processes (C).
All radar plots from dataset 2-validation are reported in Supplementary Figs  58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77. However, predictions were not equally accurate across functions (Supplementary Table  3 ). Three factors might explain these differences. First, some neuropsychological scores are more reliable than others in assessing performances. 71 Second, plasticity and interindividual variability might interact with recovery. 20, 32, 72, 73 Third, the disconnectome model may not capture all the variance of brain injuries. Indeed, hypoperfusion  74  and hypometabolism  75  factors as well as acute imaging changes such as pseudonormalization  4  are not currently accounted for. Moreover, we arbitrarily chose to create the disconnectome morphospace using two embedding dimensions for practical and intuitive purposes.
While this proved to be a reliable way to describe the latent data structure and predict symptoms, future research might explore higher dimensionality of the morphospace (data and code available on demand from the authors). Finally, we used one of the largest and the most comprehensively explored stroke dataset available in terms of neuropsychological score. However, as the validation group represented about 20% of the training group, its modest numerosity limited the power of the validation step. In order to circumvent this potential limitation, the DSD web application allows for a wider validation with crowdsourcing and the addition of new datasets. In Fig. 7A , we compared the DSD with six other models using different predictivity frameworks. The DSD UMAP model leveraging structural dysconnectivity outperformed lesion and functional dysconnectivity predictors.
The statistical comparison was made across n = 86 neuropsychological scores recorded for dataset 2. The DSD achieved the highest R 2 for motor scores (Supplementary Fig. 4C ), in agreement with the structural dysconnectivity results recently reported by Bowren et al. 44  In contrast, Bowren et al. 44  indicated that functional connectivity was the best predictor for language scores. Our structural connectivity model, enriched with the UMAP method, outperformed f-VB results for language scores. This indicates that dimensionality reduction of white matter disconnections improves the variability estimation of the language network and its statistical association with language impairments. We further prove this point by predicting the semantic fluency (animals) score for the same Bowren et al. 44  cohort; out-sample predictions achieved an R 2 = 0.
201 (Fig. 7B ). Altogether, this evidence demonstrates that disconnectivity predictors may be an optimal strategy for neuropsychological scores 1 year after a stroke. As a proof of concept, we tested the DSD model generalizability by training and testing in two entirely different cohorts (datasets 4 and 5). Figure  7C  indicates that the DSD model external cohort predictions agreed with the actual measurements with a medium effect size (goodness of fit R 2 = 0. 18 for the Bell's score). Moreover, disconnectome and lesion data modelled with UMAP outperform voxel-based approaches (Fig. 7A ). This result validates our initial observation, for which no one-to-one relationship between structures and clinical presentation is possible, but rather the integrated brain functioning necessitates high-order modelling for an accurate description.
26 n the current work, we aimed to propose a novel framework that leverages MRI data to predict patients' long-term clinical/cognitive scores. We obtained access to five independent stroke cohorts thanks to a highly collaborative research network. Dataset 1 included MRI and was acquired in >1000 patients and informed the UMAP method on the variability of white matter damage subsequent to a stroke. With dataset 2, we trained a regression model for predicting 86 neuropsychological scores. One of those, the semantic fluency test, was tested in an independent cohort, dataset 3. Datasets 4 and 5 served to train and test the same datasets 2 and 3 models for visuospatial abilities and demonstrated the proposed framework's generalizability to external cohorts. The commonality of clinical assessment across centres dictated the specific score choices.
However, to promote future model testing and collaborative research at the global level we developed the DSD web application, a free and user-friendly web browser tool that only requires an Internet connection and a lesion disconnection data. Instant software access and automatic updates make the world-wide web the ideal medium for clinical translations. The application of the DSD method can potentially help the assessment of personalized prognosis. The developed web application has been released to facilitate a broader model validation critical for future global validation testing the DSD reliability. Only after this indispensable validation step should the DSD web application be considered for clinical trial testing. Hence, the DSD aims to benefit the researchers' understanding of brain functioning and patients' treatments alike.
2  of each neuropsychological score. Moreover, to assess model stability, we randomly assigned patients in the DSD model training and validation, with a 100 permutation k-fold validation (all dataset 2-training and -validation). The training set R 2 distribution across permutations is reported in Supplementary Fig. 4C. On average the median R 2 achieved across iterations was 0. 20 ± 0. 14 (range 0. 10, 0. 83).
03 for f-VB, R 2 = 0. 07 for lesion size and age as predictors and R 2 = 0 for the group mean. The asterisks correspond to the P-value obtained from a paired t-test (2-tails, Bonferroni corrected for multiple comparisons): ***P < 0. 001. (B) DSD out-of-sample predictions for the semantic fluency test, animal category, n = 26 stroke patients (dataset 3). (C) DSD replication and out-of-sample predictions for visuospatial abilities, Bell's score, n = 193 stroke patients (dataset 5).
Funding This work was supported by the  European Union 's  Horizon 2020 research and innovation programme  under the  European Research Council (ERC)  Consolidator grant agreement No. 818521  (M. T. d. S. , DISCONNECTOME) and the  Marie Skłodowska-Curie  grant agreement No. 101028551  (S. J. F. , PERSONALISED), and the Donders Mohrmann Fellowship No. 2401515  (S. J. F, NEUROVARIABILITY). Additionally, this work was conducted in the framework of the  University of Bordeaux 's  IdEx 'Investments for the Future' program RRI 'IMPACT ', which received financial support from the  French government. P. N. is funded by the  Welcome Trust  and the  UCLH NIHR Biomedical Research Centre. V. P. is funded by the  NextGenerationEU PNRR  grant No. SOE_0000130.
Data availability All neuropsychological score maps used for defining the white matter atlas of neuropsychological components are freely available at https://neurovault. org/collections/11260/. The raw dataset imported in the BCBtoolkit software to calculate individual patient disconnectomes is available at https://www. humanconnectome. org (7 T diffusion data). In addition, processed data are available on request to the corresponding author or directly at https://osf. io/5zqwg/. The code used in the analyses is available as part of the BCBtoolkit package http://toolkit. bcblab. com and the DSD web application http://disconnectomestudio. bcblab. com. Any additional information is available upon request to L. T. and M. T. S.
11 for L-VB, R 2 = 0. 03 for f-VB, R 2 = 0. 07 for lesion size and age as predictors and R 2 = 0 for the group mean. The asterisks correspond to the P-value obtained from a paired t-test (2-tails, Bonferroni corrected for multiple comparisons): ***P < 0. 001. (B) DSD out-of-sample predictions for the semantic fluency test, animal category, n = 26 stroke patients (dataset 3). (C) DSD replication and out-of-sample predictions for visuospatial abilities, Bell's score, n = 193 stroke patients (dataset 5).
Specifically, we propose a linearly scalable approach to context modeling, formulating Nonnegative Matrix Factorization (NMF) as a differentiable layer integrated into a U-shaped architecture. The shifted window technique is also utilized in combination with NMF to effectively aggregate local information. Factorizers compete favorably with CNNs and Transformers in terms of accuracy, scalability, and interpretability, achieving state-of-the-art results on the BraTS dataset for brain tumor segmentation and ISLES'22 dataset for stroke lesion segmentation. Highly meaningful NMF components give an additional interpretability advantage to Factorizers over CNNs and Transformers. Moreover, our ablation studies reveal a distinctive feature of Factorizers that enables a significant speed-up in inference for a trained Factorizer without any extra steps and without sacrificing much accuracy.
Unlike CNNs, ViT encodes images as a sequence of 1D patch embeddings (known as tokens) and dynamically highlights the important tokens using self-attention layers, which in turn increases the capability of learning long-range dependencies. Due to lack of locality inductive bias, ViT is data-hungry and generally requires a larger dataset to perform as effectively as its CNN counterparts, leading to poor performance when trained on insufficient data, which is usually the case in medical imaging. Furthermore, the quadratic complexity of self-attention makes Transformers computationally intractable on long sequences of patches. Therefore, existing models use self-attention layers only after somehow reducing the image resolution, thereby failing to fully exploit the global context at the higher resolution.
This work proposes a family of architectures, dubbed as Factorizer, which leverages the power of low-rank matrix approximation (LRMA) to construct an end-to-end medical image segmentation model. Among LRMA methods, Nonnegative Matrix Factorization (NMF) has demonstrated a remarkable ability to compress data and automatically extract easy-to-interpret sparse factors  (Gillis, 2014 (Gillis, , 2020)). Hence, we propose a linearly scalable alternative to self-attention by formulating an NMF algorithm as a differentiable layer. Moreover, a series of matricization operations is introduced, which enables NMF to effectively exploit both global and local contexts. The Factorizer block is constructed by replacing the self-attention layer of a ViT block with our NMF-based modules and then integrated into a U-shaped architecture with skip connections.
We evaluated the effectiveness of our approach for the segmentation of brain tumors and stroke lesions in MRI data. Factorizers achieved competitive results on the BraTS  (Menze et al. , 2014; Bakas et al. , 2017)  and ISLES'22  (Petzsche et al. , 2022)  datasets, having outperformed state-of-the-art methods based on CNN and Transformer. Our experiments showed that NMF components are highly meaningful, which gives a great advantage to Factorizers over CNNs and Transformers in terms of interpretability. Furthermore, our ablation studies revealed a distinctive interesting feature of Factorizers that enables us to easily speed up the inference for a trained Factorizer model with no extra steps and without sacrificing much accuracy.
Visual transformers. Transformers with attention mechanisms  (Vaswani et al. , 2017) , introduced originally for language modeling, have recently proven promising on computer vision tasks. Particularly, the pioneering Vision Transformer (ViT) model  (Dosovitskiy et al. , 2020)  outperformed state-of-the-art CNNs on image recognition by large-scale pre-training and fine-tuning a pure Transformer applied to sequences of image patches. In contrast to CNNs, ViT lacks any inductive bias such as locality, and therefore, generally shows poorer performance than its CNN counterparts (e. g. , ResNets  (He et al. , 2016) ) when trained from scratch on small-size or mid-size datasets, which is usually the case for medical imaging. Efforts have been made to mitigate this limitation. For example, Tokens-to-Token ViT (T2T-ViT)  (Yuan et al.
Wrapped NMF module The major component of Factorizer is the Wrapped NMF module, which relies on matricization (i. e. , an operation that turns a tensor into a matrix) and NMF. As shown in Fig. 2 (b), a Wrapped NMF subblock first applies a pointwise convolution to linearly project each voxel. The output is then reshaped into a batch of matrices using a Matricize operation, which is detailed later on in Section 3. 4. 1. The resulting matrices are passed through ReLU to clamp all their elements into nonnegative values and then low-rank approximated using NMF. The reconstructed matrices are reshaped back to their original size using the Dematricize operation, i. e. , the inverse of Matricize. Finally, another pointwise convolution is applied, yielding the output. More formally, Wrapped NMF can be computed as follows:  1 = PointwiseConv( ),  2 = Matricize( 1 ),  3 = NMF(ReLU( 2 )),  4 = Dematricize( 3 ), WrappedNMF( ) = PointwiseConv( 4 ), (6) where NMF(⋅) is the NMF layer, described in 3.
In PyTorch and TensorFlow, such Einstein operations can be simply implemented using rearrange function from einops library. Once NMF is applied to the resulting batch of matrices, the inverse operation of Local Matricize, called Local Dematricize, must be applied to transform them back to the initial shape. Local Dematricize can be easily obtained by composing the inverses of sub-operations in reverse order. In practice, Local Dematricize can also be formulated and implemented as an Einstein operation. A PyTorch-style pseudocode of Local (De)matricize module is provided in Algorithm 3. Note that while Local Matricize seems to reshape an image in a similar way to the input patchifier of ViT, it concatenates patches along batch dimension rather than channel dimension. In fact, Local Matricize can be used when modeling within-patch interactions is desirable, which is different from ViT-based approaches, where interactions between embedded patches are typically modeled.
Datasets We evaluate the effectiveness of our models on the Brain Tumor Segmentation (BraTS) dataset  (Menze et al. , 2014; Bakas et al. , 2017)  from Medical Segmentation Decathlon  (Antonelli et al. , 2021)  and Ischemic Stroke Lesion Segmentation (ISLES) 2022 dataset  (Petzsche et al. , 2022)  from a MICCAI 2022 challenge. BraTS. This dataset consists of 484 multiparametric MRI (mpMRI) scans from patients diagnosed with either low-grade glioma or highgrade glioma (glioblastoma). Each scan comes with four 3D MRI sequences, namely T2 Fluid-Attenuated Inversion Recovery (FLAIR), native T1-weighted (T1), post-Gadolinium contrast T1-weighted (T1Gd), and T2-weighted (T2). Once images are preprocessed (i. e. , rigidly coregistered to the same anatomical template, resampled to the same voxel spacing 1 mm 3 , and skull-stripped), the ground truths are manually created by experts who label each voxel as enhancing tumor (ET), edema (ED), necrotic and non-enhancing tumor (NCR/NET), or everything else.
ISLES'22. This dataset is from the ISLES'22 challenge, which aims to evaluate automated methods of acute and sub-acute stroke lesion segmentation in 3D multiparametric MRI data, namely DWI, Apparent Diffusion Coefficient (ADC), and FLAIR sequences. The DWI and ADC images of a patient are aligned while the FLAIR image in its native space has a different voxel size and must be registered to the DWI space.
Setup All the models were implemented using PyTorch  (Paszke et al. , 2019)  and MONAI  (Consortium, 2020)  frameworks and trained on NVIDIA P100 GPUs. We followed the same training workflow in all the experiments. In the following, we first provide the details of this workflow and baseline models, then present the evaluation protocol and the results. Preprocessing. For each scan in a dataset, a multi-channel 3D image as the input was first constructed by concatenating the modalitiesi. e. , FLAIR, T1, T1Gd, and T2 for BraTS, and DWI and ADC for ISLES'22. The image and its ground truth were then cropped with a minimal box filtering out zero regions of the image. The image was normalized channel-wise using a z-score to have intensities with zero mean and unit variance. Random patches of size  (128, 128, 128)  for BraTS and (64, 64, 64) for ISLES'22 were extracted during training.
HD95 is a more robust version to outliers, which calculates the 95% quantile rather than the maximum of surface distances. Models. In all the experiments, for Factorizer models with the overall architecture illustrated in Fig. 1 , the number of output channels of the stem was  = 32. For Local and Swin Factorizer, we used a large window size of (8, 8, 8) and (4, 4, 4) on the BraTS and ISLES'22 datasets, respectively, to aggregate local information, which is opposed to typical CNNs comprising convolutions mostly with a small receptive field of size  (3, 3, 3). For all Factorizer models, a head dimension of  = 8 on BraTS and  = 4 on ISLES'22 was used. In NMF modules, the factor matrices were initialized with uniform distribution  (0, 1). In training, we used a rank-one approximation ( = 1) with  = 5 outer iterations of HALS (which is equivalent to MU for  = 1), as described in Section 3.
Ischemic stroke lesion segmentation (ISLES) Quantitative evaluation. Similarly to Section 4. 3. 1, 5-fold crossvalidation was performed, and pairwise Wilcoxon signed-rank tests were used for comparison purposes. The results on the ISLES'22 dataset are reported in 2 and illustrated by box plots in Figs. 3(c ) and 3(d). Swin Factorizer, with a Dice score of 76. 49% and HD95 of 11. 96 mm, demonstrated the best performance, significantly outperforming nnU-Net and all the Transformer-based models with a -value of < 0. 001 for the Dice score and < 0. 01 for the HD95 value. Moreover, Swin Factorizer showed improved performance compared to Res-U-Net, which has over three times more parameters and needs over four times more FLOPs. Despite having over 95% fewer parameters and taking over 50% fewer FLOPs, Local Factorizer yielded a Dice score of 74.
Ablation studies: Training We conducted ablation studies on Factorizer to further investigate the effects of Factorizer subblocks and positional embedding. In this section, models with an ablated layer were trained from scratch on the BraTS dataset. Factorizer subblock. Ablation results of subblocks on brain tumor segmentation are reported in Table  3. When we removed NMF subblocks from a Factorizer model, the performance substantially dropped while Factorizer models without MLP subblocks demonstrated less deterioration in results. Local Factorizer without MLP blocks yielded an average Dice of 82. 67%, still outperforming nnU-Net, and Swin Factorizer without MLP blocks yielded an average Dice of 83. 57%, which is significantly greater than that of nnU-Net and comparable with that of Res-U-Net.
Ablation studies: Inference Since the output of an NMF layer is a low-rank approximation of the input, it makes sense to perform an ablation study in the inference phase, for instance by short-circuiting an NMF layer. For all the experiments in this section, we ablated some NMF layers or changed their settings in the inference phase after training the model on the BraTS dataset. NMF layer. We investigated the impact of short-circuiting some NMF layers of the pre-trained Swin Factorizer model in the inference phase. In Fig. 8(a) , the average Dice score on BraTS is shown when we kept the first NMF layers and removed (or short-circuited) the rest. As expected, the more layers we kept, the higher the Dice score was achieved. We observed that most of the performance is achieved via the encoder, which includes the first five blocks.
• Shifted Window (SW) Matricize operation is introduced and combined with NMF to fully exploit local contexts. • Scalable interpretable U-shaped segmentation models based on NMF are proposed. • The proposed models achieve state-of-the-art results on the BraTS and ISLES'22 datasets. Notation. We denote vectors by boldface lower-case letters, e. g. , , matrices by boldface upper-case letters, e. g. , , and tensors by boldface calligraphic letters, e. g. , . Elements in a matrix (tensor) are denoted by [, ] ( [ 1 , … ,   ]). The th row and th column of a matrix is denoted by [, ∶] and [∶, ], respectively. A sequence of  vectors (a. k. a. tokens) is denoted by (  )  =1. We use [ 1 | … |  ] to denote a matrix  created by stacking   s along the columns. We show the inner product between matrices by ⟨, ⟩ = ∑ , [, ][, ] and  2 -norm of a matrix by ‖‖ =.
Comparison of different models on the BraTS dataset. Scores are obtained by 5-fold cross-validation. The best results are in boldface and second best ones are underlined. The Dice score and 95% Hausdorff distance of different models on the BraTS dataset. The asterisks indicate statistical significance according to pairwise Wilcoxon signed-rank test (Swin Factorizer is the clear overall winner in the brain tumor segmentation task. With an average Dice score of 84. 21% and HD95 of 6. 89 mm, Swin Factorizer significantly outperformed Res-U-Net, the best CNN-based and second-best overall baseline, with p-values of < 0. 01 for the average Dice score and < 0. 05 for the average HD95, while requiring six times fewer computations. Swin Factorizer was the best-performing model on ET and the second best-performing model on TC and WT, particularly yielding the highest Dice score of 79.