dataesr/scientific-paragraphs-categorization huggingface dataset:
Transfer learning Our first experiment is transfer learning. We use the pretrained networks (see Section 2) with frozen weights as encoders, and train a new classification head for each of our datasets in unimodal or multimodal settings. For transfer learning in single-label classification (sentiment on MVSA, book genre), we plug on top of the frozen feature vector encoder one dense layer (ReLu activations) bringing the dimensions down to 256, and then another dense layer (softmax activation) for the classification. We then train only the weights of these 2 layers on the classification task with a Cross-Entropy Loss; therefore the network learns to output a probability density over the classes. For multi-label classification (movie genres) the loss is a Binary Cross-Entropy Loss, and therefore the second dense layer outputs a number between 0 and 1 for each class.
As the ground-truth label vector for one sample is an 19-dimensional one-hot vector, we round the 19-dimensional prediction of the network to get a binary predicted label vector. A f1-score  (Pedregosa et al. , 2011)  comparing the predicted label vector to the ground-truth vector is reported, as raw accuracy is not a reliable measurement for multi-label classification. The f1-score is computed for each movie, and subsequently averaged over the test set of each dataset. For f1-scores, as for accuracy, the higher the better. Tables  1  and 2  show the results on the singlelabel datasets: MVSA and Book Cover. The first column corresponds to the result of the vision-only experiment, the first line to those of the text-only experiments, and the other cells display the results of the multimodal ones.
The difference between CLIP's accuracy (53. 8%) and the other two (RN50: 10. 0%; BiT-M: 29. 3%) remains high in the masked configuration (with CLIP at 33. 0% and the other two below 25%), even though CLIP has lost the ability to read the text on the covers. This indicates that CLIP's reading ability is not the sole explanation for its advantage over vision-only models. CLIP-T is again the best text-only model. Here, the best multimodal combination is CLIP+CLIP-T for both standard and masked configurations. Finally, compared to previously established SOTA performance on the Book Cover dataset by  Lucieri et al. (2020) , CLIP easily beats the previous visual SOTA (27. 8 % accuracy), CLIP-T the previous textual SOTA (55. 6%), and CLIP+CLIP-T the previous bimodal SOTA (55. 7%). Concerning our new Plotster dataset (Table  3 ), similar conclusions emerge.
In vision-only conditions, RN50 performs relatively poorly; in the standard dataset, CLIP largely outperforms BiT, and this difference decreases but remains in the masked dataset. In text-only conditions, CLIP-T is the best model, both with titles and plots as input. Finally, in the multimodal settings, CLIP+CLIP-T is always the best-performing combination, whether using standard or masked images, title or plot as textual inputs. As before, the prevalence of CLIP in all task settings, even when text has been removed from the movie posters, indicates that its superior-ity in our movie genre transfer learning task is not solely due to its reading ability. We surmise that this advantage reflects a form of semantic grounding resulting from CLIP's multimodal training. We also tested CLIP, CLIP-T and their combination on a subset of Plotster corresponding to the dataset of  Mangolin et al.
Nevertheless, CLIP and CLIP-T remain the top-performing models; as it is unlikely that these recent movie posters and captions had been included in CLIP's training dataset, we conclude that CLIP's high transfer-learning performance on Plotster is not a consequence of prior exposure to these stimuli, but a true form of generalization. In general, we see that in all the unimodal settings, CLIP outperforms the other vision models, and CLIP-T the other text models. This is true, even though CLIP has roughly the same number of parameters than RN50 or BiT-M, and fewer dimensions in its latent space (and thus, less parameters in its classifier head). Similarly, CLIP-T counts much fewer parameters than Bert-base or Bert-large (although it has a higher-dimensional latent space than Bert-small). In most of the multimodal settings, changing from one visual model to CLIP or from one textual model to CLIP-T improves performance (the only exceptions are for CLIP-T on MVSA and on Plotster with plots as text inputs).
Few-shot learning The second experiment we conduct is a visual fewshot learning task: we measure test classification accuracy based on exposure to a small number of randomly chosen training samples (or "prototypes") from each class. We can thus compare the results for our datasets with those of  Devillers et al. (2021) , who also measured visual few-shot learning performance. In their paper, Devillers et al. (  2021 ) used a single prototype vector for each class, obtained by averaging the latent representation of the N randomly drawn training samples for that class. Here, we prefer to retain all N individual samples as prototypes, and use a 1-nearest-neighbor (1-NN) classifier  (Pedregosa et al. , 2011)  to classify the new vectors. We verified that this method, when applied to the same datasets as in  (Devillers et al.
, 2021) , does not alter their conclusion (see the first plot of Figure  3 ). To select the class prototypes of Plotster (which is a multiclass dataset), we randomly choose movies with a given class label. For example, a movie with genres "adventure" and "action" could be randomly chosen as a prototype of either genre. Moreover, when predicting the genres of a movie using the 1-NN classifier, we predict all the genres of the closest prototype. Figure  3  reports the few-shot accuracy on the Book Covers and MVSA datasets as well as the f1 score for the Plotster datasets. Contrary to the conclusion of  Devillers et al. (2021)  using standard visual datasets (see Figure  3 , left), our results show a clear advantage to CLIP in our more "humancentric" visual tasks, even when masks are applied.
Summary In the visual domain, CLIP systematically outperforms the unimodal vision models in transfer learning (Tables  1 2 3 ) and in visual few-shot learning (Figure  3 ), despite having a smaller embedding space than the other two ResNet50-based models. Part of CLIP's superiority may be due to its ability to read, but the advantage remains when text is removed from the images. This conclusion goes   2021 ) -namely CIFAR10, CIFAR100, CUB, FashionMNIST, MNIST and SVHN. Accuracy was recomputed using the same method as for our datasets; the conclusions are identical to those of  Devillers et al. (2021) : CLIP does not perform better than RN50 or BiT in this few-shot learning setting. On the contrary, for our datasets CLIP outperforms the two other vision models. The advantage is reduced but still present when masks are applied.
against the observations of Devillers et al. (  2021 ) using standard visual datasets (including SVHN, a digit reading dataset), where CLIP was never better (and often slightly worse) than other ResNet50 based models, including RN50 and BiT-M. We explain this difference by the nature of the classification performed: our tasks involve human-centric concepts, as defined earlier. In the text domain, CLIP-T, despite having been trained with fewer parameters than the other two transformers (Bert-small and Bert-large), is systematically the best performing model in transfer learning. Across seven multimodal settings (MVSA dataset; Book Covers dataset [with / without masks]; Plotster with [titles / plots] × [with / without masks]), CLIP+CLIP-T was the best multimodal combination in six cases. In the remaining case (MVSA), it was a tie between CLIP+Bertlarge and CLIP+Bert-small (two language models that count many more parameters than CLIP-T).
This means that the representation of text in CLIP-T has been enriched with visual data, and symmetrically, that the representation of images in CLIP has been improved by semantic or linguistic enrichment. This is what is collectively referred to as the "semantic grounding" property  (Harnad, 1990; Bender and Koller, 2020). However, another consequence of this multimodal contrastive training is that when learning a common ground between modalities, some relevant information could be lost. For text, what cannot be directly linked to images (including grammatical or syntactic properties); and for images, what is not directly relevant to the text description (including fine-grained visual details that are rarely mentioned in the corresponding caption). This information loss might be the reason why CLIP was found to perform worse than standard vision-only models in a unimodal setting with standard visual datasets  (Devillers et al.
∅. 314 ±. 01. 323 ±. 01. 397 ±. 00. 582 ±. 00. 599 ±. 01. 612 ±. 00 RN50. 090 ±. 01. 338 ±. 01. 363 ±. 01. 393 ±. 02. 578 ±. 01. 599 ±. 01. 599 ±. 01 BiT-M. 415 ±. 01. 490 ±. 01. 499 ±. 01. 507 ±. 01. 625 ±. 01. 637 ±. 01. 631 ±. 01 CLIP. 526 ±. 01. 559 ±. 01. 558 ±. 01. 593 ±. 01. 672 ±. 00. 683 ±. 00. 687 ±. 00 RN50. 070 ±. 01. 335 ±. 02. 352 ±. 01. 383 ±. 02. 576 ±. 01. 597 ±. 01. 596 ±. 01 BiT-M. 372 ±. 00. 457 ±. 02. 480 ±. 01. 490 ±. 01. 617 ±. 01. 631 ±. 01. 621 ±. 01 CLIP. 449 ±. 01. 525 ±. 01. 534 ±. 01. 564 ±. 00. 658 ±. 00. 667 ±. 00. 676 ±. 00 f1-scores for the Plotster dataset. CLIP is the best model in vision, CLIP-T the best in text whether titles or plots are given as input, and CLIP+CLIP-T is the best multimodal combination in all cases. The masking doesn't affect the advantage for CLIP.
Acknowledgements All book cover images and book titles are copyright  Amazon. com,Inc. The display of the images are transformative and are used as fair use for academic purposes. All posters displayed here are property of The  Walt Disney Company / Marvel Entertainment  and under the CC BY-SA 2. 0 license. This research was supported by  ANITI ANR  grant  ANR-19-PI3A-0004  and  COCOBOTS ANR-DLR bilateral French-German  project  ANR-21-FAI2-0005.
However, blockade of CXCR4 with AMD3100 significantly reduced human HPC content in the IF to homing levels seen with i. v. delivery or homing levels in the non-IF (Figure  3C ). Notably, homing of i. v. injected cells was not blocked with AMD3100, suggesting that other homing receptors may play a role here. Taken together, these data show that higher expression of CXCR4, not VLA4, on HPC confers a competitive homing advantage. An alternative explanation for the competitive homing and engraftment advantage of HPC could be that they comprise the vast majority of CD34 + HSPC (98%-99%), while HSC are a small minority. We experimentally conferred the HSC-enriched CD34 + CD38 -population with slightly higher CXCR4 expression than the CD34 + CD38 + HPC (Figure  3D ). CD34 + HSPC were transduced with a BFP-CXCR4-encoding LV vector or a BFP-encoding LV control vector, and CD34 + CD38 -BFP + HSC-enriched cells were sorted along with untransduced CD34 + CD38 + BFP -HPC.
We assessed the CXCR4 expression on these populations before transplant. In control cells, CXCR4 expression on the more primitive CD34 + CD38 -cells was much lower (mean fluorescence intensity [MFI] 16,930) than on CD34 + CD38 + HPC (MFI 35,098) (Figure  3E ), consistent with prior data (Figure  2D ). In contrast, CXCR4 expression on the HSC-enriched CD34 + CD38 -BFP + cells transduced with the BFP-CXCR4 LV vector was now higher (MFI 37,453) than the CD34 + CD38 + HPC (Figure  3E ). We then mixed the sorted BFP -HPC population with the BFP + CD34 + CD38 -CXCR4 hi HSC-enriched cell population in ratios seen in CD34 + HSPC, prior to sorting, and transplanted them either i. v. or i. b. m. ; we then assessed homing of the CD34 + CD38 -BFP + CXCR4 hi HSC-enriched cells. Control CD34 + CD38 -BFP + cells were also mixed with CD34 + CD38 + BFP -HPC and transplanted i.
Acknowledgments We would like to thank  Anastacia Loberg ,  Sarah McGraw ,  Kimberly Wagner , and  Scarlett Ripberger  for their assistance in animal experiments. We also thank the  Research Flow Cytometry Core at CCHMC  for assistance with FACS. This work was funded by  NIH   R01 DK102890 , the  Doris Duke Charitable Foundation ,  Advancing Cures  award to PM, and the  Charles H. Dater Foundation  award to PM. Address correspondence to:  Punam Malik , Division of Experimental Hematology and Cancer  Biology, Cincinnati Children's Hospital Medical Center , ML 7013, 3333  Burnet Avenue, Cincinnati , Ohio 45229, USA. Phone: 513. 636. 8588; Email: Punam. Malik@cchmc. org. AS's present address is:  Aruvant Sciences, New York, New York , USA. DMP's present address is: Genetic Counseling Graduate Program, The  Ohio State University College of Medicine, Columbus, Ohio, USA.
However, there is no consensus that this integration of structural information may help the network to extract relevant expression patterns and improve the prediction performance  [14]. Transfer learning is often proposed to tackle the problem of small-training set and the high dimension of the gene expression data. The term transfer learning refers to a set of techniques to transfer information from a model (source) to another one (target). Transfer learning is widely used in image analysis and natural language processing, where some common visual or textual patterns are helpful for any classification task. They can be extracted from a large source dataset and transferred to the target dataset for the classification task. There are different scenarios of transfer learning  [15]. We can distinguish supervised transfer learning from unsupervised transfer learning.
In the gene expression context, the standard approach is the unsupervised pre-training of the hidden layers of the MLP. This approach generally involves an autoencoder (AE) that compresses the gene expression profile into a small vector. Since the training of the AE is unsupervised, we can benefit from much larger datasets. Kim et al. train a variational autoencoder (VAE) from the pan-cancer TCGA dataset, then the hidden layers of the encoder are copied to the hidden layers of the MLP that predicts the hazard ratio of patient with a specific cancer  [16]. In  [17] , each hidden layer of a MLP for cancer prognostic prediction is initialized with a denoising auto-encoder (DAE) trained from a large pan-cancer dataset. Alzubaidi et al. pre-train their MLP with a sparse auto-encoder to predict cancer subtypes and identify biomarkers  [18].
Data and experiment design Our experimentation is based on two large datasets, including 22 classification tasks. The first one comes from a pan-cancer study of cross-experiments compiling the gene expression profiles from about 40,000 publicly available Affymetrix HG-U133Plus2 Fig. 1  All experiments performed in this study arrays  [20]. It combines different gene expression datasets containing diverse tissues and experimental protocols and integrates both patient and cell line data. The dataset is accessible via the ArrayExpress database (accession number E-MTAB-3732). After quality control and normalization, the dataset contains the expression of 54,675 genes. The samples whose status is not clearly defined are dropped. We only keep samples from the most present tissues in the dataset (tissues with more than 400 samples).
For each sample, the available information is its status (cancer/non-cancer), type (patient/cell line), and tissue. 12 classification tasks are associated with this dataset. It consists of predicting the presence of cancer in the pan-cancer case or in each of the 11 specific cancers. Note that the classes are unbalanced since cancer samples are three times more present than non-cancer samples. Table  1  gives the characteristics of this dataset. This dataset is divided into a training set of 13,000 samples and a test set of 3709 samples preserving the proportion of cancer/non-cancer samples. The second dataset comes from the TCGA portal, 1  a repository of multi-omics datasets containing only real patients with several types of cancer  [21]. We use the RNA-seq datasets containing at least 350 samples.
This dataset is much more homogeneous than the first dataset, and many clinical annotations are available. We investigate two classification tasks in a pan-cancer context. The first one is the prediction of the presence of cancer, where the classes are very unbalanced since 92. 7% of samples are labeled cancer. The second pan-cancer task is the prediction of the type of cancer. We also consider eight specific cancer tasks for the prediction of the presence of cancer. LGG, OV, and LIHC data are not used for this task since no non-cancer samples are available. Table  2  gives the characteristics of this dataset. This dataset is divided into a training set of 5000 samples and a test set of 1450 samples for the binary classification task and 980 samples for the multi-class classification task by preserving the proportion of cancer/non-cancer samples and the type of cancer.
A paired t-test tests the significance of these results. NN has the best accuracy on the three tasks. We note that on the TCGA dataset, the difference of accuracy between NN and others methods is non-significant for the cancer prediction task and small for the type prediction task. All classifiers obtain the same level of accuracy, except LASSO. Note that the TCGA classification tasks are easier to achieve when the size of the training set is 5000, the classifiers reach 99% of accuracy. There is no space for significant improvements. Concerning the microarray dataset, NN is significantly better than other classifiers (p value < 0. 001) with a large margin (from 94. 75 for SVMrad to 96. 18 for NN). The performance of the models is highly dependent on the number N of available examples for the training.
Supervised transfer learning We investigate the transfer learning approach to improve the performances of the NN  [15, 22]. There are two types of data in the context of transfer learning: the source and the target data. The target dataset represents the data of interest on which a classification task is defined. The idea of transfer learning is to transfer relevant information from the source model to the target model to make it more efficient. The classification task associated with the source data must be identical or related to the target task.
Between cell lines and patient data This subsection evaluates the benefit of supervised transfer learning between cell lines and patient data on the microarray dataset. Although we mixed these two types of data to learn an accurate model for cancer prediction in the previous section, it clearly appears that cell line data and patient data do not follow the same distribution. We highlight this point in the following experiments. A NN is only trained on the training cell line data and tested separately on the cell line and patient test data. The same experiment is done with a NN trained only from training patient data. All the procedure is iterated 10 times, this means that 10 NN are trained on cell line data and tested on patient data for each value of N and F The results presented in the Table  5  show that cell line data are easier to predict than patient data.
58%, and the fine-tuned model reaches 80. 74%. The baseline models learned nothing with a very small dataset; their accuracy is around the proportion of the majority class, whereas fined-tuned models provide more relevant predictions. Pre-training is beneficial for n < 200 for cell lines to patient transfer and n < 400 for patients to cell line transfer. After these thresholds, pre-trained models obtain the same accuracy as the baseline. However, we note that the convergence of the gradient descent during finetuning takes fewer epochs than in the baseline models training. The pre-training can be interpreted as an initialization of the NN to a point in the parameter space where the gradient descent will converge faster and more efficiently in a small training set scenario. We also notice that the transfer from patient to cell lines is more efficient than from cell lines to patient; the accuracy gain is up to 0.
Since the target data is limited to a specific type of tissue, the target dataset is small (from 371 to 4283). All accuracies are therefore estimated by stratified tenfold cross-validation. From pan-cancer to specific cancer This experiment evaluates the transfer from a model trained on all types of cancer to a specific one. For a given target type of cancer, all training examples except the examples of the target type are used to train a source model. Then, the source model is transferred to the target model that is fine-tuned with a subset of n training examples from the target data. The accuracy of the target model to detect cancer in the target data is evaluated and compared with the baseline model trained directly using the n training examples of the target tissues. Figures  6  and 5  show respectively the results on the TCGA and microarray datasets.
The accuracy of baseline and fine-tuned models is represented respectively in red and blue as a function of n. For each value of n, the significance of the difference of accuracy between baseline and fine-tuned models is estimated by a paired t-test. The one star (resp. double stars) symbol means that the difference is significant with a p value of 0. 05 (resp. 0. 01) and the color of the star indicates the best model (red for baseline, blue for fine-tuned). Transfer learning performance is not as good as in the cell lines-patients case. The effectiveness of transfer learning depends on the type of cancer. We identify three types of results. The first one concerns cases where transfer learning is beneficial, the accuracy of fine-tuned models is higher than baselines for small n. It gathers leukemias, breast, liver, lung, ovary for microarray data and BRCA, HNSC, LUSC, THCA, UCEC for TCGA data.
As in the previous experiment, the less data there is, the more useful transfer learning is. In the second type of results, transfer learning does not change the performances. The accuracy of fined-tuned and baseline models is equivalent. It concerns kidney, brain, colon for microarray data and KIRC, LUAD, PRAD for TCGA data. The last type presents negative transfer; the fine-tuned models have lower accuracy than baselines. It concerns only bone marrow and skin for microarray data. Note the case of the lung cancer in microarray data and HNSC, UCEC for TCGA data, where transfer learning is clearly beneficial for small N, but becomes negative for larger N. We point out that the negative transfer is very small in these special cases. The accuracy difference is less than 1% for HNSC and UCEC, and less than 2% for lung, which is much lower than the gain of accuracy for small N.
Unsupervised transfer learning This section evaluates the performance of unsupervised transfer learning on cancer prediction tasks. We consider a small target labeled dataset ( N ≤ 500 ) and a large source of unlabeled dataset ( N ′ = 10000 for microarray and N ′ = 3000 for TCGA). Each layer of the NN is successively trained through a denoising autoencoder (DAE) with the source dataset. Then, the model is fine-tuned with a subset of N target examples. The accuracy of this model to predict the presence of cancer is evaluated and compared with the baseline model trained directly from the N target examples. Figures  9  and 10  show the difference of accuracy between the baseline and fine-tuned model for each type of tissue. In these figures, we show only the performance of the pre-training procedure, i.