dataesr/scientific-paragraphs-categorization huggingface dataset:
Control gene selection bias on Tirosh-based gene signature scoring The Tirosh-based scoring methods rely on the selection of control genes. However, Seurat, Seurat_AG, Seurat_LVG, and Scanpy can introduce biases in scoring when there is substantial variability in gene expression averages within expression bins, which is especially visible in the top 8% of the highest expressed genes (Suppl. Figure  S1 ). The ANS method was motivated to reduce the issue of bias induced by control genes. We compared the strategies for the control gene selection of the methods ANS, Seurat, Seurat_AG, and Seurat_LVG across the four preprocessed single-cell-type PBMC sub-datasets: B memory kappa, CD8 TM2, CD14 Mono, and NK 3 (Dataset preprocessing in Methods). For each dataset, we scored each gene in the top 8% of the highest expressed genes and, thus, converted the expression matrix to a score matrix.
The influence of the cell type proportions and batch effects on gene signature scoring The two following experiments explored the influence of dataset composition on gene signature scoring. The first experiment investigated the influence of different cell type proportions. The second experiment tested the influence of batch effects on scoring methods. Both experiments used the preprocessed CRC dataset and a 100-gene signature for malignant cells (Supplementary Methods) with the smallest adjusted p-values and the highest log 2 FCs. The CRC dataset contains samples sequenced with two different sequencing chemistries. We used the different sequencing chemistries as surrogates for batch effects. We scored each sample individually and the entire dataset. For each scoring method, we averaged the cell scores for each sample, cell type (malignant or non-malignant), scoring mode (scoring samples individually or all together), and sequencing chemistry (SC3Pv2 or SC3Pv3).
We compared the variance between the sample score averages for each scoring method and scoring mode in the first experiment and observed reduced or equal variance when scoring on the entire dataset. In order to evaluate the impact of batch effects, we utilized Mann-Whitney U testing to compare the sample score averages of malignant cells between the two sequencing chemistries for each scoring method and scoring mode. We used the Python package statannotations 64 (version 0. 4. 4. ) to compute the Mann-Whitney U tests and add statistical annotations in Figure  1d. Scoring samples individually showed significant differences in score ranges between chemistries. We performed identical experiments on the preprocessed ESCC dataset, excluding the batch effect component because of using uniform sequencing chemistries for all samples in those datasets (Suppl.
Comparable score ranges To assess the information quantity and ranges of scores when scoring for multiple signatures, we considered two tasks on the PBMC dataset  15. The dataset included cell type annotations at three granularity levels, with level one being the broadest cell type level and level three the one with the most subtypes. We used the published table  65  of differentially expressed genes per subcell type level 3 to get the cell-type specific signatures. For the easy task, we selected B cells, Monocytes, and NK cells (cell types on level 1) and preprocessed them as described in the first section of Methods. We chose the cell type-specific signatures for the cell types on level 1 of the hierarchy by unioning the sets of differentially expressed genes belonging to all subtypes of finer granularity.
Classification of cancer EMT cells As a first step in the signature establishment, we selected cancer cells expressing EMT, called cancer EMT cells. We computed scores for ESCC with the dataset-specific mesenchymal signature 1. We scored the entire dataset and the subset of malignant cells. Cancer cells being within the 10% highest scoring cells on the entire dataset and in the subset were classified as cancer EMT cells. The 3'860 cancer EMT cells out of 41'399 cancer cells (9. 32%) and 178'109 total cells (2. 17%), stem from 53 out of 54 samples. For LUAD and CRC, no dataset-specific mesenchymal signatures were reported. Therefore, we selected the cancer cells expressing EMT as follows. We scored the datasets for seven pan-cancer EMT signatures: Hallmark EMT signature  19  , pEMT gene module  20  , and five other EMT signatures  [21] [22] [23] [24] [25].
The pan-cancer EMT signatures were found on EMTome  67. For each pan-cancer signature, we ranked the cancer cells by ANS scores and computed for each cell the median rank. Finally, 10% of cancer cells with the smallest median signature ranks were classified as cancer EMT cells. We selected the median to avoid outlier ranks of the pan-cancer signature. For the breast carcinoma dataset, we used the program labels provided by the original paper  18  and excluded cells falling in the "grey zone" of the EMT program expression ("GM3" in the original paper). We considered only samples assigned to the "Basal" subtype and defined the "grey zone" to be cells not labeled with "GM3", but having a signature score for GM3 above 0. 15 (scoring was applied on all samples similar in the original paper). In the LUAD dataset, 822 out of 8'205 cancer (10.
02%) and 55'659 total (1. 48%) cells were classified as EMT. All 19 samples contained cancer EMT cells. In the CRC dataset, 4'328 out of 43'252 cancer (10. 01%) and 140'090 total (3. 09%) cells were classified as cancer EMT cells distributed over all 60 samples. In the breast carcinoma dataset, 2031 out of 5115 cancer (39,71%) and 27458 total (7,40%) cells were classified as cancer EMT cells distributed over all 7 samples. Dataset compositions are shown in Suppl. Figure  S8.
Establishment of a cancer-specific EMT signature in ESCC and LUAD The signature establishment consisted of finding differentially expressed genes between cancer cells expressing EMT and CAFs as well as cancer cells not expressing EMT in ESCC and LUAD. The cancer EMT cell classification for all datasets is provided in the supplementary information. We first started by finding a cancer-specific signature for ESCC, then found one for LUAD. Finally, we unioned the ESCC and LUAD-specific cancer EMT signature and refined it again on ESCC. For differential gene expression (DGEX) analysis between two cell groups of interest, we used Wilcoxon rank-sum (Mann-Whitney U) tests  68  with tie correction and Benjamini-Hochberg correction implemented in Scanpy's rank_genes_groups. Tests applied on scores for a single gene during refinement were done as described in the supplementary methods.
Cancer-specific EMT signature for ESCC: First, we created two subsets of the ESCC dataset: cancer EMT cells and CAFs and all cancer cells. Let be the set of DGEX genes ( and an adjusted 𝐴 𝑙𝑜𝑔𝐹𝐶 ≥ 2 p-value ) between cancer EMT cells and CAFs. Let be the set of DGEX genes ( < 0. 001 𝐵 and an adjusted p-value ) between cancer EMT cells and cancer cells not 𝑙𝑜𝑔𝐹𝐶 ≥ 1. 5 < 0. 001 expressing EMT. We joined sets and , called gene set , and removed all genes from that are 𝐴 𝐵 𝐶 𝐶 DGEX ( and an adjusted p-value ) in T-and Myeloid cells versus cancer EMT 𝑙𝑜𝑔𝐹𝐶 ≥ 2 < 0. 001 cells. The reduced gene set , called gene set , scored high for many cancer cells and thus required 𝐶 𝐷 more refinement to distinguish the cancer EMT cells from the other cancer cells. Let the set be all 𝑋 cancer cells that scored above for the gene set.
We selected all DGEX genes ( 0. 2 𝐷 𝑙𝑜𝑔𝐹𝐶 ≥ 1. 5 and an adjusted p-value ), called gene set , in between cancer EMT cells and cancer cells < 0. 001 𝐸 𝑋 not expressing EMT. We scored each gene in (n=451) independently and kept genes with 𝐸 -𝐷 significantly higher scores for cancer EMT cells than for CAFs, cancer cells not expressing EMT, and the rest of the cells (Supplementary Methods). We unioned the gene sets and , resulting in our 𝐷 𝐸 ESCC-specific cancer EMT signature. The detailed procedure can be found in the Jupyter Notebook find_cancer_emt_signature_ESCC. ipynb in the Supplementary material GitHub repository. Cancer-specific EMT signature for LUAD: For the LUAD-specific cancer EMT signature, we followed a similar procedure as for ESCC. First we created two subsets of the LUAD dataset: cancer EMT cells and CAFs and all cancer cells.
Joining cancer-specific EMT signatures for ESCC and LUAD and refinement on ESCC: To obtain a signature that can distinguish between cancer EMT cells and CAFs, and cancer EMT cells and cancer cells that do not express EMT, we unioned the ESCC-and LUAD-specific cancer EMT signatures. We scored for each gene in the unioned signature in ESCC (n=267) and only kept a gene if it had significantly higher scores for cancer EMT cells than for CAFs, cancer cells not expressing EMT, and the rest of the cells (signature refinement as described in Supplementary Methods). We excluded the insignificant genes and incorporated the LUAD-specific cancer EMT signature genes absent in the preprocessed ESCC dataset and the ESCC-specific cancer EMT signature genes absent in the preprocessed LUAD dataset. Suppl. Table  S3  shows the 135 ESCC-and LUAD-specific cancer EMT signature genes.
Data availability The preprocessed datasets for CRC, ESCC, LUAD, breast carcinoma, and PBMC and the used signatures can be downloaded: https://drive. google. com/drive/folders/10L2gqapJbyOn_MbrZRHQG--n0Xj7wIyg?usp=sharing. The raw transcript count matrices can be found at GEO: GSE178341 for CRC, GSE176078 for breast cancer, and GSE160269 for ESCC. The raw transcripts for LUAD can be found with the accession number HRA000154 at http://bigd. big. ac. cn/gsa-human. The raw PBMC dataset can be downloaded at https://atlas. fredhutch. org/nygc/multimodal-pbmc/. TCGA pan-cancer normalized RSEM gene expression data was downloaded from the GDC portal (htts://portal. gdc. cancer. gov/ ). TCGA clinical information was retrieved from Liu et al. 69.
The bias of a scoring method is indicated by how far the mean score of a gene deviates from zero. d, The influence of dataset composition and batch effect on scoring CRC cells using a 100-gene signature associated with malignant cells. Each dot represents the mean score for all cells within a sample, grouped by cell type (malignant in blue or non-malignant in orange), sequencing chemistry type (SC3Pv2 in grey or SC3Pv3 in beige), and scoring mode (scoring all the samples together or individually). The black horizontal bar represents the mean value of all dots within each group. e, The minimum required signature length per scoring method for perfect classification (AUCROC of 1) when discriminating between malignant and non-malignant cell scores in CRC. All scoring methods, except Jasmine_OR, need at most 13 genes for perfect discrimination.
Visual Question Answering (VQA) is a complex task requiring large datasets and expensive training. Neural Module Networks (NMN) first translate the question to a reasoning path, then follow that path to analyze the image and provide an answer. We propose an NMN method that relies on predefined cross-modal embeddings to "warm start" learning on the GQA dataset, then focus on Curriculum Learning (CL) as a way to improve training and make a better use of the data. Several difficulty criteria are employed for defining CL methods. We show that by an appropriate selection of the CL method the cost of training and the amount of training data can be greatly reduced, with a limited impact on the final VQA accuracy. Furthermore, we introduce intermediate losses during training and find that this allows to simplify the CL strategy.
INTRODUCTION Visual Question Answering (VQA) consists in answering potentially complex questions regarding the content of images. Datasets like VQA2. 0  (Goyal et al. , 2017)  and GQA  (Hudson and Manning, 2019)  were put forward in support of this task. These datasets are very large, leading to expensive training. However, since they are built from collected real images with (possibly assisted) human labeling, they inevitably contain many biases. Integrated approaches like  (Xiong et al. , 2022; Wang et al. , 2021)  have the highest overall accuracies on these databases but are prone to taking bias-promoted "shortcuts", as shown e. g. by their lower performance on out-of-distribution data  (Kervadec et al. , 2021). Furthermore, integrated approaches lack transparency in the reasoning process, even though some limited explanations can be obtained by following the flow of attention.
Alternatively, Neural Module Networks (NMN) were introduced  (Hu et al. , 2017)  with the aim to make the reasoning explicit. They were quite successful for visual reasoning on synthetic image datasets like CLEVR, where word grounding is comparatively easy and there is significant control over scene composition. NMNs are nevertheless hard to train on real images where grounding is difficult, attributes are more diverse and data biases are hard to control. To make learning more effective and less expensive, we rely on NMN but employ predefined cross-modal embeddings to "warm start" the training process on GQA, then explore Curriculum Learning to improve learning of such a complex task and reduce both the cost and the amount of data required. Curriculum Learning (CL)  (Elman, 1993; Soviany et al.
Visual question answering VQA is usually addressed with either integrated cross-modal frameworks or compositional neural module networks. Cross-modality Transformers. Transformer networks  (Vaswani et al. , 2017)  have been widely applied to multiple language and vision tasks, and they have been recently adapted for reasoning problems such as VQA. Models like ViLBERT  (Lu et al. , 2019) , VisualBERT  (Li et al. , 2019b)  and LXMERT  (Tan and Bansal, 2019)  showcased good performances on the VQA datasets VQA2. 0  (Goyal et al. , 2017)  and GQA  (Hudson and Manning, 2019). These frameworks start by extracting the text and image features: word embeddings are obtained via a pretrained BERT  (Devlin et al. , 2019)  model, while Faster R-CNN  (Ren et al. , 2015)  produces image region bounding boxes and corresponding visual features.
, 2017; Li et al. , 2019a)  perform multi-hop reasoning by decomposing a complex reasoning task into several easier subtasks. An NMN consists of a generator and an executor. The generator maps a question to a sequence of reasoning instructions (called a program). The executor assigns each sub-task from this program to a neural module and passes the results to the next modules. In  (Chen et al. , 2021)  a meta-learning approach is employed in the NMN framework to improve the scalability and generalization of the resulting model. The generator decodes the question into a program whose sub-tasks are used to instantiate a meta module. The image features are extracted by a visual encoder implemented as a transformer network and a cross-attention layer mixes word embeddings and image features. While the combination of a generator and an executor in NMNs appears more complex than an integrated model, the "hardwired" reasoning process of an NMN is inherently transparent and has the potential to avoid part of the reasoning "shortcuts" caused by data bias.
Interestingly, it was shown in  (Kervadec et al. , 2021)  that by using the programs resulting from questions as additional supervision for the LXMERT integrated model allows to reduce sample complexity and improve performance on GQA-OOD. In our work, we aim to take advantage of both the transparency of NMN architectures and the quality of transformer-encoded representations by implementing a composable NMN over multimodal transformer vision and language features.
The authors implement several heuristics reminding of active learning in order to improve SPL performance. Curriculum Learning for VQA. The definition of relevant difficulty criteria for VQA is challenging and this may explain why there is little work on the use of CL for VQA. The recent work in  (Askarian et al. , 2021)  applies CL in a modular VQA context to the synthetic CLEVR dataset  (Johnson et al. , 2017a). The base model is from  (Johnson et al. , 2017b) , with an LSTM generator and generic residual blocks for the executor modules. The experiments were conducted on the executor alone, using as input the ground-truth programs directly. Several difficulty criteria were evaluated, including program length, answer hierarchy, and question loss. The results demonstrated that CL with a question loss difficulty criterion has a positive impact in a low data setting.
However, the study in  (Askarian et al. , 2021)  was focused on the CLEVR dataset  (Johnson et al. , 2017a)  consisting of synthetic images of simple 3D objects, with a limited number of classes or attributes and reliable object detection. In our work, we employ the GQA dataset that is based on real-world images with many classes and several names for some of them, as well as more complex relations and more challenging object detection. We thus have to completely redefine the candidate CL strategies.
To extract crossmodal language and vision representations we rely on LXMERT  (Tan and Bansal, 2019) , a transformer model pretrained on multiple multi-modal tasks such as Masked Cross-Modality language models, masked object predictions, Cross-Modality Matching, and VQA. LXMERT showcases high accuracy on the training tasks, so we employ it as a feature extractor. It is worth mentioning that we only use the cross-modality encoder representations and discard the answer classification component. More precisely, we freeze LXMERT weights and we pass the image I through the object-relationship encoder and the question Q through the language encoder. Then, the Cross-Modality Encoder aligns the representations to finally output the object bounding box features v j of each object o j in the image I and the embedding h i of each word q i in the question Q.
Boolean modules such as And or Or make logical inferences and answer modules such as QueryName give a probability distribution over the answer vocabulary. Table  1  shows an example from each module category. An exhaustive module list with definitions is provided in the appendix. Table  1 : Sample module definitions. S: softmax, σ: sigmoid, r: RELU, W i : weight matrix, a: attention vector (36 × 1), V: visual features (768 × 36), t: text features (768 × 1), ⊙: Hadamard product.
The program executor is responsible of managing module dependencies by using a memory buffer to save the outputs that serve as inputs for the next modules. The design insures that a module can have at most two dependencies. The generic modules require a textual argument a m,i to determine which facet of the module to use, for example, the FilterAttribute module can be called for multiple attribute categories such as color and size. For instance, to filter the red objects we use the argument word "red" and input its cross-modal embedding to the module. Module execution supervision. The executor network takes as input an image in the form of a list of object bounding boxes and their LXMERT embeddings. Every executor network ends with an answer module, the output of which is compared to the ground-truth to compute the output loss.
Experimental setup GQA Dataset. GQA  (Hudson and Manning, 2019)  features over 18M compositional questions and 113K real-world images. The reasoning steps of the questions are represented by functional programs. The questions and programs are generated by a question engine from the corresponding image graph. The image graphs are composed from ground-truth object bounding boxes together with their names, attributes, and relations. GQA is based on the Visual Genome dataset. The GQA dataset has two versions: a balanced version (with a uniform distribution over the answers) and an unbalanced one. (Tan and Bansal, 2019) , this combination also allows us to get over 1M examples when training on the balanced set. We follow the recommendations of the dataset authors by evaluating the performance on the test-dev split instead of the val split when using the object-based features because they were trained on some images from the val set  (Anderson et al.
, 2018). In the GQA dataset, each question/image pair in the train, val, and testdev sets is associated with a functional program. The programs use 124 distinct modules, some of which only correspond to very few questions. We consider that modules should only differ when they correspond to operations that are different. We thus group specific modules into more general ones. For example, modules like ChooseHealthier and ChooseOlder are grouped in a ChooseAttribute module. This results in only 32 modules, the list of which is presented in the appendix. Metrics. Several metrics are employed to compare CL and standard learning. Since our focus is on reducing the cost of training, we measure the total number of example presentations during training (Comp. cost). We also mention the maximal number of different examples seen during training (# examples), as different methods can make use of larger or smaller portions of the training set.
While we focus on the cost of training, we nevertheless want to reach an accuracy that is close to the one obtained by standard learning, so we also report the accuracy of the predicted answers. -CL: The model is trained using Curriculum Learning and the filtering is driven by the number of objects in the programs. At every CL iteration the model sees 1M examples filtered from the unbalanced dataset by the curriculum sampler. The training needs 4 CL iterations to be complete, where each iteration has an increased difficulty given by the number of objects of its programs (ranging from 1 to 4).
Evaluated methods -Length (L): The curriculum sampler filters the programs by their lengths for every number of objects, a CL-iteration is defined by a number of objects and a program length (short or medium or long). -Weights (W): We use several sampling weights for the filtered programs: 'uniform' indicates that the sampling is uniform, so the sampling with replacement results in a sample that is uniformly distributed over all the dataset. To make the answer modules distribution of the resulting sample more uniform we use the 'answer module' weighting (denoted by W. a); this balances the answer modules appearances in the result sample so that the model equally sees all the defined answer modules. The 'modules loss' weighting (denoted by W. b) indicates that the example's weight is proportional to the sum of the average losses of the modules composing its program, to focus the model on hard examples.
Implementation details We perform the experiments using the pre-processed GQA dataset programs and we focus on investigating the effect of several CL policies on the Program Executor module. Although our system also uses a transformer model as a generator to translate the question to its corresponding program, its task is relatively easy compared to the executor training and, similar to previous works  (Li et al. , 2019a; Chen et al. , 2021) , we achieve near perfect translation results on the testdev-all set. We employ LXMERT as a feature extractor and freeze its weights. LXMERT image inputs are the object bounding boxes provided by a Faster R-CNN object detection model  (Anderson et al. , 2018) , where the number of bounding boxes per image is fixed to 36. We feed the questions and their corresponding object bounding boxes to LXMERT; the encoding yields 36 object features and the question word embeddings for each question/image pair.