dataesr/scientific-paragraphs-categorization huggingface dataset:
Classification Results We first perform the classification task on the pan-cancer dataset. Each architecture is coupled with an artificial neural network classifier composed of two hidden layers with 256 and 128 neurons. We see that the constructed layer representation succeeds at separating the data into 4 distinct clusters that we couldn't distinguish with each omic source alone. c. PAM50 gene importance: Computed SHAP values on the RNA-Seq data of the most relevant genes responsible for the discrimination between each subtype against the others using CustOmics for both integration phases. Figure  6  and table 1 show the overall classification results (More details can be found in Table  S1 ). Among the factorial methods, the MFA achieved the best results, so we coupled this method with the same ANN classifier used with the deep-learning representation methods as a basis for comparison.
Late integration is not optimal either, since interactions between sources are not properly learned. Joint integration performs well in most cases, but we see that the best results displayed in Table  S2  are achieved by the combination of only two sources, RNA-Seq and Methylation data, as it seems that CNV data only adds noise to the latent representation, meaning that its information is not handled well with this strategy. Table  2 : Classification performance for PAM50 classification on the TCGA-BRCA dataset, with 5 standard metrics. We compare machine learning methods like UMAP, NMF, and MFA with deep-learning methods. We evaluate the performances on the final predicted output of the downstream classifier. These results confirm the interest of the CustOmics architecture, as it gives the best performances for all the test cases.
Survival Analysis The second task in this study is survival analysis. The goal is to predict the risk score associated with each patient from the corresponding high-dimensional omics data. Two standard metrics evaluate the performance of this downstream task: The c-index, which is a generalization of the AUC metric for censored data  [Harrell et al. , 1982] , and the Integrated Brier Score  [Graf et al. , 1999]. Figure  4  and table  3  show the results for the different methods (more details can be found in Table  S3 ) for the survival task. The same observations as for the classification task can be done regarding the differences between integration strategies. Here again, we also evaluated the performance of the CustOmics method for each combination of omics sources (Table  S3 ). The last task consists in evaluating the model performances for survival analysis for several specific cancer types of the TCGA datasets described in the dataset section.
The objective is to evaluate the robustness of the models when dealing with smaller datasets. Finally, we perform a more thorough analysis on the survival results that we display in Figure  4. We leave out 20% of our datasets for validation purpose, and we perform 5-fold cross-validation on the remaining 80% to compute the p-values associated to the log-rank test for different combinations of the Pan cancer test case. We show the ability of CustOmics to stratify the patients into distinct risk groups using the predicted hazard ratio. This stratification ability was also measured quantitatively using the p-value associated to the log-rank test between the different categories. Even Here again, our model outperforms the other integration strategies on both metrics. b. Log-rank test: We compute the p-value associated to the log-rank test between high and low risk groups for every integration strategy on a validation set for the pan-cancer survival test case, and we compare it to mono-omic survival predictions.
c. Kaplan Meier Curves : We draw the Kaplan Meier curves and display the p-value associated to the log-rank test as computed previously for each dataset using the predicted hazard from the CustOmics model and stratify the population into high and low risk on the test set for the predicted hazard ratio. This figure shows that our method succeeds in stratifying the patients into risk subgroups. though the comparison between joint, late and mixed integration is not significant in this case, it is interesting to note that the addition of multiple omics sources has greatly affected the p-value as it was nearly multiplied by a factor 10 -5 for CNV data and 10 -2 for RNA-Seq and methylation data. Those results also show that the early integration is the strategy with minimal enrichment from other sources, which corroborates our previous intuition and is coherent with the results found in the different experiments.
Our fusion model demonstrated better performances for both classification and survival outcome prediction across all test cases. It not only achieved great results on pan-cancer data, but it also did on smaller dataset for particular cohorts, showing the robustness of our method in situations with fewer samples. Interestingly, the performances of CustOmics in comparison to other integration strategies for different combinations of omic sources show that the mixed-integration handles better the fusion between CNV, RNA-Seq and methylation data. This aspect could probably be further enhanced by adding prior knowledge of those sources in the intermediate autoencoders, for example introducing a negative binomial prior in the RNA-Seq autoencoder. Furthermore, by adapting the SHAP method to our architecture, we were able to highlight genes of importance for specific tasks.
Conclusion In conclusion, our generic and interpretable multi-source deep learning framework improves on state-of-the-art integration strategies by proposing a hybrid approach that fits well with multi-omics data. The framework is available to use on Github: https://github. com/HakimBenkirane/CustOmics. Possible steps to refine CustOmics are a more thorough study on the interpretability part of the framework and adding prior knowledge to individual autoencoders to enhance the sub-representations of omic sources during phase 1 of mixed-integration.
Figure4: a. Survival Analysis Performances: We evaluate the performances of the survival model for the pan-cancer dataset using both C-index and the Integrated Brier Score (IBS). Here again, our model outperforms the other integration strategies on both metrics. b. Log-rank test: We compute the p-value associated to the log-rank test between high and low risk groups for every integration strategy on a validation set for the pan-cancer survival test case, and we compare it to mono-omic survival predictions. c. Kaplan Meier Curves : We draw the Kaplan Meier curves and display the p-value associated to the log-rank test as computed previously for each dataset using the predicted hazard from the CustOmics model and stratify the population into high and low risk on the test set for the predicted hazard ratio.
Classification performances for multiple combinations of omics data using CustOMICS on the Pancancer dataset. We can see that the best performances are brought by Transcriptomics data, but the addition of other omics data increases the performances, showing that the integration is relevant. ± 0. 01 0. 94 ± 0. 01 0. 94 ± 0. 01 0. 94 ± 0. 0132 0. 97 ± 0. 00 CNV + RNAseq + methyl 0. 98 ± 0. 01 0. 97 ± 0. 01 0. 97 ± 0. 01 0. 97 ± 0. 01 0. 99 ± 0. 00.
Classification performances for multiple combinations of omics data using joint integration on the Pancancer dataset. ± 0. 01 0. 96 ± 0. 01 0. 96 ± 0. 01 0. 96 ± 0. 0132 0. 99 ± 0. 00 CNV + RNAseq + methyl 0. 94 ± 0. 01 0. 94 ± 0. 01 0. 95 ± 0. 01 0. 94 ± 0. 01 0. 97 ± 0. 00.
] minimise the risk of unfair biases embedded in the model [. ]". If the lack of labeled training instances and fairness are complex problems individually, avoiding biases in a semi-supervised learning scenario is even more challenging. In a worst-case scenario, the few available labeled instances could be all or almost all associated to unfair sources, thus leading to very biased results or preventing any debiasing process. On the other hand, unlabeled instances do not carry any explicit bias and could be useful for driving the learning algorithm towards a fairer model. Despite its clear potential, fair semi-supervised learning has not been deeply investigated. The few existing approaches are based on preprocessing strategies that seek to extract fair training datasets by leveraging unlabeled instances  (Zhang et al.
The classic auto-encoding architecture  (Hinton & Zemel, 1993)  is enhanced with two components. One is trained to classify instances and employs the available labeled training instances. The second is a debiasing component that removes as much information as possible about the sensitive attribute, in an adversarial fashion. Additionally, our model is inductive and, as such, it can be used to classify unseen examples as well. We name our contribution FairSwiRL, which stands for Fair Semi-supervised classification with Representation Learning. Through an extensive experimental validation on synthetic and real world datasets, we show that the representations learned by FairSwiRL as the training data for different classifiers leads to reasonably accurate models while respecting the fairness constraint, even when very few labeled examples are available at training time.
Moreover, our method compares favorably to other state-of-the-art fair semi-supervised classification approaches. To the best of our knowledge, our contribution is the first one providing a comparison of preprocessing and representation learning approaches in the Semi Supervised Learning (SSL) setting under fairness constraints. The remainder of the paper is organized as follows: Sect. 2 provides a brief review of the relevant literature from semi-supervised learning and fair representation learning; Sect. 3 formalizes the problem setting; Sect. 4 presents our method FairSwiRL; Sect. 5 describes the datasets used in the experiments; Sect. 6 presents and discusses the results of the experiments; Sect. 7 concludes the paper and discusses future work.
We note here that removing the sensitive attributes s l and s u is usually insufficient to achieve statistical parity as some information about S may be present in the remaining variables X l and X u or the labels y l. Thus, FairSwiRL seeks to optimize the SAD metric by learning a debiased rep- resentation of the original data -i. e. a new representation of the data X in which all information about S has been removed. After the debiasing, any classifier trained on the latent representation will be able to achieve low SAD values without being specifically optimized for this metric. We now move onto discussing how unlabeled instances can, in principle, be useful in improving the aforementioned debiasing process. We provide a constructive example via an analysis of the toy dataset presented in Table 1.
In the given example, s represents the sensitive attribute, x 1 and x 2 are two independ- ent variables while x 3 is computed from x 1 and x 2 with the formula x 1 ⨂ x 2 , where ⨂ is an AND if s = 0 and OR otherwise: (1) P( Ŷ = 1 | S = 0) = P( Ŷ = 1 | S = 1), (2) SAD = | | | [ Ŷ | S = 0] -[ Ŷ | S = 1] | | |. It follows that there is a functional relationship between s and x 3 , which makes this latter variable a potential source of bias. The target variable y is computed as y = x 1 + x 2 + x 3 > 1 , where [⋅] is the indicator function. We assume that the data gen- eration process described above is unknown and that we are interested in learning a classifier from the data reported in Table  1. Please note that the SAD value computed on s and y is 0. 5; the dataset is, thus, unfair (as far as the statistical parity metric is concerned), but we would like to obtain a fair classifier anyway.
Firstly, if we examine the toy dataset under a semi-supervised setting without any constraint on fairness, we would likely consider the distribution of both labeled and unlabeled instances and conclude that s + x 1 + x 2 + x 3 = 2 is a good candidate as a separation hyper- plane. The predictions induced by this choice of hyperplane are reported in the column ŷSSL in Table  1. These predictions have 100% accuracy but are as unfair as the original target values y. By introducing the fairness constraint, we would like to remove information on the sensitive attribute. If we consider the labeled instances only (first four rows), we see that the sensitive attribute s, the attributes x 1 and x 3 and the target variable y are highly (actu- ally perfectly) correlated. To remove the bias introduced by these variables we might be tempted to remove s, x 1 , x 3 from the dataset and then train a classifier only on x 2 to predict y.
This classifier, however, would be no better than random guessing. If we repeat the analysis while including the unlabeled instances, we can verify that there is no correlation between s and x 1 , or even between s and x 2 , while s and x 3 do show some correlations. To debias the dataset, we can now improve on our previous attempt and remove only s and x 3. In this latter case, a classifier should then learn to ignore the x variable, as this is not a good predictor of y. It follows that the only reasonable prediction model available is ŷ = x 1 , as shown in the second-to-last column of the Table  1. This debi- ased classifier has an accuracy of 75% and a SAD value equal to 0. Hence, the accuracy is decreased with respect to the performance of ŷSSL but the SAD value is improved. We conclude that employing unlabeled instances during the learning process can dramatically improve a classifier in both accuracy and fairness terms.
Datasets We experiment on one synthetic dataset and four real-world classification datasets (see Table  2  for summary statistics). The four real-world datasets have been extensively employed in papers dealing with fair classification and fair representation learning  (Madras et al. , 2018; Louizos et al. , 2016). Furthermore, we designed a synthetic dataset in which the data generation process is known, providing us with a controlled experimental setup. In this dataset one has full control on the number of instances, the data generation process and the level of correlation (bias) between the sensitive attribute and the target variable.
Synthetic dataset Let X 0 , X 1 , X 2 , X 3 be four independent random variables uniformly distributed in the interval (-1, 1). We will draw samples from these random variables to model features of a syn- thetic dataset that we will be using to study, in a controlled environment, the characteristics of the competing algorithms. The sensible attribute is modeled through an additional variable S that we will sample from a Bernoulli distribution with p = 1 2. We note that S is independent of X 0 , … , X 3. However, we also define a "surrogate" sensitive attribute S � = S + X 0. S ′ is functionally related to X 0 and S and therefore a potential source of bias. This setup is similar to the motivating toy dataset introduced in Table  1. To model the target variable, we start by defining an intermediate random variable Z = X 1 + S � + N , where N is a noise term which we model using a normal distribution with parameters 0, 1 2.
The target random variable Y is then defined as Y = [Z >  [Z] ] (where [⋅] is the indicator function). Examples (x, y) in the synthetic dataset are realizations of the vector ((X 1 , X 2 , X 3 , S � , S), Y). It is worth noting that the variable X 0 is not directly observed, but is an important factor in the definition of Y. This fact, together with the noise introduced by N, makes it impossible to predict the target variable with perfect accuracy by training on finite realizations of the dataset. Also, since S ′ is correlated with S, fairness cannot be achieved only by getting rid of the sensitive variable s.
Real-world datasets In our experimental study, we will use the following real-world benchmark datasets. ADULT (also known as Census Income Dataset)  (Dua & Graff, 2017 ) is an extraction of the 1994 US Census database performed by Barry Becker. It contains 14 attributes (numerical and categorical) and 48,842 instances. The target variable indicates whether a person's annual income exceeds $50 000. The sensitive attribute is the sex. BANK  (Dua & Graff, 2017)  contains data related to a phone call marketing campaign of a Portuguese bank. It has 17 attributes (numerical and categorical) and 45 211 instances. The target variable to predict is whether the client will subscribe a term deposit. The sensitive attribute is the outcome of the previous marketing campaign. CARD  (Dua & Graff, 2017)  contains the data of credit cards clients in Taiwan.
• Q3 How does FairSwiRL compare to other methods in the fair semi-supervised learning literature? • A3 Favorably. When analyzing the classification performances under fairness constraints, it is paramount to employ a trade-off analysis as it is done e. g. , in multi-task learning. We show in Sect. 6. 3 that FairSwiRL paired with a random forest decision model is a good performer among fair semi-supervised competitors on four datasets out of five when assuming a balanced fairness/accuracy trade-off. Additionally, we employ a combined fairness+accuracy metric that weighs fairness as exponentially more important than accuracy. Here, we observe that FairSwiRL is able to outperform the other algorithms or remain competitive on most of the datasets. Finally, we test our method in two extremely challenging scenarios with very few labeled instances (Sect.
Visual inspection with t-SNE We inspect the latent representations provided by FairSwiRL through a qualitative assessment with t-SNE (Van der  Maaten & Hinton, 2008). In Figs. 3  and 4 , we report the 2D visualizations for the synthetic and the adult datasets introduced in Sects. 5. 1 and 5. 2 as learned by the t-SNE algorithm. Plots on the left are obtained from the original data; the center plots employ the same original data after removing only the sensitive attribute; the plots on the right display the latent representations learned by FairSwiRL on n l = 100 labeled instances and n u = 10000 unlabeled instances. We plot 10000 data samples from the test set. Colors in the top row are assigned according to the values of the sensitive 1 3 attribute; in the bottom row, colors are assigned according to the values of the target attribute.
We can notice that in the plot on the left (original data), the instances with different values for their sensitive attribute are well separated into two clusters. This is the expected behavior, as the sensitive attribute is present in the data and it is used by t-SNE to better separate the points. By removing the sensitive attribute (central column), the fairness situation improves: while still not ideal, points are now harder to classify according to s. However, points with the same sensitive attribute value are clustered in small groups, and a non-linear classifier would recognize these patterns quite easily. The representations learned by FairSwiRL (right column) do not show such pattern: the points with the same sensitive attribute value appear to be well-mixed and distributed randomly.
FairSwiRL in combination with different supervised classifiers In this experiment, we compare different classifiers in combination with FairSwiRL, namely: random forest (FairSwiRL +RF), k-nearest neighbors (FairSwiRL +KNN), logistic regression (FairSwiRL +LR), support vector machines (FairSwiRL +SVC) and neural network (FairSwiRL +NN). We now define the data splits and the evaluation metric we will employ in this section and in the rest of the paper. Let n l , n u , n v , and n t be the number of labeled, unlabeled, validation and test examples. We start with the following configuration: n l = 100 , n u = 10000 , n v = 100 , n t = 10000 (in case of the COMPAS dataset n l = 100 , n u = 1900 , n v = 100 , n t = 1900 ). We use the validation examples to find a good configuration of the hyperparameters and then, by using the same hyperparameters, we increase the number of labeled instances n l from 100 to 2000.
For each combination of (n l , n u , n v , n t ) we repeat the experiments ten times by sampling different datasets from the original data, and compute the average performance metrics. We stress that the number of available examples for a given experimental run is computed in absolute terms, not relative. This lets us compare the performance of the methodologies across the same number of test examples, no matter how many labeled examples are available. To measure the fairness level we employ 1-SAD (see Eq. 2) while for the predictive performance we compute the Matthews Correlation Coefficient (MCC  (Baldi et al. , 2000; Chicco & Jurman, 2020) ) defined as: MCC = TP⋅TN-FP⋅FN √ (TP+FP)(TP+FN)(TN+FP)(TN+FN) , where TP, TN, FP, FN represent the entries of the confusion matrix (True Positive, True Negative, False Positive and False Negative).
The behaviors of different combinations of FairSwiRL +classifier are similar across the datasets, here we report only the results for the ADULT dataset in Fig. 5. We can notice that the trends of the different classifiers are the same in predicting the target variable and in being fair. These results show that the latent representations induced by FairSwiRL can be used by different classifiers and, as the number of labeled examples increases, the performances on the target variable tend to increase. While the 1-SAD value (higher is better) slightly suffers from the bias introduced by the additional examples, we note that it remains very close to optimal values ( > 0. 9 ) nonetheless. This behavior is consistent with the motivating example introduced in Sect. 3. In the next section, we will compare FairSwiRL with competing approaches.
FairSwiRL +RF compared to competitors In this experiment, we test the effectiveness of FairSwiRL on different datasets and against different competitors. The experiment setting is the same as in Sect. 6. 2, but we choose only one combination FairSwiRL +RF (i. e. , the worst performing one) as our candidate combination. In addition to FairSwiRL +RF, we include the following competitors: • PD+RF-An RF model trained on a dataset processed by a Perfect Debiasing method, i. e. , the dataset is manipulated to guarantee that any information (direct or derived) on the sensitive attribute is removed. This is possible only in the case of the synthetic dataset (Sect. 5. 1) where the data generation process is known. Specifically, we remove S and substitute S ′ with X 0 ; • FESF-An implementation of Fairness-Enhanced Sampling Framework  (Zhang et al.
, 2022 ); • FairSSL-An implementation of the algorithm presented by  Chakraborty et al. (2021)  with Label Spreading  (Zhou et al. , 2003)  as the pseudo-labeling algorithm. • VFAE-An implementation of the Variational Fair Autoencoder  (Louizos et al. , 2016)  used to get the latent representation on which a random forest is then trained for the classification task, as in FairSwiRL +RF. The results are reported in the left column of Fig. 6. The plots report on the x-axis the performance metric (MCC) and on the y-axis the fairness metric (1-SAD). We vary the number of labeled examples in the dataset and run the experiments ten times for each configuration. Each point in the plot represents one experiment, shapes vary according to the algorithm used and colors vary according to the number of labeled examples in the dataset.