dataesr/scientific-paragraphs-categorization huggingface dataset:
All these PCA definitions can lead to useful generalizations  [25]. Generalization of the third above-mentioned definition by introducing weights for each pair of projections was explored in [21,  [26] [27] [28]. Below, we provide a short description of this method adapted to the purpose of this study. Let us consider the standard PCA problem. Let a set of data vectors x i ∈ R d (i = 1,. , N) be given, and let P be an orthogonal projector of R d on a q-dimensional plane. We search such a q-dimensional plane that maximizes the scattering of the data projections: H = 1 2 n ∑ i,j=1 Px i -Px j 2 = 1 2 n ∑ i,j=1 P(x i -x j ) 2 → max. (1) For q = 1, the scattering of projections (1) on a straight line with the normalised basis vector e is H = 1 2 N ∑ i,j=1 (x i -x j , e) 2 = N N ∑ i=1 (x i , e) 2 -(µ, e) 2 = N(e, Qe) (2) where µ is mean vector of the dataset X, the coefficients of the quadratic form (e, Qe) are the elements of the sample covariance matrix.
For an orthonormal basis {e 1 ,. , e q } of the q-dimensional plane in data space, the maximum scattering of data projections (1) is achieved when e 1 ,. , e q are the eigenvectors of Q corresponding to the q largest eigenvalues of Q (with taking into account possible multiplicity) λ 1 ≥ λ 2 ≥. ≥ λ q. This is precisely the standard PCA. In practice, users are usually interested in solving an applied problem, such as classification or regression, rather than dimension reduction, which usually plays an auxiliary role. The first principal components might not align with the most informative from the classification point of view features. Therefore, ignoring a certain number of the first principal components has become a common practice in many applications. For example, the first principal components are frequently associated with technical artifacts in the analysis of omics datasets in bioinformatics, and removing them might improve the downstream analysis  [29, 30].
By contrast, negative weights try to minimize the distance between the corresponding pairs of data point projections, which can be viewed as an effective attraction of projections (see Figure  2 ). DAPCA considers both labeled and unlabeled datasets and computes such projection that the projection distributions would be as similar as possible. (B) Minimizing the quadratic functional for finding each linear projection can be interpreted as introducing repulsive and attractive forces between data point projections. Of course, data points (shown as 3D spheres) do not repulse or attract, remaining fixed; therefore, the terms 'repulsion' or 'attraction' are quoted in this Figure's text. PCA can be interpreted as a result of effective repulsion between all data point projection pairs. In projection onto the Supervised PCA plane, the scattering within a data point class is minimized while the scattering between the classes is maximized.
This can be interpreted as the effective attraction of data point projections for the data points of the same class. In DAPCA, four types of effective "forces" exist between data point projections: repulsive in source and target datasets, attractive between data points of the same class in the source dataset, attractive between the data points in the target and the closest data points in the source dataset. Following the same logic as for (1), we consider the projection of (3) on a 1D subspace with the normalized basis vector e and define a new quadratic form with coefficients q W lm : H W = ∑ lm ∑ i ∑ r W ir x il x im -∑ ij W ij x il x jm e l e m = ∑ lm q W lm e l e m. ( 4 ) For the q-dimensional planes the maximum of H W (4) is achieved when this plane is spanned by q eigenvectors of the matrix Q W = (q W lm ) (4) that correspond to q largest eigenvalues of Q W (taking into account possible multiplicity) λ 1 ≥ λ 2 ≥.
Domain Adaptation (DA) Problem for Classification Task We consider that we have two datasets X and Y, characterized by the same set of m features χ 1 ,. , χ m. X represents a sample from a multivariate distribution S that we will call "source domain", and Y is a sample from T which we will call "target domain". In further, we will also call X a source dataset and Y a target dataset. The dataset X is equipped with categorical or ordinal labels attributed to some data points (not necessarily to all of them) from a discrete and not very large set of unique labels. Essentially, S and T are characterized by different distributions: T is considered to be a transformed or distorted version of S. The set of such transformations does not have to be fixed, but it is reasonable to assume some characteristic ones such as (a) Different from S number of samples of each class (different class balance); (b) Arbitrary shifts and rotations, and changes of scale, representing systemic measurement bias in target domain T compared to S; (c) Adding random (in some reasonable definition) noise to some parts of S, leading to the displacement of these parts towards the center of the data point cloud, with various degrees of such displacement for different data points; (d) Different data matrix sparsity patterns between S and T.
In this study, we are interested in finding a set of optimal for the domain adaptation task linear features { f k }. At the same time, we do not assume that the family of classifiers should be restricted to linear ones. Indeed, one of the most important applications of linear domain adaptation is to define a restricted set of features { f k , k = 1,. , q} which can be used for training a non-linear classifier, obtained as a weighted sum of the initial data variables χ 1 ,. , χ m (see examples below). As usual, from the general considerations, we expect that a set of optimal, with respect to the domain adaptation problem, linear functions f k should sufficiently well approximate the initial dataset X. This means that a reasonable approach to finding the optimal functions f k should be based on some kind of adaptation of the PCA problem.
Validating Domain Adaptation Algorithms According to  [1]  there are two main ways to validate the results of the application of a domain adaptation algorithm. We will call direct validation the way which assumes partial knowledge of labels L Y for the "unlabeled" dataset Y. In this case, the domain adaptation method is applied to the source set X with labels L and labels LY are predicted for the source dataset Y. Afterward, we can use any classification quality measure to evaluate the quality of domain adaptation. Since we can have strongly unbalanced class sizes, we can use balanced accuracy for this purpose: BA(L Y , LY ) = 1 n n ∑ r=1 ∑ l(y i )= l(y i )=L r 1 ∑ l(y i )=L r 1. (9) The reverse validation idea is different. It does not require any knowledge of "true" labels in Y but assumes self-consistency of the solution in the following sense.
Firstly we split the source dataset into two parts: the training part X L with labels L L and the test part X T with labels L T. then we solve the domain adaptation problem using the dataset X L with labels L L as a source dataset and Y as a target dataset. Set of labels LY is predicted for all data points in Y. After this step, a reverse problem of domain adaptation is solved, using Y as a new source dataset with predicted labels LY and X T as a new target dataset. For X T , the domain adaptation leads to predicting new labels LT. Afterward, the final step is the calculation of the balanced accuracy BA(L T , LT ) as in the first case. The obtained accuracy value can also be called "self-consistency" of domain adaptation. One can expect that the value of domain adaptation accuracy as a function of the algorithm hyperparameters depends monotonically on self-consistency.
Semi-Supervised PCA for a Joint Data Set The main result of this study is introducing a novel linear algorithm of domain adaptation, representing a generalization of Supervised PCA to the case when one has a labeled source dataset X and an unlabeled target dataset Y. As described above, we look for a common linear representation of X and Y, in which their multivariate distributions would be as similar as possible, while the accuracy of the classification task (using an appropriate-and not necessarily linear-classifier) for X in this representation remains acceptable. We have a labeled source dataset X = {x i } with N X data points, set of labels l i for each point in X, and a unlabeled target dataset Y = {y i } with N Y points. Let n be the total number of unique labels L 1 ,. , L n. For uniformity, we define variable z ∈ X ∪ Y.
Let us define such weight matrix W in (3) that it would lead to achieving the domain adaptation. For this purpose, we would like to introduce effective attraction between projections of X and Y onto the computed components. One of the ways to do it is to use the formula for semi-supervised (4) for z with the following weight matrix: W ij =                δ pr 2N p N r if z i ∈ X, z j ∈ X, L p = l i = l j = L r δ rr N r (N r -1) if z i ∈ X, z j ∈ X, l i = l j = L r , 0 if z i ∈ X, z j ∈ YOR z i ∈ Y, z j ∈ X, β N Y (NY-1) if z i ∈ Y, z j ∈ Y. (10) We can represent this matrix as W =   W XX W XY W YX W YY   , ( 11 ) where W XX is the matrix of SPCA (  8 ), W XY = (W YX ) are zero matrices (W XY ij = 0, for all i, j), and W YY = βJ N Y N Y , where β > 0 is the coefficient of repulsion for the target dataset Y.
Supervised Transfer Component Analysis One of the simplest existing methods of linear domain adaptation is Transfer Component Analysis (TCA)  [3] , which deals specifically with translation (shifts) of the target domain T with respect to the source domain S, in a space of features (that can be arbitrary functions of the initial variables). We can generalize the TCA approach to the case when there exist class labels in the source distribution, similar to the principle of Supervised PCA. Let us consider augmented datasets X and Ỹ containing the same set of objects as X, Y but characterized by a set of some features produced from the initial datasets X, Y. Let us denote the means of these datasets as μX = 1 N X ∑ x∈ X x; μY = 1 N Y ∑ ỹ∈ Ỹ ỹ. Now we can write matrix Q W as: q W lm = ∑ i ∑ r W ir xil xim -∑ ij W ij xil xjm -φ( μXl -μYl )( μXm -μYm ), (13) where weights W ir are assigned following the same rules as in semi-supervised PCA  (10) , and φ > 0 is the attraction coefficient between the mean points of the data samples in X and Y.
Domain Adaptation Principal Component Analysis Algorithm In applying semi-supervised PCA to the joint dataset X ∪ Y, there are no effective interactions (neither repulsion nor attraction) between the projections of data points from different domains. In order to reinforce the domain adaptation, we need to make one step further and make the projections of the data points from the target domain to be effectively attracted to similar data points from the source domain. The most non-trivial task here is defining which points from the source domain are similar to a data point from the target domain. In the simplest scenario, we will define such matching through k-Nearest Neighbors (kNN) approach. Therefore, we will introduce a term describing the effective attraction of a projection of a data point from Y to the k closest points from X (see Figure  2 ): W ij =                      δ pr 2N p N r if z i ∈ X, z j ∈ X, L p = l i = l j = L r δ rr N r (N r -1) if z i ∈ X, z j ∈ X, l i = l j = L r , β N Y (NY-1) if z i ∈ Y, z j ∈ Y, 0 if z i ∈ Y, z j / ∈ kNN(z i )AND z j ∈ Y, z i / ∈ kNN(z j ), γ kN Y if z i ∈ Y, z j ∈ kNN(z i )OR z j ∈ Y, z i ∈ kNN(z j ), ( 15 ) where k is the number of the nearest neighbours, kNN(y) is set of k labeled nearest neighbours of a data point y ∈ Y, and γ is the effective attraction coefficient between the projection of y ∈ Y and the projection of each data point x ∈ kNN(y).
However, the matching between the data points in two domains using the kNN approach can be strongly affected by the differences between X and Y, including the simplest translations. Here we deal with a sort of "chicken or egg" problem. To define the neighbors between a data point in Y and the data points in X, one has to know the best representation of both datasets, so they would be as similar as possible. On the other hand, to find this representation using DAPCA, we need to know the "true" data point neighbors. As usual, this problem can be approached by iterations. We will use the definition of the nearest neighbors in the initial data variables as the first iteration (alternatively, one can use any other suitable metrics, such as reduced PCA-based representation). It gives us the q-dimensional plane of principal components (the eigenvectors of Q W ) with the orthogonal projector on it P 1.
• If the pair 'TCA', γ is specified then Supervised TCA with attraction coefficient γ is calculated using the explicitly provided feature space (see Section 3. 2). For DAPCA and Supervised PCA models, the repulsion between different classes δ can be specified as: • a scalar to have the same repulsion for all classes; • a vector R with the number of elements corresponding to the number of classes to define the repulsion force between classes i and j as δ ij = |R i -R j |; • a square matrix with the number of rows and columns corresponding to the number of classes to specify a distinct repulsion force for each pair of classes. The earlier version of Supervised PCA is freely available from https://github. com/ Mirkes/SupervisedPCA, accessed on 20 December 2022. There exists only the Matlab implementation of this function.
• Standard PCA. This option is not recommended because of the large computation time; • Normalized PCA accordingly to paper  [26] ; • Supervised PCA accordingly to paper  [26]  (with α = 0); • Supervised PCA described in Section 2. 2 but with the same repulsion strength for all pairs of classes. Supervised PCA for regression is freely available from https://github. com/Mirkes/ SupervisedPCA/tree/master/Universal%20SPCA%20from%20Shibo%20Lei, accessed on 20 December 2022.
Testing DAPCA on a Toy 3-Cluster Example We synthesized a simple 3D dataset containing two classes of data points representing well-separated unbalanced (by factor 2) clusters in the data point cloud. Each cluster represents a sample from the 3D normal distribution. The parameters of variance were chosen in such a way that both clusters were characterized by a common axis of the dominant variance. A sample from similar but distorted distribution was used as a target domain, with hidden initial class labels (Figure  3A ). The distortion included a shift along one of the coordinates such that the degree of shift was different for the two classes. The class balance in the target domain was changed by a factor of 5 such that the number of target domain data points in one class was five times smaller than in the domain.
We also found that the number of nearest neighbors in the kNN graph is not a sensitive parameter. The number of iterations of the DAPCA algorithm producing the correct alignment of the source and target datasets was approximately ten. We used the simplest support vector classifier (SVC) in order to predict labels in the target domain using known labels in the source domain. The classifier was trained using the linear features computed by DAPCA. Since in the toy example we knew the hidden labels in the target domain by design, we could estimate both accuracy and self-consistency measures of domain adaptation as described in the Methods section (Figure  3C ). The pattern of computed self-consistency in a range of parameter values α, γ was informative for anticipating the balanced accuracy of the prediction (correlation coefficient around 0.
The Python notebook with the code of this test is provided at https://github. com/mirkes/DAPCA. 3. For CORAL, projections on all three dimensions are shown together with PCA, because CORAL does not reduce the data dimensionality. Therefore, CORAL accuracy was computed in the full feature space (marked as CORAL in the table) and after reducing the dimensionality of the merged source and target datasets transformed by CORAL (marked as CORAL+PCA). The accuracy of the domain adaptations task was estimated with known ground-truth target domain labels, using the standard Support Vector Classifier implementation in sklearn, run with default parameters. The bold font indicates the maximum accuracy.
Validation Test Using Amazon Review Dataset We made a standard for domain adaptation field validation of the DAPCA method using the same Amazon review dataset as was used in  [1] , see Figure  5. The dataset repre-sents a set of items of various kinds (books, kitchen, dvd, electronics), characterized by text reviews and the annotated binary sentiment score (positive or negative). The text of the review is represented as a vector in a multi-dimensional space by using an embedding method that produces numerical features that can be ordered by their information importance. In our experiments, we took the first 1000 features from a small Amazon reviews subset obtained from https://github. com/GRAAL-Research/domain_adversarial_neural_network, accessed on 10 December 2021. We trained a simple logistic regression either on the full set of 1000 features or using the reduced dataset with PCA, SPCA or DAPCA.
Validation Using Handwritten Digit Images Data Domain Adaptation Neural Network (DANN) approach was shown to obtain impressive results for solving the domain adaptation problem for the task of digit image classification  [1]. The DANN architecture (see Figure  6A ) combines the convolutional neural network (CNN) with an adversarial classifier in order to extract such a representation of the digit images that would allow achieving good classification in the source domain (for example, in the MNIST dataset) and as close as possible multivariate distributions in the source and target domains (for example, the SVHN dataset of images taken from real street photos together with their backgrounds). We compared the performance of the DANN neural network architecture with a much lighter one, where DAPCA was used for domain adaptation of the common representation of both source and target domains but which was learned by a standard CNN part of DANN using only the source domain and its labels.
DAPCA was applied in the space of the features provided by the very last layer of the classifier which could be used to predict the image label using simple logistic regression. In our experiments, we used two tasks with two pairs of source and target domains (MNIST vs. MNIST-M and SVHN vs. MNIST). We used the components computed with DAPCA on the features learned using the source domain only and recorded from the last layer of the CNN part of the DANN architecture. For applying the logistic regression to predict the image labels, we used as many first DAPCA components as corresponded to non-negative eigenvalues of the spectral decomposition. We visualized the multidimensional data point cloud containing the points from the source and the target domains using Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP) method.
Namely, we compared three visualizations of the digit image representations: (1) one obtained by training the CNN part of the neural network and using the source domain only; (2) one obtained by applying DAPCA on top of (1), using the target domain without labels; (3) one obtained through application of the full DANN architecture, using both source domain with labels and target domain without labels. The results are shown in Figure  6C, D. We concluded that the performance of domain adaptation provided by DAPCA is relatively modest in this benchmark, especially, when compared to the full non-linear DANN architecture. The advantage of using the domain adaptation is usually measured by how much the classifier approaches the theoretically maximum accuracy A top on the target domain which is measured in benchmarks by training the classifier directly on the known labels in the target dataset.
If one indicates as A noDA the accuracy of the classifier trained on the source domain without any domain adaptation (standard CNN in this case) then the benefit of using domain adaptation can be measured as b = A DA -A noDA A top -A noDA , where A DA is the performance of the classifier with domain adaptation. The meaning of b is the fraction of the gap between the performance of a classifier without domain adaptation and the theoretical maximal performance achievable if all labels in the target domain would be known. Thus, in the task with MNIST dataset as the source and MNIST-M as the target domains, DAPCA resulted in b = 8% compared to b = 79% obtained by the DANN architecture. In another example (SVHN as the source and MNIST as the target domains), DAPCA resulted in b = 23% while DANN resulted in b = 57%.
Of note, training the DANN architecture shown in Figure  6A  is rather heavy (tens of hours on CPU), while the DAPCA-based domain adaptation shown in Figure  6B  requires much less time (few minutes on CPU). We repeated the same benchmark in the context of a small sample size by using subsampled digit image datasets (we used only 3000 images for training and testing in each domain). The qualitative conclusions remained unchanged: the simple DAPCA-based solution was less performant than the full-scale DANN architecture even if the difference between the corresponding performances was less striking.
Application of DAPCA in Single-Cell Omics Data Analysis To demonstrate that our approach is not limited to typical machine learning applications, we also applied DAPCA in the context of single-cell RNA-seq (scRNA-seq) data analysis. scRNA-seq data is obtained by measuring the abundance of messenger RNA (mRNA) transcripts expressed within the individual cells contained in a sample (e. g. , biological tissue or an in vitro cell culture). It yields for each individual cell a molecular profile represented as a long integer-valued vector, which contains for each gene the number of transcripts measured. In this framework, a sample can therefore be seen as a data point cloud of its individual cells. Single-cell data science is an active research domain  [35]  where the application of dimensionality reduction techniques is of utmost importance, as the data spaces typically possess very high dimensionality (tens of thousands of features).
In particular, PCA is widely used to reduce the initial tens of thousands of variables to only a few tens of principal components, and most of the analyses (clustering, classification, or more generally all metric-based methods) are performed within this reduced space to mitigate the limitations linked to the curse of dimensionality. One of the most important challenges in single-cell data analysis is the presence of dataset-specific biases, caused by a variety of factors: experimenter differences, variations in the genetic and environmental background when dealing with data coming from different individuals, sequencing technologies, etc. The consequence of these biases is that data point clouds associated with different datasets appear to be displaced with respect to one another, and require special alignment procedures often referred to as horizontal data integration or batch correction  [36].
In this application, we will demonstrate the capabilities of DAPCA to serve as a batch correction algorithm for scRNA-seq data. We used three annotated, public single-cell datasets of normal lung tissue  [37]. Lung tissues were obtained from patients undergoing lobectomy for local lung tumors. According to  [37] , patient 1 was a 75-year-old male diagnosed with an early stage adenosarcoma; patient 2 was a 46-year-old male diagnosed with an endobronchial carcinoid tumors; patient 3 was a 51-year-old female with mild asthma diagnosed with an endobronchial carcinoid tumors. Epithelial and immune cells were first filtered using magnetic-activated cell sorting (MACS) and sorted using a cell sorter into four categories (epithelial, immune, endothelial or stromal). Sorted cell libraries were prepared using the 10X Genomics 3' Single Cell V2 protocol, then pooled and sequenced on a Novaseq 6000 (Illumina).