# Dimension Reduction Big Data Using Recognition of Data Features Based on Copula Function and Principal Component Analysis

Authors: Fazel Badakhshan Farahabadi; Kianoush Fathi Vajargah; Rahman Farnoosh
Journal: Advances in Mathematical Physics
 (2021)

---

## Abstract

Nowadays, data are generated in the world with high speed; therefore, recognizing features and dimensions reduction of data without losing useful information is of high importance. There are many ways to dimension reduction, including principal component analysis (PCA) method, which is by identifying effective dimensions in an acceptable level, reducing dimension of data. In the usual method of principal component analysis, data are usually normal, or we normalize data; then, the principal component analysis method is used. Many studies have been done on the principal component analysis method as a step of data preparation. In this paper, we propose a method that improves the principal component analysis method and makes data analysis easier and more efficient. Also, we first identify the relationships between the data by fitting the multivariate copula function to data and simulate new data using the estimated parameters; then, we reduce the dimensions of new data by principal component analysis method; the aim is to improve the performance of the principal component analysis method to find effective dimensions.


---

## Body

## 1. Introduction

In many real-world programs, reduction of high-volume data is of high importance and necessity as a prestage of data processing. For example, in data mining programs, dimensionality reduction is considered one of the most important stages to remove data redundancy, to increase precision of measurement, and to improve decision making process. Analyzing high-volume data is intrinsically difficult via high-volume computations for many learning algorithms as well as data processing. In dimensionality reduction methods, extraction of data features is highly important. A highly used method to reduce dimension reduction of data in data mining and in the data preparing phase is the principal component analysis method. The PCA method can be used if the original variables are correlated, homogeneous, if each component is guaranteed to be independent and if the dataset is normally distributed [1, 2]. The critical issues for the majority of dimensionality reduction studies are how to provide a convenient way to generate correlated multivariate random variables without imposing constrain to specific types of marginal distributions. An appropriate approach to this problem is to use Copula’s theory [3, 4]. In this paper, we first use the copula function to study the correlation and relationships between data to determine and eliminate irrelevant properties and simulate new data using the estimated parameter; then, by using the PCA method, we reduce the dimensions of data [4–6].


### 1.1. Principal Component Analysis (PCA)

Principal component analysis method has been first developed by Karl Pearson in 1901. The analysis includes analyzing special values of the covariance matrix. Analyzing principal components upon mathematics definition is an orthogonal transformation taking data to a new system of coordinates so that the largest data variance would be placed on the first coordinate axis; the second largest variance would be placed on the second coordinate axis and etc. Principal component analysis is aimed at transferring datasetX with m dimensions to data Y with l dimensions. Therefore, it is assumed that matrix X is formed of vectors X1,X2,⋯,Xn each of which placed in m column in matrix X. So, the data matrix would be in form of m×n. Principal components are just related to covariance matrix Σ (correlation matrix p) of random variables X1,X2,⋯,Xn [7].


### 1.2. Calculating Empirical Mean and Covariance Matrix and Data Normalization

To calculate covariance matrix, data have to be normalized first. To do so, the primarily vector of empirical mean would be calculated as follows:(1)Um=1n∑i=1nXm,i.Clearly, the empirical mean would be applied on matrix lines.Then, the distance matrix to mean would be obtained as follows:(2)B=X−uh,where h is a vector with size of 1×n and value equal to 1 in each of the entries.Covariance matrixΣ with m×m dimensions would be obtained as follows:
(3)∑=EB⊗B=EB⋅B∗=1nB⋅B∗,where E is arithmetic mean, ⨂ is an external coefficient, and B∗ is the matrix B conjugate transpose.ConsiderX′=X1,X2,⋯,Xn random vector and assume that this random vector has matrix covariance Σ with special values λ1≥λ2≥⋯≥λn≥0. Consider following linear compositions:
(4)Y1=l1′X=l11X1+l21X2+⋯+ln1Xn,Y2=l2′X=l12X1+l22X2+⋯+ln2Xn,⋮Yn=ln′X=l1nX1+l2nX2+⋯+lnnXn.Using relationship (4), we have
(5)varYi=li′∑li,covYi,Yk=li′∑lk,i,k=1,2,⋯,n.Its principal components areY1,Y2,⋯,Yn unrelated linear compositions; variances of which in relationship (5) would be large to the extent possible. The first principal component of a linear composition has maximum variance. Clearly, varY1=l1′Σl1 can be maximized through multiplying each l1 by a constant. That is, the first principal component of linear composition is l1′X which maximizes varY1 with consideration of l1′l1′=1. The second principal component of linear composition is l2′X which maximizes varY2 with consideration of l2′l2=1 and covl1′X,l2′X=0, continuously to the nth principal component.According to relationship (5), we have
(6)∑i=1nvarXi=σ11+σ22+⋯+σnn=λ1+λ2+⋯+λn=∑i=1nvarYi,and ratio of total variance to Kth component k=1,2,⋯,n is
(7)TotalshareofpopulationvariancerelatedtoprincipalKthcomponent=λkλ1+λ2+⋯+λn.If for largen, the highest maximum variance of total population (80 or 90%) could be attributed to the first several components; these components can be replaced by n primary variables, losing not much information [2, 8–10].


## 1.1. Principal Component Analysis (PCA)

Principal component analysis method has been first developed by Karl Pearson in 1901. The analysis includes analyzing special values of the covariance matrix. Analyzing principal components upon mathematics definition is an orthogonal transformation taking data to a new system of coordinates so that the largest data variance would be placed on the first coordinate axis; the second largest variance would be placed on the second coordinate axis and etc. Principal component analysis is aimed at transferring datasetX with m dimensions to data Y with l dimensions. Therefore, it is assumed that matrix X is formed of vectors X1,X2,⋯,Xn each of which placed in m column in matrix X. So, the data matrix would be in form of m×n. Principal components are just related to covariance matrix Σ (correlation matrix p) of random variables X1,X2,⋯,Xn [7].


## 1.2. Calculating Empirical Mean and Covariance Matrix and Data Normalization

To calculate covariance matrix, data have to be normalized first. To do so, the primarily vector of empirical mean would be calculated as follows:(1)Um=1n∑i=1nXm,i.Clearly, the empirical mean would be applied on matrix lines.Then, the distance matrix to mean would be obtained as follows:(2)B=X−uh,where h is a vector with size of 1×n and value equal to 1 in each of the entries.Covariance matrixΣ with m×m dimensions would be obtained as follows:
(3)∑=EB⊗B=EB⋅B∗=1nB⋅B∗,where E is arithmetic mean, ⨂ is an external coefficient, and B∗ is the matrix B conjugate transpose.ConsiderX′=X1,X2,⋯,Xn random vector and assume that this random vector has matrix covariance Σ with special values λ1≥λ2≥⋯≥λn≥0. Consider following linear compositions:
(4)Y1=l1′X=l11X1+l21X2+⋯+ln1Xn,Y2=l2′X=l12X1+l22X2+⋯+ln2Xn,⋮Yn=ln′X=l1nX1+l2nX2+⋯+lnnXn.Using relationship (4), we have
(5)varYi=li′∑li,covYi,Yk=li′∑lk,i,k=1,2,⋯,n.Its principal components areY1,Y2,⋯,Yn unrelated linear compositions; variances of which in relationship (5) would be large to the extent possible. The first principal component of a linear composition has maximum variance. Clearly, varY1=l1′Σl1 can be maximized through multiplying each l1 by a constant. That is, the first principal component of linear composition is l1′X which maximizes varY1 with consideration of l1′l1′=1. The second principal component of linear composition is l2′X which maximizes varY2 with consideration of l2′l2=1 and covl1′X,l2′X=0, continuously to the nth principal component.According to relationship (5), we have
(6)∑i=1nvarXi=σ11+σ22+⋯+σnn=λ1+λ2+⋯+λn=∑i=1nvarYi,and ratio of total variance to Kth component k=1,2,⋯,n is
(7)TotalshareofpopulationvariancerelatedtoprincipalKthcomponent=λkλ1+λ2+⋯+λn.If for largen, the highest maximum variance of total population (80 or 90%) could be attributed to the first several components; these components can be replaced by n primary variables, losing not much information [2, 8–10].


## 2. Copula Function

In general, the copula function is the link function of multivariate distributions and their marginal distributions. The copula function is a multivariate distribution, marginal distribution which follows uniform distribution of [0,1] interval [11–13].


### 2.1. Characteristics of Copula Function

Assume the following characteristics forC:I2⟶I:
(1)
For everyu, v∈0,1, we will have
(8)Cu,0=C0,v=0,Cu,1=u,C1,v=v(2)
For every0≤v1<v2≤1,0≤u1<u2≤1, we will have
(9)CU2,v2+CU1,v1−CU1,v2−CU2,v1≥0Such function likeC implied in the two above conditions is called the copula function [14].


### 2.2. Sklar’s Theorem

It is indicated by Sklar’s theorem that if joint distribution function likeH would be available with marginal distributions F and G, then, there would be copula function C available. That is, for every Xi, Xj∈ℝ, we have
(10)HXi,Xj=CFXi,GXj,and if F and G would be continuous, then, copula function C would be unique. Otherwise, C would be defined as unique on RangF×RangG.The most important application of the copula function is formulation of a proper method to produce distribution of random related multivariate variables and to provide a solution for the problem of density estimation transformation [15].For reversible transformation ofn continuous random variables X1,X2,⋯,Xn based on their distribution function to n independent variables with uniform distribution U1=F1X1, U2=F2X2,⋯,Un=FnXn, the probability density function X1,X2,⋯,Xn would be equal to fX1,⋯,Xn and joint probability density function U1, U2,⋯,Un would be equal to CU1,⋯,Un. Therefore, probability density function fX1,⋯,Xn can provide a nonparametric form (unknown distribution). Here, probability density function CU1,⋯,Un for U1,U2,⋯,Un would be estimated instead of X1,X2,⋯,Xn, so that problem of density estimation becomes simpler. Then, it would be simulated so that random samples X1,X2,⋯,Xn would be obtained through reverse transformation Xi=F−1Ui.According to Sklar’s theorem, one copula function withn unique dimensions C is available in 0,1n with uniform marginal distribution U1,U2,⋯,Un. That is, every function F with margins F1,F2,⋯,Fn can be written as follows:
(11)∀X1,⋯,Xn∈ℝn,FX1,⋯,Xn=CF1X1,⋯,FnXn.To evaluate a copula function selected via an estimated parameter and to avoid defining any hypothesis on distributions, empirical distribution function can be used. An empirical copula function is useful to study the dependence structure of multivariate random vectors. In general, empirical copula function is as follows:(12)Cij=1n∑k=1nIUkj≤Uij,where I∙ would be an indicator function [16].


### 2.3. Gaussian Copula Function

Difference between Gaussian copula function and normal joint distribution function is that the first one authorizes various distribution functions to be used for joint distribution [14]. However, in probability theory and statistics, normal multivariate distribution is considered the generalization of one-dimensional normal distribution [17].Gaussian copula function is defined as(13)CΦX1,⋯,ΦXn=1∑1/2exp−12Xt∑−1−IX,where ΦXi is a standard Gaussian function and Xi has standard normal distribution and Σ is a correlation matrix. As a result, CU1,⋯Un copula function would be called a Gaussian copula function.


## 2.1. Characteristics of Copula Function

Assume the following characteristics forC:I2⟶I:
(1)
For everyu, v∈0,1, we will have
(8)Cu,0=C0,v=0,Cu,1=u,C1,v=v(2)
For every0≤v1<v2≤1,0≤u1<u2≤1, we will have
(9)CU2,v2+CU1,v1−CU1,v2−CU2,v1≥0Such function likeC implied in the two above conditions is called the copula function [14].


## 2.2. Sklar’s Theorem

It is indicated by Sklar’s theorem that if joint distribution function likeH would be available with marginal distributions F and G, then, there would be copula function C available. That is, for every Xi, Xj∈ℝ, we have
(10)HXi,Xj=CFXi,GXj,and if F and G would be continuous, then, copula function C would be unique. Otherwise, C would be defined as unique on RangF×RangG.The most important application of the copula function is formulation of a proper method to produce distribution of random related multivariate variables and to provide a solution for the problem of density estimation transformation [15].For reversible transformation ofn continuous random variables X1,X2,⋯,Xn based on their distribution function to n independent variables with uniform distribution U1=F1X1, U2=F2X2,⋯,Un=FnXn, the probability density function X1,X2,⋯,Xn would be equal to fX1,⋯,Xn and joint probability density function U1, U2,⋯,Un would be equal to CU1,⋯,Un. Therefore, probability density function fX1,⋯,Xn can provide a nonparametric form (unknown distribution). Here, probability density function CU1,⋯,Un for U1,U2,⋯,Un would be estimated instead of X1,X2,⋯,Xn, so that problem of density estimation becomes simpler. Then, it would be simulated so that random samples X1,X2,⋯,Xn would be obtained through reverse transformation Xi=F−1Ui.According to Sklar’s theorem, one copula function withn unique dimensions C is available in 0,1n with uniform marginal distribution U1,U2,⋯,Un. That is, every function F with margins F1,F2,⋯,Fn can be written as follows:
(11)∀X1,⋯,Xn∈ℝn,FX1,⋯,Xn=CF1X1,⋯,FnXn.To evaluate a copula function selected via an estimated parameter and to avoid defining any hypothesis on distributions, empirical distribution function can be used. An empirical copula function is useful to study the dependence structure of multivariate random vectors. In general, empirical copula function is as follows:(12)Cij=1n∑k=1nIUkj≤Uij,where I∙ would be an indicator function [16].


## 2.3. Gaussian Copula Function

Difference between Gaussian copula function and normal joint distribution function is that the first one authorizes various distribution functions to be used for joint distribution [14]. However, in probability theory and statistics, normal multivariate distribution is considered the generalization of one-dimensional normal distribution [17].Gaussian copula function is defined as(13)CΦX1,⋯,ΦXn=1∑1/2exp−12Xt∑−1−IX,where ΦXi is a standard Gaussian function and Xi has standard normal distribution and Σ is a correlation matrix. As a result, CU1,⋯Un copula function would be called a Gaussian copula function.


## 3. Methodology

In the research, a two-stage method would be used for dimensionality reduction. That is, primarily empirical copula function and fit of Gaussian copula function to data would be used to estimate parameterp for variables X1,X2,⋯,Xn. Important advantages of using the copula function in multivariate distributions is that correlation between variables would be considered by these functions, and in fact, there would be no need for independence of variables; instead, the correlation structure between variables would be even considered by these functions [18]. For estimation purposes, generating function is available with dependence unscaled value available in it. The correlation coefficient value has to be specified. To do so, the Pearson correlation coefficient will be used and defined as follows for two Xi and Xj variables:
(14)ρ=covXi,XjσXiσXj,where σXi and σXj are standard deviations of Xi and Xj, respectively.Then, those data with lower correlation compared to others would be eliminated and using estimated function and Gaussian copula function forX1,X2,⋯,Xm, where m uniform variables U1=F1X1, U2=F2X2,⋯,Un=FmXm would be generated m≤n and placed instead of X1,X2,⋯,Xm in the principal component analysis method. After dimensionality reduction, the results would be compared through applying the method on raw data [16, 19].


## 4. Numerical Results

During past 30 years, increasing prevalence of urinary stone disease has been observed. About 80% of kidney stones are from calcium oxalate type. Here, 79 urine samples would be analyzed to see if some of physical features of urine are related to formation of calcium oxalate or not. These data include following columns (variables), which is available athttps://cran.r-project.org/web/packages/cond.Using Gaussian copula function, correlation values of variables would be obtained as follows:Considering Table1, it is observed that correlation of variable X2 is lower than other variables; so, it would be eliminated at the first stage. After estimation of parameters, new data would be generated. Figure 1 shows the copula function for main data and data generated by this method.Table 1
Estimation of parameterρ for variables of urine.
X1X2X3X4X5X6X11-0.308560.832310.572560.811650.54872X21-0.25167-0.09762-0.27985-0.12147X310.772260.810120.58452X410.455420.43444X510.58813X61X1is urine gravity, X2 is urine pH, X3 is urine osmolarity (it is corresponding to unit of solute concentration), X4 is urine conductivity (it is corresponding to concentration of charged ions in solution), X5 is urea concentration (mM/liter), and X6 is calcium concentration (mM/liter).Figure 1
Diagram of copula function for generated data based on main and reduced data.Now, data would be generated based on estimated parameters. To specify whether data are generated correctly or not, diagram QQPlot would be drawn.Correct data generation is shown by Figure2. In the second stage, after elimination of the X2 variable on data generated, principal component analysis would be done. In Figure 2, principal components for primary data and those generated by copula function are shown after reduction of the X2 variable. Figure 3 shows principal components for main data and the data generated.Figure 2
QQ Plot diagram of real and generated data for data of example 1.Figure 3
Diagram of principal components for raw and generated data through recommended method.Ratios of population variance related to principal components are provided in following table. Its screen plot is as follows.Considering Tables2 and 3 as well as Figure 4, it is observed that in dimensionality reduction method presented in the research, two first components include more than 80% of population variances and first component includes more than 70% of population.Example 1.
To recognize image resolution in a rectangular monitor, its display would be divided into different boxes and numbers of black and white dots in these boxes would be measured. Images of these characters have been made based on 20 different images, and each box from within these 20 boxes has been randomly selected. A file including 20000 unique simulators have been produced. Each stimulator has been transformed and scaled to 7 following numerical variables so that they would be placed within 0-15 range, (which is available athttps://cran.r-project.org/web/packages/mlbench/index.html).Table 2
Ratios of population variance related to principal components for main data.
PC1PC2PC3PC4PC5PC60.618173600.157014150.115672970.078798010.029128410.00121285Table 3
Ratios of population variance related to principal components for data generated through the recommended method.
PC1PC2PC3PC4PC50.734142800.078408480.075833990.068667190.04294755Figure 4
Diagram of population variance ration related to principal components for main data and generated data through recommended method.There are 2000 observations available from these variables.Using Gaussian copula function, correlation values of variables would be obtained as follows:X is the box. X1 is the horizontal location of box, X2 is the vertical location of box (y.box), X3is width of box (width), X4 is the height of box (height), X5 is the total numbers of dots in the box (onpix), X6 is the mean value of x in dots of the box (x.bar), and X7 is the mean value of y in dots of box (y.bar).Considering Table4, it is observed that correlation between variables X6 and X7 is less compared to other variables. So, these two would be eliminated at first stage and then Gaussian copula function would be fitted to reduced data and new data would be generated through estimated parameter, which is shown in Figure 5.Table 4
Estimation of parameterρ for variables of resolution in a rectangular monitor.
X1X2X3X4X5X6X7X110,79600,87880,74390,7282-0,02630,0296X210,70440,82030,61480,0784-0,0754X310,70890,81560,06480.0119X410.01190,0618-0,0190X510,1196-0,0278X61-0,4227X71Figure 5
Diagram of copula function for main and reduced data.Now, data would be generated. QQPlot would be as follows.Now, principal component analysis would be done on generated data. Diagrams of principal components are as follows (Figure6).Figure 6
QQ Plot diagram of real and generated data for data of example 2.Screen plot of population variance ratio related to principal components for both methods are as follows.According to Tables5 and 6 as well as Figure 7, it is observed that ratio of population variance for the first two components in the recommended method includes almost 85% of population and the first component includes almost 80% of population, whereas, for main data, ratio of population variance for the three first components includes almost 85% of population.Table 5
Ratio of population variance related to principal components for main data.
PC1PC2PC3PC4PC5PC60.551232700.200184870.090081260.071694680.050749730.02071375Table 6
Ratio of population variance related to principal components for data reduced through recommended method.
PC1PC2PC3PC4PC50.790285550.054709650.053636930.051548680.0498199Figure 7
Diagrams of population variance ratios related to principal components for main data and recommended method.


## 5. Conclusion

Considering the two aforementioned examples, it has been observed that data generated according to the estimated parameters of the Gaussian copula distribution are consistent with the original data (see Figures2 and 8) by using the recommended method in the research and copula function to recognize dependencies and structural dependence between variables in addition to elimination of redundant data will increase efficiency of principal component analysis method as well as speed of obtaining analysis results (see Figures 4 and 7, Tables 2, 3, 5, and 6). Considering the point that nowadays data are generated with high-speed, appropriate, and efficient methods for dimensionality reduction without losing information are of high importance and necessity, and recommended method in the research is a useful one to do so. The recommended method in the research can be also used for other dimensionality reduction techniques so that data would be prepared for more analysis, for example in data mining.Figure 8
QQPlot diagram of real and generated data.



---

*Source: 9967368-2021-07-12.xml*