# Correlation Measures in Metagenomic Data: The Blessing of Dimensionality
## 2 Materials and Methods
### Realistic Synthetic Data Generation for Sparsity Biases Characterization on Correlation Measurement  
To generate realistic artificial data with specified characteristics such as dimensionality (\(D\)), correlation structure (\(R\)), and sparsity (\(\phi\)), we extensively used the \"Normal to Anything\" (NorTA) paradigm. This framework is capable of producing an arbitrary multivariate distribution that conforms to a pre-established correlation structure \(R\), drawing upon the principles of the copula functions theory [30]. Essentially, the NorTA method allows for the transformation of normally distributed data into any desired distribution while preserving the original correlation structure. The core principle of the NorTA approach involves two main steps: Firstly, generating a multivariate normal dataset with the desired correlation structure, and secondly, transforming this dataset to have the targeted distribution while maintaining the predetermined correlations. The transformation is mathematically represented as follows:  
\[U_{Gen}=\ F^{-1}(CDF(U))\]  
In this equation, U represents the multivariate normal data, CDF is the cumulative distribution function of the normal distribution, \(F^{-1}\) is the inverse CDF (quantile function) of the target distribution, and \(U_{Gen}\) is the transformed data with the desired distribution and correlation structure.  
We have already defined key parameters of the generated dataset; indeed, the dimensionality (D) and the correlation structure (R) are trivially integrated within the NorTA framework. However, the delineation of dataset sparsity (\(\phi\)) is a less obvious aspect, and it is determined by the selection of the marginal distribution \(\rho\). To introduce sparsity, we had to appeal to the zero-inflated or the hurdle versions of conventional distributions. These modified distributions include an additional parameter, commonly denoted as \(\phi\), whichregulates the proportion of zero-valued data. Thus, the level of sparsity within the final dataset \(\phi\) depends on the \(\phi_{i}\) parameters designated for each marginal distribution.  
Finally, we performed L1 and CLR transformations on the tuned dataset \(U_{Gen}\), yielding \(U_{L1}\) and \(U_{CLR}\), respectively, each with their corresponding correlation matrices \(R_{L1}\) and \(R_{CLR}\). The central goal of our model is to assess how these transformations impact the correlation matrices in comparison to the original matrix \(R\), not to respect the empirical matrix from \(U_{Gen}\). Specifically, we aimed also to evaluate the CLR transformation's efficacy in addressing the skewness and normalizing data with heavy-tailed distributions through logarithmic scaling.  
Since the CLR transformation is not defined for zero values, we replaced them with a value corresponding to the 65% of the sample detection limit, in order to minimize the distortion in the covariance structure [31].  
Zero counts in NGS data do not necessarily indicate true absence but may result from limited sequencing depth or detection sensitivity. They are better interpreted as a combination of biological and technical zeros, rather than classical missing data. To evaluate the impact of zero replacement, we used several zero-replacement methods from the zCompositions [32] package and compared the resulting correlations after CLR transformation. The results are shown in Supplementary Section S3 and indicate that the differences introduced by the replacement strategy were generally limited.