# Correlation Measures in Metagenomic Data: The Blessing of Dimensionality
## 3 Results  
### Compositional Biases Become Negligible with High Dimensionality  
To comprehend and quantify the compositional biases inherent in Pearson correlation, we conducted a comprehensive comparative analysis. We compared the known correlation structure initially provided as input to the model with the correlation structures obtained after applying L1 and CLR normalizations, while systematically varying the dimensionality \(D\) and the within-dataset diversity \(\underline{P}\) (see the Section 4). In total, we generated 1560 distinct datasets by systematically varying two parameters: the dimensionality \(D\), which ranged from 5 to 200 in steps of 5 (i.e., 40 values), and the within-dataset diversity \(\underline{P}\), which ranged from 0.025 to 0.975 in steps of 0.025 (i.e., 39 values). For each of the 1560 combinations of \(D\) and \(\underline{P}\), we created a dataset where the relative abundance vectors were sampled to match the target diversity within a tolerance of \(\pm\)0.005. To isolate the effects of the L1 and CLR transformations, we made deliberate efforts to minimize any known sources of error and chose the simplest experimental conditions to ensure the robustness of our findings. In line with these principles, we consistently conducted the analysis with an uncorrelated covariance structure, and we chose to work with normally distributed variables to avoid potential errors in the Pearson correlations that may result from non-normally distributed data. Furthermore, to minimize random correlations and sampling errors, we used a large number of samples in our simulations. The goal was not to reproduce realistic conditions but to create a controlled setting where the effect of compositionality could be isolated. Since the dimensionality in our simulations ranges from 5 to 200 variables, we chose N = 10,000 to ensure that even in the highest-dimensional case (D = 200), spurious correlations due to finite sample effects would be negligible. By simulating data with a fully sparse correlation structure (i.e., all true correlations set to zero), we could directly assess how different transformations--such as CLR or L1--introduce spurious associations purely due to the compositional constraint. Realistic scenarios are addressed in the following section using real 16S data from HMP2, which confirm the qualitative trends observed in this idealized setting. Finally, we quantified the biases by calculating the mean absolute error (MAE) on all values of the matrix obtained by subtracting L1- and CLR-normalized correlation matrices, denoted as \(R^{L1}\) and \(R^{CLR}\), to the original correlation matrix R, as follow:  
\[MAE_{D,\underline{P}}(K)\ =\ \frac{\sum_{i=1}^{D}\lvert R_{i}^{K}-\ R_{i}\rvert}{D^ {2}}\ \text{withK}=\text{L1 orCLR}\]  
MAE values theoretically range from 0 (perfect agreement with the ideal correlation) to 2 (maximum possible distortion, achieved only when \(R_{i}=\ \pm 1\) and the estimated correlation is exactly opposite). The distinct behaviors of the two normalizations are evident, as they introduce different biases on correlation (see Figure 2). Specifically, L1 correlations are primarily influenced by within-dataset diversity, with the biases becoming more pronounced as the values within a sample become more heterogeneously distributed. On the other hand, CLR data exhibit biases that are independent of dataset diversity, and these distortions diminish rapidly with increasing dimensionality. Building upon the premise of complete independence of the CLR biases on correlation from dataset diversity, we can estimate this effect by calculating an average overall diversity value \(\underline{P}\). We observe that the error decreases to less than 0.01 for dimensionality values greater than or equal to 100. Thus, we posit that in typical metagenomic scenarios, where the dimensionality often extends into the hundreds, the effects of compositionality are negligible.