# Correlation Measures in Metagenomic Data: The Blessing of Dimensionality
## 3 Results
### Data Sparsity Remains a Limitation  
In this section, we focus on the error on estimating correlation as a function of the ratio of zero values in the samples, similar to real-world scenarios. To achieve this, we have implemented zero-inflated negative binomial distribution as the target distribution within our modeling framework based on the NorTA approach (see Section 2). This distribution was selected to accurately capture the frequent occurrence of zero counts and the asymmetrical distributions seen in real data. In the preceding section, we discussed measures taken to minimize the impact of spurious correlations introduced by the CLR transformation. To achieve this, we standardized the dimensionality (D) of all generated datasets to 200, a choice informed by its effectiveness in ensuring that correlation errors remain consistently below the threshold of 0.01. In this analysis as well, we fixed the number of observations N to \(10^{4}\) to reduce errors within the estimated correlation matrix. Furthermore, we only took the CLR into consideration for the analysis given that L1 in real situations, with more heterogeneously distributed data, is impractical, as seen in the previous section. We generate data that closely resemble real-world observations deriving the parameters munb, size, and \(\phi\) of the zero-inflated negative binomial distribution from the actual distributions of the OTUs of subject 69-001 in the HMP2 dataset, using the fitdist function from the R package SpiecEasi [36]. Each taxon was then generated using random parameters falling within the range of the first and ninth deciles of the previously fitted ZINB parameters, distributed according to their empirical distribution using the quantile function of base R.  
To quantify the error, we consider the absolute difference between the initial data correlation matrix R and the correlation on the same data transformed through the NorTA approach and CLR, \(R_{CLR}\) with non-zero correlation only being between two taxa labelled I and J. We build the correlation matrix specifically by varying only the value between I and J, labelled as \(r\), from \(-0.9\) to \(0.9\) in steps of \(0.05\), leaving all the other 198 taxa uncorrelated. In practice, all the other taxa other than I and J only contribute to reducing the biases introduced by the CLR transformation. Moreover, we varied the ratio of zero counts (\(\phi_{I}\) and \(\phi_{J}\)) of their respective marginal distributions from 0 to 0.95 in increments of 0.025. This process enables us to track the correlation error between taxa I and J across different levels of sparsity and correlation. This process was repeated 100 times for every combination of \(\phi\) and \(r\), and MAE was calculated as follows (see Figure 4A):  
\[MAE_{r,\phi}=\frac{\sum_{i=1}^{100}\lvert R^{i}(r,\phi)\ -\ R_{CLR}^{i}(r,\phi)\rvert}{100}\]  
An important aspect to emphasize in our methodology is the deliberate decision to randomly generate parameters for each ZINB distribution. This approach was intended to observe the correlation phenomenon in a manner that is as independent as possible from any specific data distribution, ensuring that our findings are not biased by particular distributional characteristics of the data. The pseudo-code below summarizes our methodology:  
This approach was intended to observe the correlation phenomenon in a manner that is as independent as possible from any specific data distribution, ensuring that our findingsare not biased by particular distributional characteristics of the data. The pseudo-code that summarizes our methodology can be found in Supplementary Section S2. MAE significantly differs between positive and negative correlations, as clearly illustrated in Figure 4A. While the error generally grows with an increasing number of zeros, this effect is particularly marked for taxa with negative correlations, as observed in the upper left section of the figure.