# Computational Test for Conditional Independence
## 2 Computational Testing with Categorical Variables
### Calculating \(p\)-Values  
Once the null distribution is established, we generate a test statistic from a single estimation of the classification model for \(Y\) using the original, unpermuted \(X\). Again, we split the data into a random training and testing set, estimate the classification model using the training data, and calculate the performance metric using the testing data. Under the null hypothesis, the metric score calculated using the unpermuted \(X\) should be a random draw from the empirical null distribution.  
Conversely, if the alternative hypothesis is true, including the unpermuted \(X\) should enhance classification performance, thereby facilitating the \(p\)-value calculation based on the test statistic's relative position within the null distribution. The (one-sided test) \(p\)-value (e.g., using log loss as the performance metric) is given by  
\[p=\frac{\sum(\text{Null}\leq\text{test statistic})+1}{n+1}, \tag{4}\]  
where \(\sum(\text{Null}\leq\text{test statistic})\) represents the count of instances in the null distribution that are less than or equal to the test statistic, and \(n\) is the total number of observations in the null distribution. Note that the direction of \(\leq\) in Equation (4) depends on which metric score is used. If, for instance, we use Kappa scores, where a higher value means a better-performing model, we would have to change \(\leq\) to \(\geq\). The \"+1\" in the numerator and the denominator is a correction factor that prevents a \(p\)-value of 0 and makes the \(p\)-value somewhat more conservative. A \(p\)-value of 0 is good to avoid since it implies absolute certainty.  
Testing CI by classification modeling for an accurate \(p\)-value calculation demands a large sample. When the sample size is small, the statistical power is often low. With an insufficient sample size, minor yet meaningful improvements in performance by including the original \(X\) rather than the permuted \(X^{*}\) may not be detected from a single train-test split of the unpermuted data. Therefore, when testing CI with a relatively small sample, one should not rely on one \(p\)-value but instead perform multiple train-test splits and examine the resulting distribution of \(p\)-values, providing additional robustness to the analysis. According to the probability integral transform theorem, if a random variable \(T\) is drawn from a known cdf \(F_{T}(t)\), as with a test statistic under the null hypothesis, the transformed variable \(P=F_{T}(T)\) follows a uniform(0,1) distribution [17] (pp. 54-55). This means that CCI test \(p\)-values should be approximately uniformly distributed under the null hypothesis.  
By calculating and plotting several \(p\)-values in Quantile-Quantile (QQ) plots, one can visually assess if the empirical \(p\)-values approximately follow the distribution against their theoretical counterparts. It is important to emphasize that evaluating the distribution of \(p\)-values is not a test per se, and one cannot and should not test statistically if the empirical distribution of \(p\)-values follows a theoretical uniform distribution, for instance, by a Kolmogorov-Smirnov test [18]. Firstly, such tests assume an underlying continuous distribution, which is not the case. Secondly, these tests are susceptible to sample size, where a large sample size almost guarantees a low \(p\)-value, and the \"sample size\" can be manipulated by the number of MCCV iterations. Thirdly, the Kolmogorov-Smirnov and other similar tests test against uniformity, which is, in this case, incorrect.