# A Model-Agnostic Algorithm for Bayes Error Determination in Binary Classification
## Appendix A  
It is very useful to give a measure of how perfect a dataset is with a single number. To achieve this, a perfection index (PI) \(I_{P}\) can be defined. The Perfection Index \(I_{P}\) is defined as  
\[I_{P}\equiv\frac{1}{M}\sum_{i=1}^{N_{B}}|m_{1}^{[i]}-m_{0}^{[i]}|. \tag{1}\]  
Note that if a bucket \(i\) is perfect then either \(m_{1}^{[i]}\) or \(m_{0}^{[i]}\) are zero. In a perfect dataset \(B/P=\varnothing\). In this case, from Equation (1) it is easy to see that \(I_{P}=1\) as  
\[I_{P}=\frac{\sum_{i\in B_{1}}m_{1}^{[i]}+\sum_{i\in B_{0}}m_{0}^{[i]}}{M}=1 \tag{2}\]  
where  
\[B_{1}\equiv\{j\in[1,N_{B}]|m_{0}^{[j]}=0\} \tag{3}\]  
and  
\[B_{0}\equiv\{j\in[1,N_{B}]|m_{1}^{[j]}=0\} \tag{4}\]  
For an imperfect dataset, it is to see that \(I_{P}\) will be less than 1. In fact,  
\[I_{P}=\frac{\sum_{i\in B_{1}}m_{1}^{[i]}+\sum_{i\in B_{0}}m_{0}^{[i]}+\sum_{i \in B_{01}}|m_{1}^{[i]}-m_{0}^{[i]}|}{M} \tag{5}\]  
where  
\[B_{01}\equiv\{j\in[1,N_{B}]|m_{1}^{[i]}>0\text{ and }m_{0}^{[i]}>0\} \tag{6}\]  
that cannot be one as long as \(B_{01}\) is not empty. There is a special interesting case when the dataset \(B\) is completely imperfect, meaning \(B_{0}=B_{1}=\varnothing\), and for every bucket \(i\) with \(i=1,\ldots,N_{B}\) is true that \(m_{1}^{[i]}=m_{0}^{[i]}\). In this case, regardless of the predictions a model may make, the accuracy will always be 0.5. In this case, \(I_{P}=0\). In facts, one can see that in this case  
\[a=\frac{1}{M}\sum_{i\in\bar{B}}[p^{[i]}(m_{1}^{[i]}-m_{0}^{[i]})+m_{0}^{[i]}]= \frac{1}{2} \tag{11}\]  
since \(m_{1}^{[i]}=m_{0}^{[i]}\) and that  
\[\sum_{i\in\bar{B}}m_{0}^{[i]}=\sum_{i\in\bar{B}}m_{1}^{[i]}=\frac{M}{2} \tag{12}\]  
This index is particularly useful since the following theorem can be proved. Theorem A1: _The perfection index satisfy the relationship_  
\[I_{P}=\max_{S_{P}}a-\min_{S_{P}}a \tag{13}\]  
_where \(S_{P}\) is the set of all possible prediction vectors for the bucket set \(B\). The perfection index measures that the ranges of possible values that the accuracy (a) can have._  
Proof: To start the proof the formula for the maximum (\(\max_{S_{P}}a\)) and minimum (\(\min_{S_{P}}a\)) possible accuracy must be derived. Starting from Equation (12) and choosing (to get the maximum accuracy) \(p^{[i]}=1\) for \(m_{1}^{[i]}\geq m_{0}^{[i]}\) and \(p^{[i]}=0\) for \(m_{1}^{[i]}\leq m_{0}^{[i]}\) the result is  
\[\max_{S_{P}}a=\frac{1}{M}\bigg{[}\sum_{m_{1}\geq m_{0}}m_{1}^{[i]}+\sum_{m_{1 }\leq m_{0}}m_{0}^{[i]}\bigg{]} \tag{14}\]  
at the same time, by choosing \(p^{[i]}=0\) for \(m_{1}^{[i]}\geq m_{0}^{[i]}\) and \(p^{[i]}=1\) for \(m_{1}^{[i]}\leq m_{0}^{[i]}\min_{S_{P}}a\) can be written as  
\[I_{P}=\frac{\sum_{i\in B_{1}}m_{1}^{[i]}+\sum_{i\in B_{0}}m_{0}^{[i]}}{M}=1 \tag{2}\]0  
To prove the theorem, let us rewrite the maximum accuracy from Equation (14) as  
\[I_{P}=\frac{\sum_{i\in B_{1}}m_{1}^{[i]}+\sum_{i\in B_{0}}m_{0}^{[i]}}{M}=1 \tag{2}\]1  
and using Equations (16) and (15) previous equation can be re-written as  
\[I_{P}=\frac{\sum_{i\in B_{1}}m_{1}^{[i]}+\sum_{i\in B_{0}}m_{0}^{[i]}}{M}=1 \tag{2}\]2  
This concludes the proof. Interpretation of the Perfection Index \(I_{p}\)In the two extreme cases, if \(I_{P}\) is small then the ILD algorithm will give very useful information. The dataset is \"imperfect\" enough that an analysis as described is very useful. The more the value of \(I_{P}\) gets close to one, the more the analysis, as it is formulated here, is less helpful. This is because, in the case of a perfect dataset, a perfect prediction model can be easily built, and therefore the question of what is the best possible model loses significance. Note that there is a big assumption made, namely, that a feature bucket with a few observations contains the same information as one with one thousand observations in it. In real life, one feature bucket with just one (or few) observation will probably be due to the lack of observations collected with the given set of features and therefore should be doubted in its importance.