# Overlap in Automatic Root Cause Analysis in Manufacturing: An Information Theory-Based Approach
## 5 Evaluation  
### Evaluation Setup  
To validate the performance of the proposed method, experiments using several datasets were conducted. These datasets are the same as the ones used in [32] to ease comparison. All the datasets have a structure similar to the conceptual examples given in Section 3 and 4. However, we do not have the liberty to divulge or depict the structure of the manufacturing process used in the experiments due to a non-disclosure agreement. The datasets can be divided into three experiments:  
* **Mockup** data are generated with the aim of representing simple situations where a part of the factors are overlapped and where it is possible to control if the root cause is one of the overlapping factors. A mockup dataset generator was developed, allowing the control of (i) the number of products and steps in the manufacturing process, (ii) the percentage of factors overlapping with each other, and (iii) if the root cause is an overlapping factor or not. Each dataset represents a problematic moment, and its analysis does not require problematic moment identification;
* **Stochastic Simulation** data are generated using a stochastic simulator, which aims to emulate the data from a case study in semiconductor manufacturing. The aim of using these datasets is to use data as similar to a real case as possible, but where we are still able to control the locations of root causes. Contrary to the mockup datasets, it is not possible to preemptively establish overlap as a parameter. The overlap surges spontaneously through the emulation of a manufacturing process. In these experiments, the proposed method is validated and compared to another factor-ranking algorithm, as well as a classifier. A total of three datasets were used. These simulations' datasets emulate different conditions regarding the presence of noise. Dataset 1 represents a medium level of noise, while dataset 2 has low noise. Finally, the third dataset has a high noise level. The noise is included in the dataset through the labeling process. As products are labeled as problematic or not, a certain amount of products are mislabeled (either they were problematic without passing through the root cause, or were normal despite having passed through a root cause). The higher the noise, the greater the amount of mislabeled products. Figure 3: Detail of an example of visualization of PMI overlap among different tuples in simulated data. * **Real Case-Study** data were provided by a semiconductor manufacturing company. The objective when testing these data is two-fold: (i) to illustrate that overlap is indeed a real problem that occurs when analyzing data from a manufacturing process, and (ii) to validate the proposed algorithm with real data. The actual root causes were not identified by the company used as a case study, and as such, validation of the RCA algorithms is carried out by comparing their results and verifying if there is a convergence in the proposed lists of most likely root causes. The proposed method (denominated PMI) is compared with the Co-Ocurrences (CO) algorithm (a factor ranking algorithm proposed in [32]) and a Decision Tree (DT) classifier. The CO algorithm has no parameters to be tuned, while the DT classifier has its parameters (minimum number of instances per split and a confidence factor) tuned in each dataset using grid search and 5-fold cross-validation. The idea behind the CO algorithm is to compare the proportion of products with problems that went through a certain step-machine tuple with the proportion of problematic products that did not go through that tuple. If a tuple has a greater proportion of problematic products that went through it than normal ones, it is likely a root cause. The Co-Occurrences (CO) algorithm is shown in Algorithm 2.