## 4 Implementation and Results
### Evaluation Results  
Figure 25 shows the structure of the facility log file transmission and management system for data gathering. The facility log file transmission and management system was integrated into the actual implementation environment to operate three laser processing machines (GTW4, GTW5, and GTW6). Each processor had to have a DV agent installed. The DV agents were installed in the processors. They detected changes in the log files that were targeted by each processing machine and when there was a change in a log file, that file was transmitted to the server. Because the machine's hardware requirements were low, it tried to use as few resources as possible. The server collected and saved the log files that were sent by the agent. It compared the collected files to those in the database and saved the differences. The database structure was identical to that shown in Table 2, which shows the information about the DV agents and 61 log files. Using this, it was possible to extract the processing data from the log files of 24 laser processing machines, as shown in Table 3.  
The inspection accuracy of the AI method was measured by comparing it to the accuracy of human visual inspection:  
\[Accuracy=(TP+TN)/(TP+TN+FP+FN) \tag{1}\]  
where _TP_ is the number of true positives, _FP_ is the number of false positives, _FN_ is the number of false negatives, and _TN_ is the number of true negatives. Figure 26 shows a diagram that contrasts the accuracy of human visual inspection and the accuracy of the AI results. In order to compare the results from the two methods, 1000 data points were  
Figure 24: Task assignment status and link to work order by facility. randomly selected and the accuracy of the methods was measured. The accuracy improved to 96.7% when the AI method was applied compared to 87% when human visual inspection was used. \begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{3}{c}{**DV Agent Information**} \\ \hline
**File** & **Data Type** & **Explanation** \\ \hline DEVICE\_NAME & VARCHAR(10) & Device name \\ \hline DEVICE\_TYPE & VARCHAR(10) & Device type \\ \hline DEVICE\_IP\_ARR & VARCHAR(10) & Device IP list \\ \hline DEVICE\_DT & DATETIME2 & Heartbeat date \\ \hline MOD\_DT & DATETIME2 & Update date \\ \hline REG\_DT & DATETIME2 & Registration date \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The structure of the database. Figure 26: The ROC and AUC values for the results from the two methods (AI and human visual inspection). Figure 25: The structure of the facility log file transmission and management system. Table 4 shows KPI evaluation results, in which the facility operation rate refers to the increase/decrease rate (%) of the operation time of the production equipment compared to 2021 (based on there being 24 h in a day and excluding downtime due to reduced order volumes) and the process defect rate refers to the increase/decrease in the process defect rate compared to 2021 (PPM, laser drill processing standards). For the overall achievement rate, the overall improvement rate was calculated as 26.7 \(\times\) 0.7 + 40.6 \(\times\) 0.3% = 30.87% and the overall goal achievement rate was calculated as 155.8 \(\times\) 0.7 + 122 \(\times\) 0.3% = 145.66%. In the improvement rate calculation method, the increase characteristics were calculated using (performance \(-\) present)/current \(\times\) 100 (%) and the decrease characteristics were calculated using (current \(-\) performance)/current \(\times\) 100 (%). In the goal achievement rate calculation method, the increase characteristics were calculated using (performance \(-\) current)/(target \(-\) present) \(\times\) 100 (%) and the decrease characteristics were calculated using (current \(-\) performance)/(current \(-\) goal) \(\times\) 100 (%). In the improvement/goal achievement average rate calculation method, the improvement rate was calculated using (index 1 improvement rate \(\times\) weight) + (index 2 improvement rate \(\times\) weight) + (index 3 improvement rate \(\times\) weight).