## 4 Results and Discussion
### Performance Comparison with the Dynamic Retraining Module  
In the previous section, learning and classification were conducted under the assumption that all data were collected in advance. This assumption is easily observed in most ML-based classification studies. However, this is far from reality. Therefore, we provide a module that accommodates new retention by including the results of the previous cycle \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \begin{tabular}{c} **ML** \\ **Algorithms** \\ \end{tabular} & \begin{tabular}{c} **Class** \\ **Recall** \\ \end{tabular} & \begin{tabular}{c} **Imbalanced Dataset** \\ **Precision** \\ \end{tabular} & \begin{tabular}{c} **Our Framework** \\ **Recall** \\ \end{tabular} & \
\begin{tabular}{c} **F-Measure** \\ **Precision** \\ \end{tabular} &
\begin{tabular}{c} **F-Measure** \\ **F-Measure** \\ \end{tabular} \\ \hline \multirow{3}{*}{J48} & Low & 0.992 & 0.984 & 0.988 & 0.967 & 0.995 & 0.981 \\  & Medium & 0.917 & 0.917 & 0.917 & 0.972 & 0.820 & 0.889 \\  & High & 0.915 & 0.966 & 0.940 & 0.968 & 0.910 & 0.938 \\ \hline \multirow{3}{*}{KNN} & Low & 0.999 & 0.967 & 0.983 & 0.997 & 0.989 & 0.993 \\  & Medium & 0.852 & 0.992 & 0.916 & 0.959 & 0.977 & 0.968 \\  & High & 0.863 & 0.997 & 0.925 & 0.954 & 0.997 & 0.975 \\ \hline \multirow{3}{*}{PART} & Low & 0.991 & 0.961 & 0.976 & 0.975 & 0.977 & 0.976 \\  & Medium & 0.825 & 0.945 & 0.881 & 0.889 & 0.857 & 0.873 \\  & High & 0.839 & 0.958 & 0.894 & 0.903 & 0.911 & 0.907 \\ \hline \multirow{3}{*}{SVM} & Low & 0.999 & 0.940 & 0.968 & 0.996 & 0.981 & 0.988 \\  & Medium & 0.843 & 0.988 & 0.909 & 0.898 & 0.906 & 0.902 \\ \cline{1-1}  & High & 0.664 & 0.994 & 0.796 & 0.856 & 0.946 & 0.899 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance comparison between conventional methods and our framework in an imbalanced data environment. Figure 2: Comparison of recall between different user groups: (**a**) low consumption users; (**b**) medium consumption users; (**c**) high consumption users. in the training data at every fixed cycle. In this section, we verify the performance of the proposed module. We divided the imbalanced dataset used in the previous section into 10 sub-datasets to reflect the changes in the OTT market environment. If each sub-dataset is assumed to contain data collected for a single day, the dataset is divided into 10 sub-datasets from days 1 to 10. Because the dataset is based on data collected over 10 days, this is a reasonable assumption. First, to check whether the conventional methods properly respond to the changes in the OTT market environment, we used the data collected on day 1 as the training data and those collected on days 2-10 as the test data to measure performance. To evaluate the performance, we measured after retraining using the data classification results of up to the previous day daily. Figure 3 shows the change in recall on each day for each group, and Figure 4 shows the change in F-measure for each day for each group. First, recall was compared for each group, as shown in Figure 3. In the case of our proposed method, stable recall values were maintained without significant changes, even when time elapsed. However, in the case of conventional methods, the deviation was very large between different days. Even in the low consumption group with many samples, the performance was significantly different between our proposed method and that of conventional methods, and in the case of medium and high consumption groups, the recall dropped below 50% on severe days when only the conventional methods were used. This means that more than half of the consumers belonging to those groups were not correctly classified, which may have a critically adverse effect on the reliability of the analysis results. Similar trends were observed from the comparison of the F-measure for each  
Figure 3: Change of recall on each for each user group: (**a**) low consumption users; (**b**) medium consumption users; (**c**) high consumption users.