## 4 Results
### Pilot Implementation Results of the Product Recommender System  
The goal of this study was not only to implement the recommended algorithm as mentioned in the introduction but also to evaluate and validate the system's functionality in actual stores. To accomplish this, the implemented recommendation model was integrated into Company E's order system, and a pilot operation was conducted. For the pilot operation, stores with well-performing order operations were selected to apply and assess the recommendation system in real-world scenarios. The pilot operation took place from 13 March to 9 April 2023, spanning four weeks. Each store was instructed to order three or more products from the top 10 recommended products based on the store owner's judgment. The ordered products were promptly displayed and sold. Table 5 provides an overview of the proposed recommendation system's pilot operation. Among the eight stores participating in the pilot operation, including Store A, four stores placed orders for the recommended products for a duration of 4 weeks, two stores placed orders for 3 weeks, and Store F placed orders for 2 weeks. Store K used the recommendation system only in the first week and was excluded from the evaluation. In total, 91 SKUs recommended and ordered from seven stores were used to evaluate the pilot operation results. Table 6 presents the number of ordered SKUs each week on Mondays and the number of sold SKUs for each week. Out of the total 91 newly introduced SKUs across the seven stores, 80 SKUs were sold within 1 week, while 15 SKUs had no sales during that week. Throughout the entire 4-week pilot operation period, all SKUs were sold at least once. While opinions may vary regarding the sales criterion within 1 week, considering that Company E cells approximately 500 SKUs on average per day and each store manages around 3500 SKUs, the average sales period for newly introduced products was considered as 1 week. Based on this 1-week sales period, the accuracy of the recommendation system can be calculated as follows:  
Accuracy = Number of Sold SKUs/Number of Recommended Ordered SKUs = 80 SKUs/91 SKUs = 0.88 \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{
\begin{tabular}{c} **Store** \\ **Name** \\ \end{tabular} } & \multicolumn{2}{c}{**1 Week**} & \multicolumn{2}{c}{**2 Weeks**} & \multicolumn{2}{c}{**3 Weeks**} & \multicolumn{2}{c}{**4 Weeks**} & \multicolumn{2}{c}{**1–4 Weeks**} \\ \cline{2-10}  & **Order** & **Sales** & **Order** & **Sales** & **Order** & **Sales** & **Order** & **Sales** & **Unsold/Total** \\  & SKU & SKU & SKU & SKU & SKU & SKU & SKU & SKU & SKU & SKU \\ \hline A & 4 & 4 & 4 & 4 & 4 & 3 & 4 & 2 & 0/16 \\ B & 4 & 4 & 4 & 4 & 4 & 3 & 3 & 2 & 0/15 \\ C & 4 & 4 & 4 & 4 & 4 & 3 & 4 & 4 & 0/16 \\ D & 4 & 4 & 4 & 1 & 4 & 1 & 4 & 3 & 0/16 \\ E & 0 & 0 & 4 & 4 & 2 & 2 & 2 & 2 & 0/8 \\ F & 0 & 0 & 0 & 0 & 4 & 2 & 4 & 4 & 0/12 \\ G & 0 & 0 & 0 & 0 & 4 & 4 & 4 & 4 & 0/8 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Orders and sales of SKUs. \begin{table}
\begin{tabular}{l l} \hline \hline \multicolumn{1}{c}{**Cat.**} & \multicolumn{1}{c}{**Contents**} \\ \hline Op. Period & 13 March 2023–9 April 2023 (4 weeks). \\ No. of Stores & 8 stores. \\ New Prod. Order & Order 3 or more of the recommended top 10 SKUs by each store. \\ Order Way & Order every Monday through the recommended ordering screen in the ordering system. \\ Performance Criteria & 1. Sales status of recommended/ordered items (1 week); \\ Comparison criteria & Average sales volume per store for each product. \\  & 1. Percentage of recommended introduced products that are sold (1 week); \\ Evaluation criteria & 2. Average daily sales of recommended products by store vs. overall average daily sales by all handling stores. \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Overview of the proposed recommendation system’s pilot operation. The accuracy of the recommendation system calculated in this way is 88%.