# A CNN-Based Advertisement Recommendation through Real-Time User Face Recognition
## 1 Introduction  
As the number of Internet users increases, users' interactions with multimedia devices are also growing rapidly. Recently, in fast food restaurants and cafes, non-face-to-face services that take orders through a smartphone app or a kiosk are increasing rather than receiving orders through people. Moreover, because of COVID-19, non-face-to-face orders are increasing at an explosive rate. Advertisers have not missed this opportunity and want to place various advertisements while processing orders through kiosks or smart apps. In addition, online video platform providers such as YouTube and TikTok are working to increase advertising revenue by commercializing video products [1]. Particularly, video content providers have introduced a recommender system to attain more website traffic by making it easy for users to find their favorite videos. However, online advertising video or video recommendations face a unique problem. Users often become bored with advertising videos while watching them, even if they run for less than 30 s [2]. It is challenging to capture changes in user interests from profiles using users' historical data over a brief period. Recent advances in digital image processing technology have made it possible to track the facial expressions of online advertising video viewers in real-time. Some researchers have argued that facial expressions are important in predicting a user's preference [3, 4]. For example, a user frowns when watching brutal scenes. That is, users' facial expressions make it possible to predict theiremotions. Therefore, we can utilize facial expressions to advertise video recommendations to users. They allow us to discover a user's dynamic preferences while watching an advertising video. In this study, we propose an advertising video recommendation procedure based on facial expression changes to cope with users' dynamic preferences within a short period. When a user is watching an advertisement, their emotion is captured by a webcam mounted on a kiosk, monitor, or smartphone. Then, their emotions are compared with those of other users to decide whether to keep the advertising video the user is watching or quickly replace it with another advertising video. To achieve the purpose of this study, we adopted a deep learning approach. Many studies have proposed a recommendation approach named collaborative filtering (CF), a traditional technique that recommends items suitable for users based on neighbors' preferences or purchasing history [5; 6; 7]. However, using only such a CF technique creates a \"cold start\" issue, whereby the recommendations for new users suffer from unpredictability because of a lack of historical data on their past purchases. Additionally, it has a \"first start\" issue that cannot offer recommendations until a user has reflected their preferences. In addition, there are also issues regarding the scalability of the model, arising from the continued growth of users' purchasing history or preference data [8; 9; 10]. In other words, it is challenging to offer recommendations in real time because it takes a long time to calculate data owing to model scalability problems. Many studies have been conducted to supplement issues such as data sparsity and scalability [11; 12; 13]. The recent deep learning approach has shown high performance in image processing or natural language processing areas and has received much attention [14; 15]. Many studies that applied the deep learning approach have steadily been proposed in the study recommender systems [16; 17; 18]. However, recommender systems using deep learning techniques have focused on encoding various types of input data, which was not possible in traditional CF. Recommender systems using deep learning have been developed to improve the performance of recommender systems by reflecting vast amounts of internet review data or images in existing sales data or service use records.