[image] [image] Ressource protégée - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu

[image] [image]

* Documentation

* FR
  Français (FR)

* Anglais (EN)
  Se connecter

* Portail HAL Inria

* Recherche
  Loading...

## Recherche avancée

Information de documents

Titres

○ Titres

○ Sous-titre

○ Titre de l'ouvrage

○ Titre du volume (Série)

○ Champ de recherche par défaut (multicritères) + texte intégral des PDF

○ Résumé

○ Texte intégral indexé des documents PDF

○ Mots-clés

○ Type de document

○ Sous-type de document

○ Tous les identifiants du document

○ Identifiant HAL du dépôt

○ Langue du document (texte)

○ Pays (Texte)

○ Ville

○ À paraître (true ou false)

Ajouter
<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
## Auteur

### Auteur (multicritères)

○ Auteur (multicritères)

○ Auteur : Nom complet

○ Auteur : Nom de famille

○ Auteur : Prénom

○ Auteur : Complément de nom, deuxième prénom

○ Auteur : Organisme payeur

○ Auteur : IdHal (chaîne de caractères)

○ Auteur : Fonction

○ Auteur : personID (entier)

○ Auteur : Identifiant de l'organisme payeur

○ Auteur : Identifiant de la structure

○ Directeur de thèse

○ Éditeur

○ Éditeur scientifique

○ Éditeur de la série

## Ajouter

### Structure

### Structure (multicritères)

○ Structure (multicritères)

○ Structure : Acronyme

○ Structure : Nom

○ Structure : Code

○ Structure : Pays

○ Structure : Type

○ Structure : État dans le référentiel

○ Structure : Identifiant HAL de la structure

○ Structure/équipe de recherche : Pays

○ Structure/regroupement d'équipes : Pays

○ Structure/laboratoire : Pays

○ Structure/regroupement de laboratoires : Pays

○ Structure/institution : Pays

○ Structure/regroupement d'institutions : Pays

## Ajouter
# Examining the effective of web advertising on pupil size

Ryuta Yamashita<fnref n="1" />, Tadaaki Shimizu<fnref n="2" />, Natsuki Yoshinaka<fnref n="3" />, Rintaro Kataoka<fnref n="4" /> and Naoki Sawada<fnref n="4" />

**Abstract.** The purpose of this study is to estimate the emotions that users experience when they see advertisements displayed by websites. It is possible to provide users with the products and services they want by displaying web advertisements appropriately. This is a desirable situation for companies, users, and websites. We thought that users would have a physical reaction when they viewed a website. We especially focused on change of pupil size. In our experiment, the subjects were presented several types of web advertisements.

The types of web ads presented are: 3 types of ads display methods (banner-ad, front-ad, Click-ignore ad), 2 types of advertising images (positive and negative), for a total of 6 combinations.

We found a characteristic response of pupil size by ads types by use of golden cross and dead cross analysis. And we show possibility of estimating emotion by pupil size.

**Keywords:** Recommendation System, Pupil Size, Web Advertising, Multi-modal Information, golden-cross

## 1 Introduction

The Internet has been widely used, and 80-90% of Japanese use it on a daily basis [1]. As a result, many web sites are published on the Internet, and many of them are provided by the benefits through the web advertisement. Many Web advertisements are posted on major web sites, such as “Google” and “Yahoo! Japan”. These major websites are developing ways to present advertisements that are suitable for users [2]. One example is to build a recommender system using BIG-DATA. It uses the user’s ID information and web operation history data to present advertisements suitable for the user.

<footnote n="1">¹ Graduate School of Engineering, Tottori University, 4-101 Koyama-cho Minami, Tottori, 680-8550, Japan</footnote>

<footnote n="2">² Cross-informatics Research Center, Tottori University, 4-101 Koyama-cho Minami, Tottori, 680-8550, Japan</footnote>

<footnote n="3">³ Department of Engineering, Graduate School of Sustainability Science, Tottori University, 4-101 Koyama-cho Minami, Tottori, 680-8550, Japan</footnote>

<footnote n="4">⁴ Department of Engineering, Information and Electronics Course, Tottori University, 4-101 Koyama-cho Minami, Tottori, 680-8550, Japan</footnote>
Autres

Champ de recherche par défaut (multicritères)

○ <u>Champ de recherche par défaut (multicritères)</u>

○

○ <u>Revues (multicritères)</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Titre abrégé</u>

○ <u>Revue : Titre</u>

○ <u>Revue : Identifiant interne</u>

○ <u>Revue : Date de début de publication</u>

○ <u>Revue : ISSN électronique</u>

○ <u>Revue : ISSN</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Couleur dans SHERPA/RoMEO</u>

○ <u>Revue : État dans le référentiel</u>

○

○ <u>Colloque (multicritères)</u>

○ <u>Colloque : Titre</u>

○ <u>Colloque : Organisateur</u>

○ <u>Colloque : date de début (Année)</u>

○ <u>Colloque : date de fin (Année)</u>

○ <u>Conférence invitée (oui/non)</u>

○

○ <u>Projets ANR (multicritères)</u>

○ <u>Projet ANR : Acronyme</u>

○ <u>Projet ANR : Acronyme du programme</u>

○ <u>Projet ANR : Code décision (référence)</u>

○ <u>Projet ANR : Nom</u>

○ <u>Projet ANR : Identifiant interne</u>

○ <u>Projet ANR : État dans le référentiel</u>

○

○ <u>Projets européens (multicritères)</u>

○ <u>Projet européen : Acronyme</u>

○ <u>Projet européen : Identifiant de l'appel à projet</u>

○ <u>Projet européen : Référence</u>

○ <u>Projet européen : Nom</u>

○ <u>Projet européen : Date de fin</u>

○ <u>Projet européen : Financement</u>

○ <u>Projet européen : Date de début</u>
In this way, the environment of Web advertising is constantly evolving. However, according to the “Awareness Survey on Internet Advertising” [3], more than 80% of users responded that they “feel that advertising on the Internet is intrusive or annoying.” This result shows that the user’s impression of the web advertisement is bad, and it is not a good situation for websites, advertisers, and users. Web advertising connects users and products, and it is an important medium to promote better consumer behavior. For all three parties to get good results, the advertising system needs to serve ads more appropriately to the user.

We aim to build a recommendation system that displays advertisements that are more suitable for users. Therefore, we focused on the multimodal information of users while browsing a website. Multiple methods of communication, including visual and auditory, are referred to as multimodal information.

## 2 Related study on multimodal information

In recent years, there has been a lot of study purported at making systems understand the emotions of users. There are many studies that use multimodal information of users.

Kato's study is to develop a conversation system for users who were not experienced with ICT devices [4]. This system attempted to recognize facial expressions and estimate emotions from the subject's face image.

Furukawa's study [5] is an attempt to estimate emotions from pupillary responses and eye movements. This study is not target to Web advertising. However, the results showed that user's multimodal information tends to express the user's emotions.

Q. Zhao's study [6] purpose to estimate the user's next gaze direction and utilize it for recommendation functions. The study limited on the grid user interface of smartphones. However, this study is an example of the use of multimodal information for recommendation functions. In this study found that it is possible to estimate the user's next gaze even in the limited environment of a "grid-based UI" of a smartphone. The purpose of this study is to realize a system that can display recommendations efficiently by utilizing gaze.

The study by Simola examines the effect of advertising on reading web content [7]. Their study focused line of sight of subjects. In their study examines the effects of changing conditions, such as animating Web advertisements and adjusting display time intervals, on the line of sight. The subjects' gaze dwell time tended to be longer for animated ads than for static ads. Also, there is an effect between the appearance time of the advertisement and the movement of the gaze.

Yamanaka's study [8] proposes a convenient method of evaluating mental stress. An experiment was carried out to clarify the relation between changes in pupil diameter and autonomic nervous activity by measuring an electrocardiogram and pupil diameter. Several relationships between changes in pupil diameter and autonomic nervous activity were revealed and indicated that measurement of pupil diameter was an effective indicator of autonomic nervous activity.
○ Projet européen : État dans le référentiel

○ Projet européen : Identifiant interne

○

○ Date de production : année

○ Date de mise en ligne : année

○ Date de publication : année

○ Date d'écriture : année

○ Date de modification du dépôt : année

○ Date de dépôt : année

○ Date de publication électronique : année

○

○ Collection HAL (multicritères)

○ Collection HAL : catégorie

○ Collection HAL : Code

○ Collection HAL : Nom

○ Collection HAL : Identifiant interne

○

○ Identifiant interne du contributeur/déposant

○ Nom complet du contributeur/déposant

○

○ Domaines

○ Domaine primaire

○ Domaine racine

○ Sous-domaine niveau 1

○ Sous-domaine niveau 2

○ Sous-domaine niveau 3

○

○ Statut du document

○ Version du document

○ Type de dépôt

○ Type de document

○ ISBN

○ Numéro - référence

○ Identifiant DOI

○

○ Classification

○ Audience

○ Vulgarisation

○ Comité de lecture - texte (oui ou non)

○ Actes de colloque
## 3 Purpose

Based on related research, we hypothesized that "stimuli received from Web advertisements (hereafter referred to as "advertising stimuli") characterize users' multimodal information.

Total of six types of multimodal information were obtained in this experiment.

- Mouse clicks - Mouse cursor trajectory - Eye gaze - Pupil size - Emotion estimation from facial expression images - Posture estimation from body images.

The reason for targeting this multimodal information is that no medical knowledge or skills are required to handle them, and the data can be acquired with readily available devices.

In order to fairly describe the experimental environment, all multimodal information acquired in the experiment was enumerated, however this paper focused on "pupil size" and "mouse clicks". Other multimodal information is obtained for the co-experimenters. This multimodal information is not discussed in detail in this paper.

Therefore, the objective of this study was to "find characteristic changes in the magnitude of the tendency when users are exposed to advertising stimuli. If characteristic multimodal information can be found in the experimental results, it will be possible to estimate the subject's emotions based on these characteristics.

## 4 Experiment details and Subjects

An experiment was conducted to obtain multimodal information about the subject's browsing of a Web site. For the experiment, we created our own two-choice quiz website (hereafter referred to as the "experimental website").

Subjects were given advertising stimuli while operating the experimental website. The details of the experimental website are described in Section 5.1. The types of advertising stimuli given to the subjects are described in Section 5.2. The details of the system for acquiring multimodal information are described in Chapter 6.

20 subjects (10 males and 10 females, aged 18-24) were asked to participate in this experiment. The subjects were told in advance that they would receive additional rewards (gift certificates) depending on their quiz performance. We created an environment in which the subjects could operate the quiz site seriously.

# 5 Experimental website and Advertising stimuli

## 5.1 Overview of the experimental web system

The experimental website has the following specifications: (1) to (5). The subjects were given advertising stimuli during the actions of (1) to (2).

(1) Display the question text (see Fig. 1).

(2) Click on the answer that the subject judges to be correct.

(3) The experimental quiz system displays the correctness of the quiz (see Fig. 2).
○ Référence interne

○ Financement

○ Collaborations

Ajouter

Lancer la recherche

Recherche experte (SolR)

## Recherche experte (SolR)

Lancer la recherche
Vers la recherche avancée

• Déposer

Portail INRIA.HAL.SCIENCE
Les publications, logiciels... des scientifiques Inria

• Accueil

• Publications Inria

○ Consulter les derniers dépôts

○ Par équipe de recherche

○ Par auteur

○ Rapports de recherche et techniques

○ Livres blancs

○ Rapports d'activité des équipes

○ Statistiques

• Boite à Outils

○ X2Hal : importer par lot

○ Logiciels : créer le Codemeta.json

○ Créer mon IDHal

○ Créer mon CVHal

○ Haltools : créer ma page de publications

○ Haltools : extraire des données (réservé Inria)

○ Les bonnes pratiques

■ Comment bien s'affilier ?
(4) When the subject clicks the "Go to Next Question" button, the experimental quiz system returns to (1).

(5) After about 14 minutes, the experiment is finished.

<figure><img src="image_2.png" /><figcaption>Fig. 1 The question screen</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2 The correct answer screen</figcaption></figure>

The total number of questions in the quiz on the experimental website was 738. The questions were randomized, but the same question was not displayed during one experiment. However, when all the questions have been submitted, they will be randomized again. The experimental website was built using an AWS (Amazon Web Service) [9]. virtual server. The specifications of the virtual server used are shown in Table 1.

<table><caption>Table. 1 specifications of the virtual server.</caption><thead><tr><td>AWS Spec (t2.micro)</td><td>Detail</td></tr></thead><tbody><tr><td>OS</td><td>Amazon Linux release 2</td></tr><tr><td>Memory</td><td>1 GiB</td></tr><tr><td>Storage</td><td>8 GiB</td></tr><tr><td>Availability zone</td><td>us-east-2c</td></tr></tbody></table>

The experimental website was implemented using python (Django framework), html, CSS, and JavaScript. In addition, PostgreSQL was installed in the virtual server. In the experimental website, the question texts and correctness errors were saved in PostgreSQL.

## 5.2 Types of advertising stimuli and stimulus intensity

There are a wide variety of factors that interfere with the user's experience with ad displays. It is very difficult to investigate all factors. In this study, we investigated 1) images of advertisements and 2) methods of displaying advertisements.

### 1) Images of advertisements.

Various images are displayed in web advertisements. First, we decided to roughly investigate the relationship between the displayed images and multimodal information. For this purpose, we prepared two patterns of images that were thought to affect the user's pleasantness or unpleasantness.
■ Quelle version puis-je déposer ?

■ Choisir le bon « type de document »

■ Mettre en valeur ses publications

○ Les tutos IES Inria

■ Lier les notices dans HAL

■ Umbrhal Inria pour l'accessibilité

■ Personnaliser son CVHAL

■ Créer et Personnaliser sa collection

• Documentation

○ Aide en ligne HAL

○ Aide en ligne Haltools

○ Aide en ligne X2hal

○ Tutoriels video HAL

○ Documentation API HAL

○ Ajouter des vignettes

○ Identifiant auteur IdHAL et CV

○ Compléter son dossier ORCID avec ses dépôts HAL

○ Inria et la Science Ouverte

X

X

X

Loading...

X

Le document que vous demandez est en access restreint
jusqu'au 2025-01-01 !

Revenir a la notice

Retourner à l'accueil

Contact

•

Ressources
In this experiment, OSAIS (Open Affective Standardized Image Set) [10] was used as an image that mimics an advertising image. OASIS is an open access online stimulus set that contains 900 color images of humans, animals, etc. Every image in OASIS is assigned two values: Valence value and Arousal. In addition, OASIS assigns a positive value and a negative value to every image. First, we selected images for our experiment that met the criteria based on two values, Arousal_mean and Valence_SD. The specific criterion is that Arousal_mean > 3.66 and Valence_SD <= 1.1. We selected 30 images that met the criteria of Arousal_mean and Valence_SD in the order of positive and negative values, respectively. However, we did not use those images that were offen- sive to public order and morals even though they met the criteria. (e.g., dead bodies, nudes, wounds, blood, etc.) Examples of positive and negative images, respectively, are shown in Fig. 3 and 4.

<figure><img src="image_4.png" /><figcaption>Fig. 3 Example of Positive Images</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4 Example of Negative Images</figcaption></figure>

## 2) Methods of displaying advertisements.

In this study, the subjects were presented with three types of advertising stimuli: 2a) Banner ads, 2b) Front-page ads, and 2c) Click-ignore ads.

### 2a). Banner Ads.

Displays images like banner advertisements used in popular websites. This advertise- ment display will not interfere with the user's operation. An example is shown Fig. 5.
* Documentation

* FAQ

* API

* OAI-PMH

* AuréHAL

## Informations

* À propos

* Données personnelles

* Mentions légales

* Accessibilité

* Conformité RGAA

## Questions juridiques

* Je publie, quels sont mes droits ?

* Loi pour une République numérique

* Stratégie de non-cession des droits

* SHERPA/RoMEO

## Portails

* Portails institutionnels

* HAL

* HAL SHS

* HAL Thèses

* MédiHAL

## CCSD

* CCSD

* Episciences

* Sciencesconf
<figure><img src="image_6.png" /><figcaption>Fig. 5 Example of Banner Advertisement</figcaption></figure>

(Left: positive / Right: negative)

2b). *Front-page Ads.*

Front page advertisements are advertisements that are used on some websites. It is a display type that interferes with the user's operation of the web contents. An example of the advertisement display in this experiment is shown in fig. 6.

<figure><img src="image_7.png" /><figcaption>Fig. 6 Example of Front-page advertisement</figcaption></figure>

(Left: positive / Right: negative)

2c). *Click-ignore Ads (Click-ignore + front page advertisement).*

The third is a display pattern like the front-page advertisement, but with different click controls. This display pattern ignores the "Close" button in the Front-page advertisement with a certain probability. The specific process is as follows: First, a value between 1 and 6 is randomly determined when the Front-page advertisement is displayed. Then, ignore the click of the "Close" button for the number of times the value is determined. This means that the subject will have 1 to 6 clicks ignored. This display pattern is very disturbing to the user. This pattern will henceforth be referred to as "Click-ignore Ad".

Subjects were given a total of 7 different advertising stimuli (2 types of advertisement images * 3 display types + no advertisement stimuli) in an unbiased manner. Fig. 7 illustrates how the advertisement stimulus is given to the test subject.
<figure><img src="image_8.png" /><figcaption>Fig. 7 Time Chart of Advertising Stimulus</figcaption></figure>

The following is an explanation of Fig. 7.

(1) The first 60 seconds after the subject started the experiment was the interval without advertisement stimuli. This is the interval for the subjects to get used to the display and operation of the experimental website. After 60 seconds, the first time the question is asked, the experimental website proceeds to (2).

(2) The experimental website provides advertising stimuli to the subjects. This advertising stimulus is randomly selected from the 7 types of displays already mentioned.

(3) The subject either closes the ad or answers the displayed questions.

(4) After the operation of (3), the quiz is displayed without advertisement stimulus for 15 seconds. When the quiz is displayed for the first time after 15 seconds, the system proceeds to (5).

(5) The subject is again given the advertising stimulus. The advertisement stimulus to be given is randomly selected from the remaining display types.

(6) Repeat (3) through (5) until all 7 types of displays have been completed.

(7) Repeat (2) through (6) three times.

(8) After the completion of (7), the quiz will continue until the end of the experiment (14 minutes and 15 seconds). We set the experiment time to 14 minutes and 15 seconds so that all the ads could be displayed.

We also defined a measure of stimulus intensity for the four types of advertising displays in this experiment. The stimulus intensity was defined as the degree to which the advertisement interfered with the subject's operation. The stimulus intensity is shown in Table 2. In this paper, we also focus on the effect of stimulus intensity on multimodal information.
<table><caption>Table 2. The stimulus intensity of the advertisement</caption><thead><tr><th>The stimulus intensity</th><th>Display Type</th><th>Strength Basis</th></tr></thead><tbody><tr><td rowspan="4">Weak ↓<br>Strong</td><td>No ads displayed</td><td>Because there are no ads on the screen.</td></tr><tr><td>Banner Ad</td><td>Because the display of advertisements does not interfere with the subject's operation.</td></tr><tr><td>Front-page Ad</td><td>Because advertisements that interfere with the operation of the experimental website are displayed.</td></tr><tr><td>Click-ignore Ad</td><td>Because the process of ignoring the subject's click is performed in addition to the display of Front-ad.</td></tr></tbody></table>

In addition, we focused analysis for difference of displayed positive and negative images. We hypothesized that there may be multimodal information that is featured only in positive or negative images.

In other words, we decided to focus on "stimulus intensity" and "positive/negative images" for our analysis.

## 6 Multimodal information acquisition system

The multimodal information of the subject's 6 points is obtained using the devices shown in (a)-(e) in Fig. 8 and Fig. 9 shows the scene during the experiment.

<figure><img src="image_9.png" /><figcaption>Fig. 8 Multimodal Information Acquisition System</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 9 Scene during an experiment</figcaption></figure>

Table 3 shows the specifications of the devices used in the multimodal acquisition system. As we described in Chapter 3, the devices (c) and (e) were prepared for co-experimenters.

<table><caption>Table 3. List of devices used in the experiment</caption><thead><tr><th>Device</th><th>Details</th></tr></thead><tbody><tr><td>CPU</td><td>Intel Core i9-9900X</td></tr><tr><td>GPU</td><td>NVIDIA GeForce RTX 2060 SUPER</td></tr><tr><td>OS</td><td>Windows10 Education</td></tr><tr><td>Web Browser</td><td>Google Chrome</td></tr></tbody></table>
<table><tr><td>(a) PC monitor</td><td>iiyama ProLite XUB2792HSU</td></tr><tr><td>(b) Mouse</td><td>ELECOM M-BL27UBBK</td></tr><tr><td>(c) Azure Kinect DK</td><td>-</td></tr><tr><td>(d) Tobii Pro nano [11]</td><td>-</td></tr><tr><td>(e) Web Camera</td><td>Logicool C920n +</td></tr></table>

There were no restrictions on the wearing of glasses in the above devices. Therefore, the wearing of the glasses was left up to the subject. On the other hand, wearing a thick jacket or a mask affects the data to be acquired. Therefore, we asked the subjects participating in the experiment not to wear them.

# 7 Pupil size analysis method

## 7.1 Pretreatment

We performed preprocessing on the acquired pupil data. The purpose of the preprocessing is as follows:

- To standardize the scale between subjects

- To express a rough change rather than a detailed change

The detail of the preprocessing is as follows:

Original data (left and right pupil size)

→ Interpolate missing data with data from the previous time

→ Resampling process (60hz to 20Hz)

→ Average left and right eye data

→ Normalize by 0~1

There were 20 subjects, but there was one subject for whom eye data could not be obtained due to the forelock. Therefore, the number of subjects for analysis was 19.

## 7.2 Analysis by moving average

A moving average of 10 seconds (5 seconds before and after) was calculated for the data that had undergone "7.1 Preprocessing". Fig. 10 shows an example of the calculated moving average data and time trends. The areas with colored backgrounds in the figure are the areas where advertising stimuli are provided.

<figure><img src="image_11.png" /><figcaption>Fig.10 An example of moving average data and advertising stimuli</figcaption></figure>

The following trends were common among the 19 subjects
- Click-ignore ad has a large value. Ads with a long display time often have a large value.

- When an ad is displayed, pupil size rises and falls after the display, regardless of the type of advertising stimulus. This is especially noticeable for click-ignore and front-page.

- There is no feature difference due to the negative/positive influence of the displayed image.

## 7.3 Analysis by Golden Cross

The analysis in Chapter 7.2 allowed us to show the features in the moving average of pupil size. In addition, we focus on Golden Cross, which is one of the analysis methods using moving averages.

Golden cross is methods used especially when predicting stock price trends. golden is calculated using two moving averages of different periods. This is one of the analysis methods using moving averages.

In Chapter 7.2, we used a 10-second moving average (5 seconds before and after). In this chapter, we added a 40-second moving average (20 seconds before and after) calculate the two crosses. Results of the calculated crosses are shown in Fig. 11.

<figure><img src="image_12.png" /><figcaption>Fig. 11 Analysis results of the golden cross</figcaption></figure>

Figure 11 shows that the Golden Cross tends to appear under the influence of the display of advertisements. This can be said from the fact that the frequency of appearance of the golden crosses during and after the display of advertisements is different.

When strong ads are displayed, both or one of the following two tend to occur.

- Pupil size is enlarged in the long-term moving average

- A golden cross occurs.

However, a series of golden crosses indicates areas where pupil fluctuation is small.

The results of a detailed review of each advertising stimulus are as follows. In click-ignore ads, the pupil size is often large and there is often a golden cross. Banner ads
tend to have a Golden Cross, although the pupil size is not large. Front-page ads tend to be somewhere in between.

As described above, by using golden cross, we were able to show the characteristics of pupil size. The above results suggest that advertising stimuli affect pupil size fluctuations.

## 7.4 Analysis by variance value using MACD

MACD (Moving Average Convergence Divergence) is the difference between two exponential smooth moving averages, one short term and one long term. MACD is also an indicator to examine the movement of stock prices. It was introduced in this study to process time series data as well as stock prices.

We compared the MACD values of the advertising stimulus intervals for each advertising stimulus. We focused on the variance values of MACD.

The variance values for subject 02, which has been shown so far as an example is shown in Fig. 12, and the variance values for all 19 subjects are shown in Fig. 13.

<figure><img src="image_13.png" /><figcaption>Fig. 12 Variance value of the interval under analysis for subject 02</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Fig. 13 Variance value of the interval to be analyzed for all subjects</figcaption></figure>

Fig. 13 shows that the variance value is higher in the order of Click-ignore > Front-page >No ad ≥ Banner. There was no difference in the positive/negative images as in the analysis in Chapter 7.2. Therefore, the variance values were calculated by eliminating the difference between positive/negative images. As above, the variance values for subject 02 and all subjects are shown in Fig. 14 and 15, respectively.
<figure><img src="image_15.png" /><figcaption>Fig. 14 Variance values for each display method for subject 02</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 15 Variance values by display method for all subjects</figcaption></figure>

Fig. 15 clearly shows that the variance value of CDMA results in Click-ignore > Front-page > No ad ≈ Banner. In most subjects (16/19), Click-ignore had the highest variance value. However, as shown in Fig. 16, there were some subjects who had different size relationships.

The features calculated from the pupil size may be useful indicators for detecting the display of strong advertising stimuli.

# 8 Conclusion and Interpretation

## 8.1 Discussion

In this study, we asked subjects to operate a web browser and gave them advertising stimuli. Multimodal information was obtained and analyzed during the experiment. In this paper, we focus especially on the features of pupil size in our analysis.

The results of the analysis suggested that the advertisement display affected the pupil size of the subjects. Due to strong advertising stimuli, the size of the pupil tends to increase significantly.

Furthermore, we used golden cross to express the characteristics.

Golden Crosses tend to appear due to subjects' responses to advertising displays. Golden crosses tend to occur regardless of ad type. This revealed that a golden cross was displayed regardless of the complexity of the targeted ad avoidance behavior.

Especially when click-ignore ads were displayed, large fluctuating golden crosses occurred.

Therefore, the golden cross appears regardless of the type of ad display, however, it can be said that the appearance interval and pupil size at the time of appearance differed depending on the advertisement type.

We calculate the variance values were calculated from the MACD values of all 19 subjects. The results showed that the features by MACD variance values tended to appear in the order of Click-ignore > Front-page > No ad ≈ Banner.
Analysis by variance values shows a strong tendency for strong stimuli to have larger variance values than weak stimuli. However, it is not possible to separate whether this difference is due to the influence of the strength of the advertising stimulus or the complexity of the ad avoidance action.

## 8.2 Interpretation

Based on the rationale defined in Table 2, we believe that this feature of variance indicates the size of the perceived "disturbance" during web browser operation. In the future, I would like to use this feature to build in a recommendation system. By utilizing the feature movement of the pupil size discovered in this paper, it is possible to determine if the user feels disturbed.

On the other hand, there were some subjects whose features were not easily visible. It is hypothesized that there were subjects who did not feel disturbed by the click-ignore advertisement even when it was displayed. Conversely, there were subjects who showed the feature movements shown in this paper in Banner ads. The reasons for these movements were thought to be largely due to individual differences based on the level of experience in PC operation and personality.

But that's just a hypothesis. In the future, we would like to analyze this hypothesis by focusing on multimodal information other than the pupil size feature focused on in this paper.

In addition, two types of images, positive and negative, were prepared for the experiment. As discussed in reference[7], there are many images in web advertising. We would like to refine the modeling of the effects of images by investigating various factors based on our findings.

From a medical point of view, it has been reported that the pupil size changes with stress, wakefulness, attention, and sleepiness [12]. The results were similar for the advertising stimuli in this study. We found that the pupil size was clearly changed by the web advertisement stimuli. Thus, we believe that the results of this experiment, including the medical findings, are generally valid.

## 8.3 Future Issues

Through this study, we found that a feature movement in pupil size appears depending on the type of web advertisement displayed. However, the following issues remain to be solved.

* Issue 1: Features for positive/negative images have not been found.

* Issue 2: Consideration of a judgment method that includes the individual differences indicated in 8.2

* Issue 3: Investigation of automatic decision-making methods using machine learning, etc.

First, for issue 1 and 2, we would like to use the pupil size data to study the calculation of features other than those concluded in this paper. Next, we would like to analyze multimodal information other than pupil size to show feature movements.
As for issue 3, we think it is necessary when building a recommendation system. Calculate other features based on the features concluded in this paper. Using these calculated features, we construct a machine learning model. As we continue to analyze the data acquired in this experiment, we will find new features of the movements and build a more robust machine learning model.

## References

1. Ministry of Internal Affairs and Communications, 2020 Internet Usage Status, https://www.soumu.go.jp/johotsusintokei/whitepaper/ja/r02/html/nd252120.html (in Japanese)

2. Yahoo! JAPAN Tech Blog, Introducing case studies of cross-sectional data utilization in the recommendation domain #machine learning, https://techblog.yahoo.co.jp/entry/20200331826447/ (in Japanese)

3. Research Plus, Awareness Survey on Internet Advertising, https://www.research-plus.net/html/investigation/report/index127.html (in Japanese)

4. Takatoshi Kato and Kazuo Ishida, A Method for Estimating Emotional States for Emotionally Friendly Dialogue Using Face Images, Proceedings of the 81st National Convention. 2019.1, P.63-64 (in Japanese)

5. Shigeto Furukawa, Makio. Kashino, Makoto Yoneya, and Hsin-I Liao. Reading the mind through the eyes -Heart-Touching-AI Key Technology, NTT Technical Journal 2016, vol.28 no.2 p22-25 (in Japanese)

6. Q. Zhao, S. Chang, F. M. Harper, and Joseph J. A. Konstan. Gaze prediction for recommender systems. In Proceedings of 10th ACM Conference on Recommender Systems. 2016. 131.

7. Simola, J., Kuisma, J., Öörni, A., Uusitalo, L., & Hyönä, J., The impact of salient advertisements on reading and attention on web pages., Journal of Experimental Psychology (2011): Applied, 17(2), 174–190.

8. Kimihiro Yamanaka, Mitsuyuki Kawakami, Convenient Evaluation of Mental Stress With Pupil Diameter, International Journal of Occupational Safety and Ergonomics (JOSE) 2009, Vol. 15, No. 4, 447–450

9. AWS(Amazon Web Service), https://aws.amazon.com/?nc1=h ls

10. OASIS(Open Affective Standardized Image Set), http://www.benedekkurdi.com/#!portfolio/project-4.html

11. Tobii Pro nano, https://www.tobiipro.com/product-listing/nano/

12. Naoto Hara, Pupillographic Stress, Arousal and Emotion Assessments in Visual Stress, Japanese journal of visual science 2012 vol. 33 no. 2 p. 47-51

13. Takanori Oyama, Itaru Kaneko, Fumitaka Ono, Junji Sone and Tsuyoshi Hanimura, Lecture evaluation using pupil diameter, FIT2011 (The 10th Forum on Information Technology). Third volume. p781-782 (in Japanese)