Venue: NAACL 2022

# Aspect-based Analysis of Advertising Appeals

for Search Engine Advertising

Soichiro Murakami\({}^{1,2}\), Peinan Zhang\({}^{1}\), Sho Hoshino\({}^{1}\), Hidetaka Kamigaito\({}^{2}\), Hiroya Takamura\({}^{2}\), Manabu Okumura\({}^{2}\)

\({}^{1}\)CyberAgent, Inc., \({}^{2}\)Tokyo Institute of Technology

{murakami_soichiro,zhang_peinen,hoshino_sho}@cyberagent.co.jp, kamigaito@lr.pi.titech.ac.jp, {tabamura,oku}@pi.titech.ac.jp

###### Abstract

Writing an ad text that attracts people and persuades them to click or act is essential for the success of search engine advertising. Therefore, ad creators must consider various aspects of advertising appeals (A\({}^{3}\)) such as the _price_, _product features_, and _quality_. However, products and services exhibit unique effective A\({}^{3}\) for different industries. In this work, we focus on exploring the effective A\({}^{3}\) for different industries with the aim of assisting the ad creation process. To this end, we created a dataset of advertising appeals and used an existing model that detects various aspects for ad texts. Our experiments demonstrated that different industries have their own effective A\({}^{3}\) and that the identification of the A\({}^{3}\) contributes to the estimation of advertising performance.

## 1 Introduction

Search engine advertising (SEA) displays an ad text that consists of a title and a description that are relevant to search queries in search engines, as illustrated in Figure 1. SEA plays an important role in sales promotion and marketing as it allows advertisers to approach users who are interested in specific search queries effectively Fain and Pedersen (2006). Ad creators write an ad text that attracts the attention of users and persuades them to click or act by introducing various aspects of advertising appeals (denoted as A\({}^{3}\) in this paper for short), such as _special deals_, as shown in Figure 1. However, products and services exhibit unique effective A\({}^{3}\) for different industries. For example, _limited offers_ may be attractive to users in the e-commerce (EC) industry, whereas the _quality of products_ may be more important in the automobile industry.

Thus, we argue that the suggestion of effective A\({}^{3}\) for various industries can offer assistance to ad creators. Therefore, we need to discover the effective aspects. However, although aspect-based text analysis has attracted significant attention in the review analysis for products and services Akhtar et al. (2017); Chen et al. (2019), it has received less focus in the advertisement field.

In this work, to deal with this problem, we defined the A\({}^{3}\) and constructed a dataset of ad texts that are annotated with A\({}^{3}\) in various industries as a first attempt towards assisting ad creators with A\({}^{3}\). Subsequently we developed an aspect detection model to identify different A\({}^{3}\) and performed correlation analysis between A\({}^{3}\) and the click-through rate (CTR), which is used for supporting ad creation, as an advertising performance metric to explore the effective aspects in different industries. Furthermore, we investigated the effectiveness of A\({}^{3}\) in CTR prediction as a potential application for ad creation support.

Through correlation analysis in our experiments, we found that different industries exhibit unique effective A\({}^{3}\). Furthermore, we found that the identification of the A\({}^{3}\) contributes to the CTR prediction.

## 2 Related Work

Ad Creation SupportAttempts have been made to perform automatic generation of ad texts and keywords Ravi et al. (2010); Hughes et al. (2019); Kamigaito et al. (2021) as well as the estimation of advertising performance metrics such as the CTR Richardson et al. (2007); Zhang et al. (2014); Mishra et al. (2021) to support the ad creation process. In this work, we tackle the discovery of the effective A\({}^{3}\) for various industries and apply the A\({}^{3}\) to CTR prediction with the goal of improving the efficiency

Figure 1: Example ad text and its corresponding A\({}^{3}\).



of the ad creation process.

Aspect-based Text AnalysisAlthough aspect-based text analysis has attracted significant attention, the majority of studies have been limited to specific domains such as hotels, restaurants, and home appliances (Pontiki et al., 2016; Akhtar et al., 2017; Chen et al., 2019). Moreover, as the product review analysis focuses on the aspects of each product, the defined aspects are extremely fine grained (e.g., the _modes_, _energy efficiency_, and _noise_ for refrigerators (Li et al., 2020)). These aspects are not suitable for ad creation because ad creators must deal with ad texts for various products in multiple industries. Therefore, ad creators are required to consider numerous A3. In this study, we carefully designed labels that cover the A3 for the general purpose of exploring these in a wide range of industries. Furthermore, we explored methods for aspect detection, as in the previous work (Bagheri et al., 2013), as well as the identification of the effective aspects in terms of advertising performance metrics such as the CTR.

Footnote 1: [https://ads.google.com/](https://ads.google.com/)

## 3 Construction of A\({}^{3}\) Dataset

### Data Collection

We constructed a dataset of advertising appeals to understand the A\({}^{3}\) in ad texts. Many A\({}^{3}\) exist in real-world advertisements, including product features, price, and campaigns. We collected 782,158 ads from March 1, 2020 to February 28, 2021 through Google Ads,1 which is an online advertising platform, to cover the expressions of advertising appeals in a wide range of industries. In this work, we used ads in Japanese. Each ad consists of a title, a description, and a landing page (LP), which is a web page for a specific advertising campaign. We used the meta-description2 of each LP as the LP content. We sampled 5,000 ad texts for each advertiser to alleviate the bias owing to a different quantity of ad texts for the advertisers. Moreover, we excluded ad texts that comprised less than 15 characters or more than 200 characters. The aforementioned two steps yielded 34,952 ad texts. Furthermore, we excluded duplicates and highly similar texts using the normalized Levenshtein distance metric (Levenshtein, 1966; Greenhill, 2011), because the majority of the ad texts were created from templates for the sake of cost efficiency (Fujita et al., 2010). As a result, we collected 2,738 ad texts consisting of 666 titles, 1,532 descriptions, and 440 LP contents from 13 types of industries.3 We provide the detailed statistics of the collected ad texts in Appendix A.

Footnote 2: A meta-description is an HTML attribute that provides a brief summary of a web page, such as an LP.

Footnote 3: EC, Media, Finance, VOD&eBook, Cosmetics, Human resources, Education, Travel, Automobile, Entertainment, Real estate, and Beauty&health

### Label Types and Annotation Scheme

Owing to the existence of various A\({}^{3}\), we believe that the systematic organization of the A\({}^{3}\) can aid the ad creation process. We manually defined aspect labels in the following two phases. First, we conducted a preliminary analysis of the collected ad texts and found that approximately eight aspects appeared: _special deals_, _quality_, _problem solving_, _speed_, _user-friendliness_, _limited offers_, _product lineup_, and _trend_. Second, we presented these aspects and the collected ad texts to experienced ad creators and asked for their opinions on the A\({}^{3}\) with the aim of refining the aspect labels. Consequently, the ad creators suggested that we further subdivide _special deals_ and _limited offers_. For example, _special deals_ was subdivided into _discount price_, _reward points_, _free_, and _special gift_. The reason for this is that there are differences in the strength of the aspects between _free_ and _special gift_, even though they appear to be similar. Furthermore, _largest/no.1_ was added as another aspect label because it attracts a lot of users.

Table 1 lists the A\({}^{3}\) that we manually defined. Detailed descriptions and examples are provided in Appendix B. Finally, we carefully designed a hierarchical scheme for A\({}^{3}\) to help ad creators and annotators to understand the differences between

\begin{table}
\begin{tabular}{r l r|c c c} \hline \multicolumn{2}{l}{**Labels**} & \multicolumn{1}{c}{**\#spans**} & \multicolumn{1}{c}{**Labels**} & \multicolumn{1}{c}{**\#spans**} \\ \hline (1) & Special deals & 343 & (12) Limited offers & 52 \\ (2) & Discount price & 120 & (13) Limited time & 61 \\ (3) & Reward points & 85 & (14) Limited target & 114 \\ (4) & Free & 430 & (15) First-time limited & 25 \\ (5) & Special gift & 126 & (16) Track record & 75 \\ (6) & Features & 1,360 & (17) & Largest/no. 1 & 141 \\ (7) & Quality & 65 & (18) Product lineup & 258 \\ (8) & Problem solving & 17 & (19) Trend & 99 \\ (9) & Speed & 142 & (20) Others & 182 \\ (10) & User-friendliness & 337 & (21) Story & 98 \\ (11) & Transportation & 89 & & \\ \hline \end{tabular}
\end{table}
Table 1: A\({}^{3}\) and statistics of annotated dataset, where “#spans” represents the number of span texts annotated with each label.

 

the labels. The aspect hierarchy consists of five types of coarse-grained labels including _special deals_, which are underlined in Table 1, and 16 types of fine-grained labels such as _discount price_.

Because an ad text often contains multiple expressions of advertising appeals, as depicted in Figure 1, we defined an advertising expression as a span text to be annotated. For example, annotators provide the aspect label (e.g., _special deals_) for the span text "_best price guarantee_." Each span was annotated during the annotation work. Moreover, we allowed the annotators to provide multiple labels for each span because an expression of advertising appeals may contain multiple aspects. For example, the advertising expression "_members get an extra 20% off_" contains two aspects _discount price_ and _limited-target offer_, because it means that only users belonging to a membership program can receive an extra 20% discount.

### Annotation Process

We recruited six participants who worked at an advertising agency. We separated 2,738 collected ad texts into two sets consisting of 1,100 and 1,638 texts, and assigned three participants to each set. We presented a one-hour lecture to the participants to explain the detailed definitions of the labels and to provide annotation examples. Furthermore, we asked them to annotate 30 ad texts that were separated from the collected dataset as a practice session. After the session, we answered questions from the participants. During the annotation, we answered any additional questions from them and shared information when a difficult case appeared, which was relatively rare.

### Annotated Dataset Statistics

Table 1 displays the statistics of the annotated dataset. We adopted annotated spans only if at least two of the three annotators for each span text agreed with their boundaries and labels. The annotation work for the 2,738 ad texts required a total of 42 hours; thus, the average time per ad text was 55.2 seconds. A single ad text contains 1.54 spans on average. Furthermore, we calculated the Cohen's Kappa coefficients (\(\kappa\)) between the tokens annotated by different pairs of annotators to determine the inter-annotator agreement. Moreover, following the previous work Brandsen et al. (2020), we also report the \(F_{1}\) scores that were calculated between the spans annotated by different pairs of annotators, where we considered one annotation as the ground truth and another as the prediction. We obtained relatively high agreement among the annotators: \(\kappa=0.612,F_{1}=0.451\).

## 4 Aspect Detection Model

We investigate two existing models for aspect detection, i.e., the span-based Zheng et al. (2019) and document-based (doc-based) models Devlin et al. (2019). These models receive an ad text \(\mathbf{x}=(x_{i})^{|\mathbf{x}|}_{i=1}\) as an input and predict aspect labels \(\mathbf{y}=(y_{i})^{K}_{i=1}\), where \(x_{i}\) and \(y_{i}\) represent a token of an ad text and a binary label for each aspect label, respectively. As each span may contain multiple aspects, both models perform label prediction in the form of multi-label classification Kurata et al. (2016). \(K\) is the number of aspect labels defined in Table 1. We consider an expression of the advertising appeals in an ad text, such as "_best price guarantee_" in Figure 1, to be a _span_. We use \(S(i,j)\) to represent the span from \(i\) to \(j\), where \(1\leq i<j\leq|\mathbf{x}|\). The span-based model consists of two steps: (i) extracting a span \(S(i,j)\) from \(\mathbf{x}\) and (ii) predicting the aspect labels \(\mathbf{y}\) for each span. In contrast, the doc-based model predicts the aspect labels \(\mathbf{y}\) for an entire ad text \(\mathbf{x}\). We employed a pre-trained BERT Devlin et al. (2019) for both models owing to the limited amount of the annotated dataset.

### Span-Based Model

Figure 2 presents an overview of the span-based model. The task of extracting a span from an ad text can be considered as named entity recognition, and we introduce the boundary-aware neural model proposed by Zheng et al. (2019). We consider characters as a unit (token) in the span-based model. We use the BIOE scheme to create boundary labels \(\mathbf{l}=(l_{i})^{|\mathbf{x}|}_{i=1}\) for the input tokens \(\mathbf{x}\). We feed \(\mathbf{x}\) into the BERT to obtain a vector \(h_{i}\) for \(x_{i}\) for span detection. Subsequently, we obtain the distribution of the boundary labels \(v_{i}\in\mathbb{R}^{L}\) by applying a multilayer perceptron (MLP) \(v_{i}=\texttt{MLP}(h_{i})\), where \(L\) is the number of boundary types (BIOE). We also use a linear-chain conditional random field (CRF)

Figure 2: Overview of the span-based model.

 

(Lafferty et al., 2001) to model the dependencies of the boundary labels (e.g., label E must appear after B or I). As a result, we can obtain the boundary labels that are predicted by viterbi decoding for the input \({\mathbf{x}}\).

For label prediction, we create a vector representation \(h^{avg}_{(i,j)}\) for a span \(S(i,j)\) using the average of the output vectors of the BERT (i.e., \(h_{i},h_{i+1}\cdots h_{j}\)). Thereafter, we obtain the probability that each span \(S(i,j)\) belongs to the aspect labels \({\mathbf{y}}\) by applying an MLP and a sigmoid function \({\mathbf{m}}=\texttt{Sigmaid}(\text{MLP}(h^{avg}_{(i,j)}))\), where \({\mathbf{m}}=(m_{k})_{k=1}^{K}\) and \(m_{k}=p(y_{k}=1|S(i,j))\). For example, in Figure 2, the expression "_Get the First Month Free_" is detected as a span, and the model predicts two aspect labels _free_ and _first-time limited offer_ for the detected span.

### Doc-Based Model

Although the span-based model offers the advantage of detecting a specific expression using span detection, we are concerned that errors in span detection could affect label prediction. Therefore, we also introduce the doc-based model as an alternative to the span-based model.

The doc-based model is a BERT-based classification model. Following the original BERT-based classifier (Devlin et al., 2019), the doc-based model consists of a BERT and an MLP, which take an entire ad text \({\mathbf{x}}\) as an input and outputs labels \({\mathbf{y}}\). Specifically, we first input the ad text \({\mathbf{x}}\) into the BERT and obtain the vector representation \(h^{[\texttt{CLS}]}\) for a [CLS] token. Subsequently, we feed the vector \(h^{[\texttt{CLS}]}\) into the MLP to obtain the probability that the ad text \({\mathbf{x}}\) belongs to the aspect labels \({\mathbf{y}}\) as a multi-label classification task \({\mathbf{m}}=\texttt{Sigmaid}(\text{MLP}(h^{[\texttt{CLS}]}))\), where \({\mathbf{m}}=(m_{k})_{k=1}^{K}\) and \(m_{k}=p(y_{k}=1|{\mathbf{x}})\).

## 5 CTR Prediction with A\({}^{3}\)

Within the context of ad creation support, the estimation of advertising performance for an ad text (e.g., the CTR) plays a key role in both the improvement and cost efficiency of the ad creation because it helps us understand the user's interest. Therefore, we also investigate whether the A\({}^{3}\) contributes to the prediction of the advertising performance. For this task, we input an ad text \({\mathbf{x}}\) consisting of a title and description, an industry type of the ad \({\mathbf{t}}\) (e.g., EC), and keywords \({\mathbf{k}}\) (e.g., _tokyo_ and _hotel_). We also introduce the predicted aspect labels \(\hat{{\mathbf{y}}}\) (e.g., _features_) for \({\mathbf{x}}\) as additional features, which were detected by either the span-based or doc-based model. In this case, we use the CTR \({\mathbf{z}}\in[0,1]\) as the advertising performance (\(\text{CTR}=\texttt{clicks}\div\) impressions).

Figure 3 presents an overview of the regression model. Similarly to recent work (Mishra et al., 2021), we design this regression model based on the BERT. In the model, we feed the three types of tokens \({\mathbf{x}}\), \({\mathbf{t}}\), \({\mathbf{k}}\) into the BERT to obtain the vector \(h^{[\texttt{CLS}]}\) for a [CLS] token. Subsequently, we input \(h^{[\texttt{CLS}]}\) and the aspect labels \(\hat{{\mathbf{y}}}\) for the ad text \({\mathbf{x}}\) into the following MLP. Thereafter, we obtain the concatenated vector \(h^{out}=[h^{ad};h^{aspect}]\), where "\(:\)" is a concatenation operator. The final MLP then predicts a CTR score \({\mathbf{z}}\) from \(h^{out}\) as \({\mathbf{z}}=\texttt{Sigmaid}(\text{MLP}(h^{out}))\).

## 6 Experiments

We conducted experiments on three tasks: (1) aspect detection for the A\({}^{3}\), (2) correlation analysis between the A\({}^{3}\) and CTR, and (3) CTR prediction.

### Experimental Settings

DatasetWe used the annotated dataset in Table 1 for the aspect detection. We separated the dataset into 1,857 samples for training, 465 for development, and 410 for testing after excluding 6 ad texts that we determined were inappropriately annotated. We collected 168,412 pairs of ad texts, keywords, and industry types from March 1, 2020 to February 28, 2021 through Google Ads for the CTR prediction. We carefully separated the dataset into 136,352, 16,084, and 15,976 samples for training, development, and testing, respectively. The detailed statistics of the dataset for the CTR prediction are presented in Appendix C. We used the training dataset for the CTR prediction for the correlation analysis between the CTR and A\({}^{3}\). We used the campaign ID of each ad for data division to prevent leakage between the datasets.

ImplementationWe used the character-level BERT4 for the span-based model, and the word

Figure 3: Overview of the CTR prediction model.

 

level BERT5 for the doc-based model and CTR prediction. We fine-tuned the models on the dataset and applied an early stopping strategy with 10 epochs. The training was stopped if there was no improvement in the validation loss for three consecutive epochs in all experiments. Further implementation details are described in Appendix D.

Footnote 5: [https://huggingface.co/cl-tohoku/bert-base-japanese](https://huggingface.co/cl-tohoku/bert-base-japanese)

Evaluation MetricsWe calculated the \(F_{1}\) scores of the aspect labels for the aspect detection. For the span-based model, a detected label was considered as a true positive if both its span and label were correctly detected. We used the area under the receiver operating characteristic curve (AUC) Fawcett (2006), which is a widely used metric in the field of CTR prediction Zhou et al. (2018); Xiao et al. (2020). Moreover, we used the root-mean-squared error (RMSE) and mean absolute error (MAE) to measure the differences between the ground-truth and predicted scores.

### Aspect Detection

In this experiment, we evaluated two models, the span-based and doc-based models. As errors in the span prediction may affect the label prediction in the span-based model, we also introduced the Oracle model, which predicts their labels, provided with _oracle_ spans, in addition to the Pred model, which predicts both the spans and labels.

The evaluation results for the aspect detection are presented in Table 2. The doc-based model outperformed the span-based model, including the Oracle model, for most A3. As the Pred model is required to predict both the spans and labels correctly, its task is relatively more difficult than that of other models. In fact, we found that the \(F_{1}\) score for the span detection is 0.69 for the Pred model. Therefore, we conclude that it is the reason why the macro-average \(F_{1}\) score of Pred was lower than those of the doc-based and Oracle models.

Footnote 3: footnotemark:

In the comparison between the Oracle and doc-based models, the doc-based model outperforms the Oracle model. We hypothesize that its training objective for the span-based model is more difficult as it is more fine grained than the doc-based model.

We observed that the scores for _free_, _speed_, and _largest/no. 1_ are high in the doc-based model. This implies that the advertising expressions for these aspects are relatively monotonous and easy to detect compared to the other aspects. For example, the advertising expression "_free shipping_," which belongs to _free_, often occurs frequently in ad texts for a wide range of industries. The aspect detection was difficult for several aspects in which the numbers of annotated cases were limited, such as (8) and (15), as indicated from Tables 1 and 2. Hence, they exhibited an \(F_{1}\) score of 0.00.

We also conducted an analysis of the attention in the doc-based model to understand to which signals the model attended in the aspect detection. Figure

Figure 4: Visualization of attention weights in the doc-based model. Each example consists of the original Japanese ad text with the literal translation for each subword and the corresponding English ad text.

 

4 depicts the visualized attention patterns with respect to the [CLS] token of the final layer of the BERT. We found that many of the attention heads attend to the words "_design_" and "_for free_" for the ad text (a) and (b), respectively. This suggests that the doc-based model classified the ad text (a) and (b) as _features_ and _free_, respectively, because these words were related to the aspects.

### Correlation between Aspects and CTR

To realize the ad creation process considering the A\({}^{3}\), we analyzed which A\({}^{3}\) were effective in each industry through correlation analysis between the CTR6 and the aspect labels that were predicted by the doc-based model. Because the aspect labels are binary for each aspect (e.g., whether or not each aspect is included in an ad text) and the CTR is continuous, we used the point-biserial correlation coefficient **r** for the analysis. Table 3 lists the point-byerial correlation coefficients **r** between the aspect labels and the CTR. We investigated the correlation among the industry types _VOD&eBook (eBook)_, _EC_, _Finance (Fin)_, _Human resources (HR)_, and _Travel_. As indicated in **bold text** in Table 3, we observed a weak correlation (0.25 < |**r**| < 0.5) between the CTR and the labels, such as (3) _reward points_ for _Finance_. This implies that ad texts that include effective A\({}^{3}\) tend to attract more attention from users. However, there was no correlation with regard to the other aspects. This may be because (1) _features_, for example, is considered to be a general-purpose aspect and can be used in any situation.

Footnote 6: We used the _actual_ CTR for each ad rather than the _predicted_ CTR.

Based on the above insights, we also investigated the expressions for the effective A\({}^{3}\) in our annotated dataset. For example, regarding the _VOD&eBook_ industry, we found that the expression "_one of the largest websites in Japan_" ([R 

better correlation and prediction for the CTR.

## 7 Conclusions

In this work, we have explored the effective A\({}^{3}\) by means of aspect detection and correlation analysis towards ad creation support with the A\({}^{3}\). Our experimental results demonstrated that each industry exhibits unique effective A\({}^{3}\) and that identification of the A\({}^{3}\) can contributes to CTR prediction.

We demonstrate two possible directions for future studies. First, we will investigate whether introducing the effective A\({}^{3}\) in the ad creation process can help ad creators write effective ad texts in real-world applications. Second, we will develop an aspect-aware model to automatically generate ad texts to support the ad creation process. For the latter, we will train the model with a dataset that includes pairs of ad texts and their corresponding aspect labels predicted using aspect detection.

## References

* Akhtar et al. (2017) Nadeem Akhtar, Nashez Zubair, Abhishek Kumar, and Tameem Ahmad. 2017. Aspect based sentiment oriented summarization of hotel reviews. _Procedia Computer Science_, 115:563-571. 7th International Conference on Advances in Computing & Communications.
* Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takenu Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In _Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
* Bagheri et al. (2013) Ayoub Bagheri, Mohamad Saraee, and Franciska de Jong. 2013. An unsupervised aspect detection model for sentiment analysis of reviews. In _Natural Language Processing and Information Systems_, pages 140-151.
* Brandsen et al. (2020) Alex Brandsen, Suzan Verberne, Milco Wansleeben, and Karsten Lambers. 2020. Creating a dataset for named entity recognition in the archaeology domain. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 4573-4577.
* Chen et al. (2019) Qibin Chen, Junyang Lin, Yichang Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019. Towards knowledge-based personalized product description generation in e-commerce. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3040-3050.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4171-4186.
* Fain and Pedersen (2006) Daniel C. Fain and Jan O. Pedersen. 2006. Sponsored search: A brief history. _Bulletin of the American Society for Information Science and Technology_, 32(2):12-13.
* Fawcett (2006) Tom Fawcett. 2006. An introduction to ROC analysis. _Pattern Recognition Letters_, 27(8):861-874.
* Fujita et al. (2010) Atsushi Fujita, Katsuhiro Ikushima, Satoshi Sato, Ryo Kamite, Ko Ishiyama, and Osamu Tamachi. 2010. Automatic generation of listing ads by reusing promotional texts. In _Proceedings of the 12th International Conference on Electronic Commerce: Roadmap for the Future of Electronic Business_, pages 179-188.
* Greenhill (2011) Simon J. Greenhill. 2011. Levenshtein distances fail to identify language relationships accurately. _Computational Linguistics_, 37(4):689-698.
* Hughes et al. (2019) J. Weston Hughes, Keng-hao Chang, and Ruofei Zhang. 2019. Generating better search engine text advertisements with deep reinforcement learning. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2269-2277.
* Kamigaito et al. (2021) Hidetaka Kamigaito, Peinan Zhang, Hiroya Takamura, and Manabu Okumura. 2021. An empirical study of generating texts for search engine advertising. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers_, pages 255-262.
* Kurata et al. (2016) Gakuto Kurata, Bing Xiang, and Bowen Zhou. 2016. Improved neural network-based multi-label classification with better initialization leveraging label co-occurrence. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 521-526.
* Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of the Eighteenth International Conference on Machine Learning_, pages 282-289.
* Levenshtein (1966) Vladimir Iosifovich Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. _Soviet Physics Doklady_, 10(8):707-710.
* Li et al. (2020) Haoran Li, Peng Yuan, Song Xu, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020. Aspect-aware multimodal summarization for chinese e-commerce products. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 8188-8195.

 

Shaunak Mishra, Changwei Hu, Manisha Verma, Kevin Yen, Yifan Hu, and Maxim Sviridenko. 2021. Tsi: An ad text strength indicator using text-to-ctr and semantic-ad-similarity. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 4036-4045.
* Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphee De Clercq, Veronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud Maria Jimenez-Zafra, and Gulsen Eryigit. 2016. SemEval-2016 task 5: Aspect based sentiment analysis. In _Proceedings of the 10th International Workshop on Semantic Evaluation_, pages 19-30.
* Ravi et al. (2010) Sujith Ravi, Andrei Broder, Evgeniy Gabrilovich, Vanja Josifovski, Sandeep Pandey, and Bo Pang. 2010. Automatic generation of bid phrases for online advertising. In _Proceedings of the Third ACM International Conference on Web Search and Data Mining_, pages 341-350.
* Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: Estimating the clickthrough rate for new ads. In _Proceedings of the 16th International Conference on World Wide Web_, pages 521-530.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45.
* Xiao et al. (2020) Zhibo Xiao, Luwei Yang, Wen Jiang, Yi Wei, Yi Hu, and Hao Wang. 2020. Deep multi-interest network for click-through rate prediction. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 2265-2268.
* Zhang et al. (2014) Yuyu Zhang, Hanjun Dai, Chang Xu, Jun Feng, Taifeng Wang, Jiang Bian, Bin Wang, and Tie-Yan Liu. 2014. Sequential click prediction for sponsored search with recurrent neural networks. In _Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence_, pages 1369-1375.
* Zheng et al. (2019) Changmeng Zheng, Yi Cai, Jingyun Xu, Ho-feng Leung, and Guandong Xu. 2019. A boundary-aware neural model for nested named entity recognition. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_, pages 357-366.
* Zhou et al. (2018) Guorui Zhou, Xiaoqing Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for clickthrough rate prediction. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1059-1068.

 

Collected Ad Texts for Annotation

Table 7 lists the detailed statistics of the collected ad text. We collected 2,738 ad texts comprising 666 titles \({\mathbf{x}^{title}}\), 1,532 descriptions \({\mathbf{x}^{desc}}\), and 440 LP contents \({\mathbf{x}^{lp}}\) from 13 industries: _EC_, _Media_, _Finance_, _VOD&eBook_, _Cosmetics_, _Human resources_, _Education_, _Travel_, _Automobile_, _Entertainment_, _Real estate_, and _Beauty&Health_.

## Appendix B Descriptions and Examples of A\({}^{3}\)

Table 5 lists the detailed descriptions and examples of A\({}^{3}\) that we have defined. For example, the expression "_enjoy free shipping_" is labeled with (4) _free_, as it represents free offers for products or services. In the table, "#spans" represents the number of span texts annotated with each label.

## Appendix C Dataset for CTR Prediction

Table 8 lists the detailed statistics of the datasets used for CTR prediction. We carefully separated the dataset into 136,352, 16,084, and 15,976 samples for training, development, and testing, respectively. For correlation analysis between the CTR and aspect labels of advertising appeals, we used the training dataset for CTR prediction.

## Appendix D Additional Implementation Details

Table 6 lists the implementation details, e.g., hyperparameters, for the aspect detection and CTR prediction models. We developed our models using pre-trained BERT models, which are publicly available from the Transformers library Wolf et al. (2020).8 The framework is available under the Apache 2.0 license. We trained the models with a Tesla V100 GPU on the Google Cloud Platform, which is the cloud computing infrastructure. Moreover, we performed a hyperparameter search, using Optuna Akiba et al. (2019) with default parameters for the aspect detection models on the validation set. In the experiment, the hyperparameter search is limited to 30 trials. Therefore, we performed our experiments in a single run.

Footnote 8: [https://huggingface.co/cl-tohoku](https://huggingface.co/cl-tohoku)

We used CRF and binary cross-entropy (BCE) loss for span detection and label prediction in the 



\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & **Aspect labels** & **Description \& Example** & **\#spans** \\ \hline
1 (1) & Special deals & Expressions representing special deals (e.g., _Compare hotels and save money_) & 343 \\
2 (2) & Discount price & Specific discount rate or amount (e.g., _Buy 1 get 1 50% off_) & 120 \\
3 (3) & Reward points & Customers can earn points (e.g., _Use our app to earn points_) & 85 \\
4 (4) & Free & Free offer for products or services (e.g., _Enjoy free shipping_) & 430 \\
5 (5) & Special gift & Special gifts or presents for customers (e.g., _Joint today and get a free brush set_) & 126 \\ \hline
6 (6) & Features & Features of services or products (e.g., _Ergonnically designed to protect children_) & 1,360 \\
7 (7) & Quality & Top-quality or high-grade services (e.g., _Find premium kitchen appliances_) & 65 \\
8 (8) & Problem solving & Solutions to customer problems (e.g., _Get bright, clear skin_) & 17 \\
9 (9) & Speed & Speed of delivery and services (e.g., _Fast \& fire shipping_) & 142 \\
10 (10) & User friendliness & Usability of services and products (e.g., _Quick, simple, and easy to use_ ) & 337 \\
11 (11) & Transportation & Convenience of transportation (e.g., _Centrally located in the heart of Tokyo_) & 89 \\ \hline
12 (12) & Limited offers & Limited availability of services and products (e.g., _Limited to 1,000 items per day_) & 52 \\
13 (13) & Limited-time offer & Offers available for a limited time only (e.g., _Three days only at 20\% off_) & 61 \\
14 (14) & Limited-target offer & Offers available for target customers only (e.g., _Discount for members only_) & 114 \\
15 (15) & First-time limited offer & Limited offers for first-time customers (e.g., _Take 15\% off your first order_) & 25 \\ \hline
16 (16) & Track record & Track records of services or companies (e.g., _45M+ users worldwide_) & 75 \\
17 (17) & Largest/no. 1 & Largest/No. 1 products or services (e.g., _Boston’s no. 1 hair salon_) & 141 \\
18 (18) & Product lineup & Wide range of products or stores (e.g., _Large selection of hotels_) & 258 \\
19 (19) & Trend & Popularity or favorable reputation (e.g., _Top trending shoes and boots_) & 99 \\ \hline
20 (20) & Others & Other advertising appeals (e.g., _An experience like no other_) & 182 \\
21 (21) & Story & Synopsis of a movie or drama (e.g., _After Peter Parker is bitten by \(a\cdots\)_) & 98 \\ \hline \hline \end{tabular}
\end{table}
Table 5: A\({}^{3}\) and statistics of annotated dataset.