Venue: NAACL 2025

# AdTEC: A Unified Benchmark for Evaluating Text Quality

in Search Engine Advertising

Peinan Zhang\({}^{\spadesuit}\), Yusuke Sakai\({}^{\diamondsuit}\), Masato Mita\({}^{\spadesuit}\), Hiroki Ouchi\({}^{\spadesuit\diamondsuit}\), Taro Watanabe\({}^{\diamondsuit}\)

\({}^{\clubsuit}\)CyberAgent, \({}^{\diamondsuit}\)Nara Institute of Science and Technology

{zhang_peinan,mita_masato}@cyberagent.co.jp

{skakai.yusuke.sr9,hiroki.ouchi,taro}@is.naist.jp

###### Abstract

As the fluency of ad texts automatically generated by natural language generation technologies continues to improve, there is an increasing demand to assess the quality of these creatives in real-world setting. We propose AdTEC, the first public benchmark to evaluate ad texts from multiple perspectives within practical advertising operations. Our contributions are as follows: (i) Defining five tasks for evaluating the quality of ad texts, as well as constructing a Japanese dataset based on the practical operational experiences of advertising agencies, which are typically maintained in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on this dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark. Our results show that while PLMs have a practical level of performance in several tasks, humans continue to outperform them in certain domains, indicating that there remains significant potential for further improvement in this area. The dataset is publicly available at: [https://cyleeragentailab.github.io/AdTEC](https://cyleeragentailab.github.io/AdTEC).

## 1 Introduction

Online advertising, especially sponsored search advertising (Figure 1), is a dominant sector for vendors to promote their products, and the market size is estimated to grow by billions of dollars over the next few years Murakami et al. (2023). To meet the increasing demands of advertising operations (AdOps), such as creating ad texts from product information (Step 2 in Figure 2), the remarkable success of natural language generation (NLG) by pre-trained language models (PLMs) Dong et al. (2021); Vaswani et al. (2017); Brown et al. (2020); Touvron et al. (2023) has given a boost to practical applications Hughes et al. (2019); Kamigaito et al. (2021); Golobokov et al. (2022), making advertising a huge industrial use case for natural language processing (NLP).

In this paper, we focus on evaluating the quality of ad texts generated by such models. We refer to this process as _ad text evaluation_, as depicted in Step 3 of Figure 2. Ad text evaluation is crucial because low-quality ad texts , which may lack fluency, present inappropriate appeals, or convey misleading representations, can negatively impact advertisers. Since it is costly and not scalable to have humans verify the quality of each text in high-volume domains, such as sponsored search advertising, there is a high demand for the development of automated quality estimators for ad texts. The quality has multiple dimensions, such as appropriate wording, effective appeals, consistency between ad text and product information, and high predicted performance. Although these dimensions should be included when evaluating automatic quality estimators, no benchmark currently exists to assess them. Consequently, the bottleneck lies in

Figure 1: Overview of sponsored search ad and its key terms.



verifying the quality of ad texts despite the ability to generate numerous creatives automatically, which hinders delivery volume scalability. Thus, we aim to construct a benchmark to evaluate the quality of ad texts.

The primary challenge in constructing an ad text evaluation benchmark is the absence of a clear definition for tasks [146, 23]. The lack of domain knowledge in AdOps complicates the understanding of high-quality ad text standards and the accurate definition of tasks. However, the AdOps workflow is complex, relies on various platforms and formats, and encompasses multiple methodologies and metrics. In addition, only a few companies possess the expertise to operate online advertising at scale. Owing to legal and contractual obligations, advertising workflows and data are predominantly managed in-house, leading to a lack of publicly available datasets. This scarcity makes it challenging to systematically reproduce and validate diverse methodologies in academia. Consequently, research activity is limited in the advertising field, which leaves potential issues unaddressed and delays the application and development of cutting-edge technology.

To address these challenges, we propose **AdTEC**, the first public benchmark that defines and unifies tasks based on generalized AdOps workflows. Our major contributions are as follows:

#### The First Public Dataset on Ad Text Evaluation.

We organized real-world advertising workflows and carefully designed five tasks to evaluate the quality of ad texts. This was based on the practical operational experiences of advertising agencies, using Japanese data as a case study. We are releasing this dataset1, marking it as the first publicly available dataset for ad text evaluation, as such data are typically maintained in-house and difficult to obtain.

Footnote 1: [https://cyberagentaillab.github.io/AdTEC](https://cyberagentaillab.github.io/AdTEC)

**Benchmark Experiments.** We validated the performance of existing PLMs, such as BERT, RoBERTa, and large language models (LLMs), as well as human evaluators on our proposed benchmark.

**Dataset Analysis.** We analyzed the characteristics and identified potential issues of the dataset through experiments, demonstrating that our benchmark is challenging and highlighting potential areas for improvement and future research.

## 2 Understanding Sponsored Search Advertising and its Workflow

In sponsored search advertising, titles and descriptions of ads that are relevant to keywords entered into search engines by customers appear as part of the search results. As shown in Figure 1, when a customer clicks an ad's URL, they are directed to a web page known as a landing page (LP). The LP contains texts and images related to the advertiser's products, prompting the customer to view the product or make purchase decisions.

Operating such ads requires high expertise, owing to the variety of delivery platforms, properties, and formats. To gain insights, we interviewed two types of experts familiar with AdOps: those overseeing AdOps departments at ad agencies and those directly involved in on-site operations. Through these interviews, we explored the intricacies of AdOps, generalized the workflow,

Figure 2: Generalized AdOps workflow described in §2: (1) The advertiser creates an LP to promote a product. (2) Based on the product information in the LP and target customers, text and graphics are designed by creators. (3) The creatives are evaluated based on fluency, attractiveness, regulations, legality, and other factors. (4) Once the creatives pass the quality evaluation, they are submitted to a delivery platform. (5) Customers respond to the displayed ads, such as page views, clicks, and purchases. (6) Based on the customer engagement, ad performance is reported back to the advertiser, and Steps 1-5 are repeated to improve the quality of the LP and ads.

 

and delineated six steps, as illustrated in Figure 2. In this work, we focus mainly on Step 3 of quality evaluation.

## 3 Constructing AdTEC

Our goal is to develop a benchmark that captures multi-dimensional aspects of quality that are relevant to practical scenarios. In addition, these aspects will be carefully curated to provide value for both real-world applications and various research purposes.

### Task Design

We followed the workflow outlined in SS2 and asked the same experts from the interview to identify crucial parts within Step 3 in Figure 2. Based on their insights, we generalized and defined tasks that adhere to the principle of evaluating ad texts either directly or indirectly. _Direct_ evaluation tasks are used to test texts against strict criteria, such as a binary pass/fail outcome or numerical score to quantify the text quality. These tasks serve as a checklist to ensure that minimum delivery standards are met. _Indirect_ evaluation tasks are used to assist human evaluators in reviewing or refining texts or serve as a bridge to connect downstream tasks. Based on the above principles, we designed five tasks: three direct evaluation tasks (Ad Acceptability, Ad Consistency, and Ad Performance Estimation) and two indirect evaluation tasks (A\({}^{3}\) Recognition and Ad Similarity).2

Footnote 2: Note that all examples in this paper are translated from Japanese into English for presentation brevity.

Ad AcceptabilityAs most ad delivery platforms impose text length restrictions, minor grammatical errors are tolerated to enhance the readability and engage customers within limited space. However, excessive compression can mislead customers, and such poor-quality ads should be detected before delivery to avoid negative impacts on the advertiser. To assess this, we defined the Ad Acceptability task, which predicts the acceptance of overall quality with binary labels: acceptable/unacceptable. Based on expert feedback, unacceptable ad phenomena include "collapsing symbols," "unnatural repetition," "incomprehensible meaning', and "underspecified/vague". This differs from the general concept of linguistic acceptability, which checks for grammatical correctness, such as CoLA Warstadt and Bowman (2019). Examples of Ad Acceptability are shown in Table 1. The ad text "_Engineeer's career / Engineer's Jobs_" is _unacceptable_ because the meaning is duplicated.

Ad ConsistencyVerifying consistency between the ad text and LP content is crucial. If a feature or price mentioned in the ad text is not referenced in the corresponding LP, it may violate the Law for the Prevention of Unjustified Extra or Unexpected Benefit and Misleading Representation, resulting in damages to the advertiser. However, these inconsistencies are difficult to detect as some factual expressions do not appear in LPs. For example, the term "_official_" is often used in ad text but rarely appears in LP content. To assess this, we defined the Ad Consistency task, which predicts the consistency between LP content and ad text with binary labels: consistent/inconsistent. Examples of Ad Consistency are shown in Table 1. The first line is labeled inconsistent because the LP refers to a "_single-family home_," while the ad text mentions a "_condominium_."

Ad Performance EstimationThe most straightforward way to measure ad quality is to publish ads online and let end customers evaluate them. However, delivering all ads without alterations is impractical, as low-quality ads can negatively impact advertisers. Therefore, prior studies have investigated offline methods to measure ad text quality by simulating customer behavior, such as click-through rate (CTR), based on past delivery history. These methods are currently standard practices in many organizations. Inspired by such works Gharibshah et al. (2020); Niu and Hou (2020); Yang and Zhai (2022), we

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Ad text** & **LP text** & **Ad Accept. / Consist.** \\ \hline Condominium Sales / Free Assessment Now & Let XX assess your single-family home! & acceptable / inconsistent \\ \hline Engineer’s career / Engineer’s Job & Find jobs at website XX! & unacceptable / consistent \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of Ad Acceptability and Ad Consistency tasks. Note that LP texts are only used in the Ad Consistency task.

 

adopted the Ad Performance Estimation task to estimate a quality score in the range of [0, 100] based on ad texts, keywords, and industry, as shown in Table 2. The score simulates customer behavior based on past delivery history and is nonlinearly transformed to maintain the original label distribution for contractual reasons.

A\({}^{3}\)RecognitionOne of the most crucial factors in advertising is the _aspect of advertising appeals_ (A\({}^{3}\)). At its core, advertising aims to connect advertisers with customers, and A\({}^{3}\) serves as a bridge between them. For example, an ad emphasizing _low cost_ may resonate with price-conscious customers, while one focusing on _high performance_ may not. Thus, recognizing appealing expressions in advertising and using appropriate A\({}^{3}\) can enhance downstream tasks, such as CTR prediction Murakami et al. (2022). In the A\({}^{3}\)Recognitiontask, we follow a previous study Murakami et al. (2022), which predicted all relevant A\({}^{3}\) labels in a given ad text. Figure 3 shows an example of ad texts and the corresponding A\({}^{3}\)s. All labels and distributions can be found in Table 11.

Ad SimilarityRepeatedly showing the same ads to customers leads to _ad fatigue_Abrams and Vee (2007) and ad performance declines. Therefore, it is essential to avoid displaying the same ads for extended periods and regularly replace them with different ones. However, the transition from old to new ads must be carefully managed to maintain the product and its appeal. Specifically, while the wording and representations are being updated, there is a risk of disengagement of customers who were attracted to previous ads. Thus, when measuring similarity, we should particularly focus on this situation, which enables us to determine whether to replace the ad based on a quantified score.

Building on this motivation, we defined the AdSimilarity task, which predicts the similarity score for an ad text pair on a scale of [1, 5]. The lower the values, the less similar the pair, and vice versa. Examples are shown in Table 3. The first example pair illustrates high similarity, where both the product of "_Suppon Black Vinegar_" and the A\({}^{3}\) of _luxury_ are identical. Conversely, the second example pair differ in A\({}^{3}\) with _budget_ and _discount_, resulting in relatively low similarity.

### Dataset Construction

Data CollectionFor the Ad Acceptability and Ad Consistency tasks, data were collected from the ad creation phase of the actual AdOps workflow, including both human creators and NLG model outputs. For the AdPerformance Estimationand Ad Similarity tasks, we used Japanese sponsored search ads delivered between 2021 and 2022. For the A\({}^{3}\)Recognitiontask, we used data from Murakami et al. (2022).

Data Pre-processingIn the Ad PerformanceEstimation task, we negotiated with our clients to use a subset of the data approved for public release, as the data include ad performance metrics 

that are sensitive to advertisers. Furthermore, we applied a nonlinear transformation to the raw CTR, scaling the values to the range of \([0,100]\) to preserve their distribution. Additionally, we masked proper nouns (e.g., product and company names) to prevent identification of advertisers and avoid any potential negative repercussions upon data release. In the Ad Similarity task, creating sentence pairs through random sampling proved inefficient because most pairs are not similar. Therefore, we utilized the account structure configured during ad delivery, which includes _client_, _account_, _campaign_, _ad group_, and _keyword_ information, as depicted in Figure 4. We created pseudo-similar pairs by sampling texts from the same _ad group_, as preliminary results suggested. To balance the label distribution, we created pseudo-dissimilar pairs by ensuring that the two texts belonged to different _clients_. Consequently, we sampled pseudo-similar and pseudo-dissimilar pairs at a ratio of 9:1.

Annotation WorkflowAll annotators were native Japanese speakers and experts with at least two years of professional experience in AdOps. The annotation workflow for all tasks, except Ad Performance Estimation and A3Recognition, followed these steps: (1) We first removed duplicate entries and filtered out data in languages other than Japanese. (2) We iteratively revised the annotation guidelines until we achieved a satisfactory level of agreement through pilot annotations on small sampled datasets. (3) Following the guidelines, we conducted a pilot annotation by asking three annotators to annotate the same sampled data used in Step 2. Thereafter, we compared the annotators' results with our own and resolved any inconsistencies to further refine the guidelines. This cycle was repeated at least twice. (4) After completing Steps 1 to 3, we conducted the main annotation on the full test set using the finalized guidelines. The complete guidelines are provided in Appendix B.

Data SplittingDespite efforts at deduplication, similar ad expressions can still be easily found, potentially leading to data leakage with a simple random split. Furthermore, assuming the data are used in industry, it is crucial to generalize effectively without overfitting to specific ad expressions. Therefore, for Ad Acceptability and Ad Consistency, we split the data, considering the ad hierarchical structure in Figure 4, ensuring that _clients_ do not overlap across training, development, and testing. For Ad Performance Estimation, we used the delivery structure, with the non-overlapping layer as the campaign. For A3Recognition, we used the same split from Murakami et al. (2022). For the Ad Similarity, we randomly split the data to maintain label distribution consistency.

Figure 4: Account structure in ad delivery is hierarchical. A client represents a single company, and the account typically encompasses the commercial products offered by that client. Campaigns are created to promote these commercial products, while ad groups are used to organize keywords and ad texts. At higher levels of the hierarchy, there are more ads and greater variance. Conversely, at lower levels, there are fewer ads, which tend to be similar.

 

Table 5 presents the statistics of our dataset. The full tables of label distribution for each task are provided in Appendix Table 11.

## 4 Experimental Settings

Table 4 provides a brief overview of each task and the corresponding metrics used to measure task performance. Our tasks are categorized into three setups: binary classification, multi-label classification, and regression. For binary classification, we use accuracy and F1-score to evaluate the binary labels. In multi-label classification, we follow Murakami et al. (2022) and use the F1-score with both macro and micro settings. For regression, we use Pearson and Spearman correlation coefficients.

We employed two types of evaluators: the PLMs, including both fine-tuned settings with encoder models and zero-/few-shot settings for LLMs, as well as human evaluators. Note that the detailed model description, hyperparameters, prompts, and additional information are provided in Appendix C.

Fine-tuning Setting with Encoder Models.We utilized publicly available encoder models as baselines, specifically Tohoku BERT, Waseda RoBERTa, and XLM-RoBERTa Conneau et al. (2019), which are commonly employed for Japanese NLP tasks. These models differ in pre-tokenizer, tokenization unit, and pre-training dataset. All the aforementioned models are of LARGE size.

Zero-/Few-shot Setting with LLMs.We employed CALM\({}_{\rm 27b}\) and ELYZA\({}_{\rm 7b}\)as baselines for open LLMs. CALM\({}_{\rm 27b}\) and ELYZA\({}_{\rm 7b}\) are based on the Llama 2 Touvron et al. (2023) architecture but differ in training methods and data; CALM\({}_{\rm 27b}\) was trained from scratch, whereas ELYZA\({}_{\rm 7b}\) was continuously trained from the original Llama 2.3

Footnote 3: To further assess the gap between fine-tuning and zero-/few-shot learning of LLMs, we also fine-tuned an LLM. Due to the additional nature of this experiment, which falls beyond the scope of our study and is constrained by our computational resources, we selected only the best-performing open-source model from the zero-/few-shot settings. The settings and results are provided in Appendix C.

Human Evaluators.We enlisted three human evaluators who are _not_ involved in AdOps to evaluate all tasks except Ad Performance Estimation. We followed the same procedure for instruction as described in SS3.2. Pilot evaluations were carried out twice on 100 randomly sampled instances from the training set for each task before the main run. In the final assessment, a majority vote per instance was conducted for the Ad Acceptability and Ad Consistency tasks, while the average scores of evaluator assessments were reported for other tasks.

## 5 Results and Discussion

Table 6 provides an overview of the results. For the complete result including base size encoder models and GPT-3.5, please refer to Appendix Table 12.

Fine-tuned Setting with Encoder Models.XLM-RoBERTa and Tohoku BERT achieved the highest or competitive scores in two or more tasks. In addition, the LARGE model outperformed the BASE model in most tasks, suggesting that

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Evaluator** & **Ad ACCEPT.** & **Ad CONist.** & **Ad Perf. Est.** & A\$ Recognition & **Ad Similarity** \\  & Accuracy/F1-score & Accuracy/F1-score & Pearson/Spearman & F1-micro/-macro & Pearson/Spearman \\ \hline \multicolumn{8}{c}{_Fine-tuned Encoder Models_} \\ Tohoku BERT & 0.711/0.688 & **0.767**/0.552 & **0.480***0.497* & 0.774/**0.694*** & 0.773/0.807 \\ Waseda BERT & 0.598/0.637 & 0.755/0.474 & 0.445/0.457 & 0.663/0.517 & 0.740/0.800 \\ XLM-RoBERTa & 0.705/0.690 & 0.758/0.519 & 0.453/0.457 & **0.778**/0.677 & **0.878***0.878*** \\ \hline \multicolumn{8}{c}{_Zero-/Few-shot LLMs_} \\ CALM\({}_{\rm 27b}\) & 0.520/0.115 & 0.381/0.472 & 0.006/0.013 & 0.154/0.042 & 0.036/0.036 \\ ELYZA\({}_{\rm 7b}\) & 0.352/0.520 & 0.628/0.771 & 0.003/0.046 & 0.196/0.044 & 0.015/-0.004 \\ GPT-4 & 0.325/0.433 & 0.583/0.612 & 0.028/0.073 & 0.417/0.113 & 0.776/0.811 \\ \hline Human & **0.732***0.790*** & 0.703/**0.807*** & — & 0.564/0.538 & 0.699/0.765 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance of PLMs and human evaluators on the test set. Underlined indicates the best result for each setting, and **bold** indicates the best result across all methods. We conducted a t-test to compare the best result for each metric with the second-best, considering results statistically significant at p ¡ 0.05. Statistically significant results are marked with an asterisk (*). See Appendix Table 12 for the full results.

 

an increased parameter size plays a key role in understanding ad expressions.

Zero-/Few-shot Setting with LLMs.GPT-4 achieved high performance across all three tasks, while \({\rm ELYZA_{7b}}\) performed the best among the LLMs in Ad Consistency. A substantial difference was observed between the open LLMs and OpenAI's LLMs in the Ad Similarity task. \({\rm CALM2_{7b}}\) and \({\rm ELYZA_{7b}}\) scored close to 0, indicating no correlation, whereas OpenAI's models, especially GPT-4, achieved competitive scores of 0.776/0.811. Thus, the open LLMs used in this study struggle to handle semantic similarity or numerical answers, whereas GPT-4 performs considerably better. In the Ad Performance Estimation task, all LLMs produced uncorrelated responses close to 0, indicating that accurately predicting ad performance remains a challenge for open LLMs.

Fine-tuned Encoder Models vs.Zero-/Few-shot LLMs.Overall, the fine-tuned models outperformed the LLMs. The difference was substantial, ranging from 0.2 to 0.6, particularly for the Ad Acceptability, Ad Performance Estimation, and \({\rm a^{3}}\) Recognition tasks. This can be attributed to the characteristics of the tasks; Ad Performance Estimation involves predicting numbers in the range \([0,100]\), and \({\rm A^{3}}\) Recognition requires selecting all suitable labels from more than 20 labels, suggesting that the variety of data features and outputs could not be handled by few-shot alone. Both Ad Acceptability and Ad Consistency are binary classification tasks, yet the performance gap between fine-tuned models and LLMs is more noticeable in the Ad Acceptability task, with a difference as large as 0.2 points, compared to the relatively smaller difference observed in the Ad Consistency task. The task with the smallest difference was Ad Similarity, with a difference of only 0.06 to 0.10.

PLMs vs. Humans.Human evaluators outperform models in Ad Acceptability and Ad Consistency.In both tasks, models tends to have high accuracy and low F1-score, while humans exhibit the opposite trend. The evaluations by PLMs demonstrate high precision but low recall in both tasks, resulting in a low F1-score. According to Table 11, both datasets exhibit an imbalanced label distribution. Models tend to predict the majority label, which likely leads to high accuracy. In contrast, human experts achieve a balanced precision and recall, demonstrating robustness against the imbalanced label distribution and suggesting a higher level of generalization performance compared to the PLMs. Particularly in the Ad Consistency task, where the label distribution is unbalanced, even the best model achieves an F1-score of only 0.55, whereas human performance reaches 0.8. This suggests that humans can make better predictions in both precision and recall on unbalanced data. Meanwhile, fine-tuned models outperform humans on the \({\rm A^{3}}\) Recognition and Ad Similarity tasks. In the \({\rm A^{3}}\) Recognition task, human evaluators struggled with the diversity of output labels, similar to LLMs. However, the difference between the F1-micro and F1-macro scores is relatively small at 0.03 points for humans. In contrast, for the fine-tuned PLMs, the difference ranges from 0.08 to 0.20 points. This indicates that human evaluators have a strong ability to generalize and can maintain higher performance, even when a label appears infrequently. In the Ad Similarity task, GPT-4, in addition to fine-tuned models, outperforms human evaluators. This is the only task where LLM outperforms humans, suggesting that GPT-4 has a high level of alignment with expert humans in terms of semantic similarity and numerical understanding.

Error Analysis for Ad Acceptability and Ad Consistency Tasks.We conducted a detailed analysis of two tasks: Ad Acceptability and Ad Consistency, where the model has not yet outperformed human evaluators. We analyzed three types of errors: human incorrect and model correct (IC-C), human correct and model incorrect (C-IC),

\begin{table}
\begin{tabular}{c c c c|r r} \hline
**Error** & **GT** & **H** & **M** & **Accept.** & **Consistent.** \\ \hline \multirow{2}{*}{**C-ICE**} & T & T & F & 30.1\% & 17.2\% \\  & F & F & T & 0.6\% & 5.5\% \\ \hline \multirow{2}{*}{**IC-C**} & T & F & T & 0.6\% & 1.9\% \\  & F & T & F & 18.0\% & 10.9\% \\ \hline \multirow{2}{*}{**IC-IC**} & T & F & F & 4.9\% & 6.2\% \\  & F & T & T & 3.9\% & 6.6\% \\ \end{tabular}
\end{table}
Table 7: Type-specific error rates for Tohoku BERT (BERT) and XLM-RoBERTa (XLM-R) in the Ad Acceptability and Ad Consistency task. Ground truth (GT), human (H), and model (M) labels are represented as T (acceptable or consistent) and F (unacceptable or inconsistent).

 

and both incorrect (IC-IC), as shown in Table 7. In both tasks, the model more frequently predicted False labels than True labels for both IC-C and C-IC errors. In contrast, humans provided more True labels in both correct and incorrect cases. This suggests that humans exhibit a relatively higher degree of leniency compared to models, which tend to be overly cautious when making decisions in the Ad Acceptability and Ad Consistency tasks.

Additionally, in the Ad Acceptability task, the model can easily identify elements that can be detected heuristically, such as "incorrect use of symbols" or grammatically incorrect or non-fluent expressions. However, it struggles with evaluating expressions that are fluent but semantically unnatural, such as _"Engineer's career / Engineer's Job"_ (redundancy in the same appeal) or _"Condominium Sales / Single-family Home"_ (contradictory appeals). To address these challenges, strategies such as adding training data or providing the LLM with more detailed judgment criteria could enhance accuracy.

Regarding the Ad Consistency task, the model generally performs well when expressions from the LP are included in the ad text. However, it tends to make errors when this is not the case. There are two potential causes for this scenario. One is the use of meta-expressions, such as the word "_official_." In ad text, "_official_" is often used to reassure customers, but it is frequently omitted in LPs because there is no need to emphasize it. To address this issue, accuracy might be improved by further training on similar data or by incorporating a specific mechanism to infer whether the LP is "_official_." The second cause involves referring to non-verbal information, such as images, in the ad text. Although this is beyond the scope of the current study, it might be addressed by leveraging multimodal models.

**Case Study on the Model's Behavior on Ad Similarity Task.** We used integrated gradients [25] to visualize the attention of Tohoku BERT, aiming for a deeper understanding of the model's behavior, which can potentially enhance its performance. Figure 5 shows examples of low and high errors between predictions and the ground truth in the Ad Similarity task. In the first example, the model fails to pay sufficient attention to "_online seminar_," which has a similar meaning to "_webinar_." However, high attention to the date expression "_9/9 (Thursday) at 13:00_" allows the model to predict a score close to the correct answer. However, in the second example, the model did not pay enough attention to "_English conversation coaching_," which has a similar meaning to "_gaining English skills_," resulting in an incorrect prediction of dissimilarity. As we found many such cases, we believe that the model tends to prioritize surface-level information, especially for entities such as dates, times, and numbers. This may prevent the model from correctly identifying cases that are semantically similar, even when the surface-level information differs, and vice versa.

Another instance where we observed shortcomings in the models is in reasoning. For example, the models mistakenly interpret the phrases "_half price_" and "_buy 2 get 1 free_" as having the same meaning, whereas humans easily comprehend the difference. Additionally, pairs of phrases like "_Available 24 hours a day, 7 days a week_" and "_Closed every Wednesday_," as well as "_Property located near the station_" and "_20-minute walk from the station_" often confuse the models in understanding. These types of paraphrasing are commonly used in the ad creation process; however, it is challenging to automatically capture them in a natural dataset. Therefore, a specialized dataset focusing on these phenomena is necessary to enable models to capture them accurately.

Figure 5: Examples of integrated gradient visualization with Tohoku BERT model’s outputs showing the difference in attention for small (top) and large (bottom) gaps between ground truth and predicted labels in the Ad Similarity task. Red indicates a negative influence, while green indicates positive influence on the predictions.

 

Related Work

Evaluating Ad Text QualityMita et al. (2024) proposed an ad text generation benchmark that aims to evaluate NLG models (Step 2 in Figure 2) with surface- and semantic-level overlap metrics, such as BLEU Papineni et al. (2002), ROUGE Lin (2004), and BERTScore Zhang et al. (2020), along with simple heuristics, including keyword insertion rate Mishra et al. (2020) and text length. However, this approach is inadequate for real-world applications because (1) it requires reference texts to assess generated texts, making it difficult to evaluate new ads that have no references, and (2) simple heuristics cannot guarantee the quality of the ad text for delivery, as more specialized criteria are required in actual production, such as appropriate wording, effective appeals, consistency between ad text and product information, and high predicted performance.

Ad text quality is typically represented by metrics of user behavior obtained after ad delivery as illustrated in Step 6 of Figure 2, and CTR is widely used to predict ad performance and make marketing strategies. The CTR prediction task was first studied by Robinson et al. (2007) and has been actively researched by Rosales et al. (2012), McMahan et al. (2013), Chapelle et al. (2014), Yan et al. (2014), and Kumar et al. (2015), alongside traditional machine learning techniques, such as logistic regression-based and factorization machine (FM)-based models Rendle (2010) over the past decades Yang and Zhai (2022). While there are many methods for CTR prediction, few studies have evaluated the quality of ad texts at a more granular level. Coarse-level feedback may be insufficient to improve ad quality, as it is unclear where edits should be made or what changes are necessary. Meanwhile, ADTEC provides more detailed evaluation results, such as acceptability and consistency, making it easier to integrate into actual AdOps workflows.

## 7 Conclusion

We defined five tasks to verify the quality of ad texts and developed the ADTEC, a large, versatile, and comprehensive benchmark of Japanese advertising data constructed for the NLP community, based on real-world AdOps workflows. We conducted evaluations with both PLMs and human evaluators on ADTEC to explore its characteristics, offering insights into practical workflow applications and identifying potential areas for future improvement and research. Our findings suggest that sampling directly from real data generally benefits the model, highlighting the importance of tasks centered on natural language inference and semantic understanding. We hope that the combination of our defined tasks and datasets will advance research in ad text evaluation, bridging the fields of advertising and NLP and paving the way for new discoveries and applications.

### Limitations

Language.The language was limited to Japanese, as the advertising agency is located in Japan. Expressions related to online advertising vary according to language and culture. For instance, in Japan, for example, there is a cultural preference for cluttered design Masuda and Nisbett (2001), which results in a lot of information being scattered around, compared to the clean LPs in the US and Europe. This means that different strategies and models may be required for different languages and countries. Although exploring the adaptability of datasets across languages is beyond the scope of this study, it represents an exciting opportunity for future research, especially because the field of NLP in advertising remains largely unexplored.

Additionally, existing research indicates that machine translation techniques could potentially mitigate these limitations Miyazaki and Shimizu (2016); Masad et al. (2023); Wang and Hershcovich (2023). Moreover, the methodology for designing these tasks and datasets is language-independent, making it applicable to other languages. Tasks such as assessing the quality of ad text Ad Acceptability), checking the consistency of ad text with landing pages (Ad Consistency), and rephrasing ad texts (Ad Similarity) can enhance operational efficiency across different languages. By referencing the motivations and methods outlined in this paper, future researchers and engineers can design similar tasks without starting from scratch when working in different languages.

Finally, it is important to note that the primary contribution of our study is the definition of real-world advertising operations as distinct tasks and the integration of these tasks into a comprehensive benchmark, using the Japanese dataset as a case study 

Image Data.This study focuses solely on textual data, with image data excluded from the scope for the following reasons: (1) search engine ads consist only of text data, (2) LP images often convey content through text, (3) the performance improvement gained is small compared to the effort required to handle image data. Thus, we consider image data as supplementary in this study. While certain types of ads, such as display ads, primarily use images, further exploration of this area would be a promising direction for future work.

Data Bias.Since the clients are limited to those who have contracts with the advertising agency, the dataset may not entirely represent all industries. Consequently, the findings of this study may not be fully generalizable to the broader internet advertising market. Future research could benefit from including a wider range of data sources to address these limitations.

## Ethical Consideration

Our AdTEC dataset comprises documents such as ad texts, LP texts, keywords, and industry types. The dataset includes only a subset of the original data, sampled over a specific period and refined through preprocessing, as detailed in SS3.2 of our paper.

Each instance in the dataset consists of ad text data along with related information, and each instance is labeled, as discussed in the task design section (SS3). While the dataset is self-contained, it does not include any confidential information, as all data intended for internal use were made public through appropriate channels. However, it is important to note that the use of automatically generated ad texts by NLG models may introduce unnatural expressions, which could be considered a source of noise.

The data were collected in 2023 using both manual human curation and software programs. Authors and other full-time employees were involved in the data collection process. The were compensated at rates above the minimum wage and provided with basic social security benefits. Although no formal ethical review was conducted, individuals involved were directly notified and their consent was obtained before data collection..

Preprocessing, cleaning, and labeling were performed on the data, and the raw data were saved for potential future uses. This dataset has already been used to evaluate ad text quality at the author's company and could be utilized for tasks such as language modeling on acceptable ad texts and hallucination detection. The dataset will be distributed to third parties via a URL and will be available under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC-BY-NC-SA 4.0) International License. The authors will support and maintain the dataset, and updates may be made to correct errors or delete instances upon request. Users will be informed of any updates through our website or codebase.

Please refer to the Datasetsheets Gebru et al. (2021), a framework designed to thoroughly cover essential information about the dataset, in Appendix A for comprehensive and detailed information.

## Acknowledgements

We would like to thank Yoshiko Shinozuka and other members of the KiwamiTD4 team at CyberAgent for their valuable comments and assistance with the background research and data collection.

Footnote 4: [https://www.cyberagent.co.jp/news/detail/id=24670](https://www.cyberagent.co.jp/news/detail/id=24670)

## References

* Abrams and Vee (2007) Zoe Abrams and Erik Vee. 2007. Personalized ad delivery when ads fatigue: An approximation algorithm. In _Proceedings of Workshop on Internet and Network Economics_, pages 535-540.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. _Preprint_, arXiv:2005.14165.
* Chapelle et al. (2014) Olivier Chapelle, Eren Manavoglu, and Romer Rosales. 2014. Simple and scalable response prediction for display advertising. _ACM Transactions on Intelligent Systems and Technology_, 5(4):61:1-61:34.
* Conneau et al. (2019) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. _arXiv preprint arXiv:1911.02116_.

 

Cited by: SS2.
* C. Dong, Y. Li, H. Gong, M. Xu Chen, J. Li, Y. Shen, and M. Yang (2021)A survey of natural language generation. ACM Computing Surveys55, pp. 3-8. Cited by: SS4.1.
* T. Gebru, J. Morgenstern, B. Vecchione, J. Wortman Vaughan, H. Wallach, H. Daume Iii, and K. Crawford (2021)Datasheets for datasets. Communications of the ACM64 (12), pp. 86-92. External Links: Document, ISSN 0740-0228, Link Cited by: SS2.
* Z. Gharibshah, X. Zhu, A. Hainline, and M. Conway (2020)Deep learning for user interest and response prediction in online display advertising.. Data Science and Engineering5 (1), pp. 12-26. External Links: Document, ISSN 0021-909X Cited by: SS2.
* K. Golobokov, J. Chai, V. Ye Dong, M. Gu, B. Chi, J. Cao, Y. Yan, and Y. Liu (2022)DeepGen: diverse search and generation and real-time customization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 191-199. Cited by: SS2.
* J. Weston Hughes, K. Chang, and R. Zhang (2019)Generating better search engine text advertisements with deep reinforcement learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2019), pp. 2269-2277. External Links: ISBN 978-1-4503-3871-1, Link, Document Cited by: SS2.
* H. Kamigaito, P. Zhang, H. Takamura, and M. Okumura (2021)An empirical study of generating texts for search engine advertising. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers (NAACL-HLT 2021), pp. 255-262. External Links: Document, Link Cited by: SS2.
* R. Kumar, S. Manjunath Naik, V. D. Naik, S. Shiralli, S. V.G, and M. Husain (2015)Predicting clicks: ctr estimation of advertisements using logistic regression classifier. In 2015 IEEE International Advance Computing Conference (IACC), Vol. , pp. 1134-1138. External Links: Document, ISSN 1558-1221, Link Cited by: SS2.
* Chin-Yew Lin (2004)ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81. External Links: Document, Link Cited by: SS2.
* O. Masad, K. Bar, and A. Cohen (2023)Automatic translation of span-prediction datasets. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Nusa Dua, Bali, CA, USA, pp. 160-173. External Links: Document, Link Cited by: SS2.
* T. Masuda and R. Nisbett (2001)Attending holistically vs. analytically: comparing the context sensitivity of japanese and americans. journal of personality and social psychology81, pp. 922-934. External Links: Document, ISSN 0021-909X, Link Cited by: SS2.
* H. Brendan McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Childkerur, D. Liu, M. Wattenberg, A. Mar Harfahrelsson, T. Boulos, and J. Kubica (2013)Ad click prediction: a view from the trenches. In The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2013), pp. 1222-1230. External Links: Document, ISSN 1558-1221, Link Cited by: SS2.
* Shaunak Mishra, M. Verma, Y. Zhou, K. Thadani, and W. Wang (2020)Learning to create better ads: generation and ranking approaches for ad creative refinement. CIKM '20, pp. 2653-2660. External Links: Document, ISSN 0021-909X, Link Cited by: SS2.
* M. Mita, Soichiro Murakami, A. Kato, and P. Zhang (2024)Striking gold in advertising: standardization and exploration of ad text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, pp. 955-972. External Links: Document, Link Cited by: SS2.
* Takashi Miyazaki and Nobuyuki Shimizu (2016)Crosslingual image caption generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 1780-1790. External Links: Document, Link Cited by: SS2.
* S. Murakami, S. Hoshino, and P. Zhang (2023)Natural language generation for advertising: a survey. CoRRarXiv:2306.12719. External Links: Link, 2306.12719 Cited by: SS2.
* S. Murakami, P. Zhang, S. Hoshino, H. Kamigaito, H. Takamura, and M. Okumura (2022)Aspect-based analysis of advertising appeals for search engine advertising. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track (NAACL-HLT 2022), pp. 69-78. External Links: Document, Link Cited by: SS2.
* T. Niu and Y. Hou (2020)Density matrix based convolutional neural network for click-through rate prediction. In 2020 3rd International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 46-50. External Links: Document, Link Cited by: SS2.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. External Links: Document, Link Cited by: SS2.
* A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A.

 

Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems 32_, pages 8024-8035. Curran Associates, Inc.
* Rendle (2010) Steffen Rendle. 2010. Factorization machines. In _ICDM 2010, The 10th IEEE International Conference on Data Mining, Sydney, Australia, 14-17 December 2010_, pages 995-1000. IEEE Computer Society.
* Robinson et al. (2007) Helen Robinson, Anna Wysocka, and Chris Hand. 2007. Internet advertising effectiveness: the effect of design on click-through rates for banner ads. _International Journal of Advertising_, 26(4):527-541.
* Rosales et al. (2012) Romer Rosales, Haibin Cheng, and Eren Manavoglu. 2012. Post-click conversion modeling and analysis for non-guaranteed delivery display advertising. In _Proceedings of the Fifth International Conference on Web Search and Web Data Mining (WSDM 2012)_, pages 293-302.
* Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In _Proceedings of the 34th International Conference on Machine Learning, (ICML 2017)_, volume 70 of _Proceedings of Machine Learning Research_, pages 3319-3328.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. _Preprint_, arXiv:2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esioub, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molbygo, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angelena Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scaliom. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_, arXiv:2307.09288.
* Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Neural Information Processing Systems_.
* Wang and Hershcovich (2023) Zi Wang and Daniel Hershcovich. 2023. On evaluating multilingual compositional generalization with translated datasets. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1669-1687, Toronto, Canada. Association for Computational Linguistics.
* Warstadt and Bowman (2019) Alex Warstadt and Samuel R. Bowman. 2019. Grammatical analysis of pretrained sentence encoders with acceptability judgments. _arXiv preprint arXiv:1901.03438_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface's transformers: State-of-the-art natural language processing. _Preprint_, arXiv:1910.03771.
* Yan et al. (2014) Ling Yan, Wu-Jun Li, Gui-Rong Xue, and Dingyi Han. 2014. Coupled group lasso for web-scale CTR prediction in display advertising. In _Proceedings of the 31th International Conference on Machine Learning (ICML 2014)_, volume 32 of _JMLR Workshop and Conference Proceedings_, pages 802-810.
* Yang and Zhai (2022) Yanwu Yang and Panyu Zhai. 2022. Click-through rate prediction in online advertising: A literature review. _Information Processing and Management: an International Journal_, 59(2).
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net 

Datasheets

We adhere to the existing dataset documentation framework proposed by Gebru et al. (2021) to provide comprehensive information about our dataset.

### Motivation

**For what purpose was the dataset created?** This dataset is designed to evaluate the quality of ad texts in multiple aspects from the perspective of practical advertising operations.

**Who created the dataset (for example, which team, research group) and on behalf of which entity (for example, company, institution, organization)?** The authors of this paper created the dataset and conducted the research.

**Who funded the creation of the dataset?** N/A

### Composition

**What do the instances that comprise the dataset represent (for example, documents, photos, people, countries)?** The dataset represents documents, e.g., ad texts, LP texts, keywords, and industry types.

**How many instances are there in total (of each type, if appropriate)?** See Table 5 and 4.

**Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?** As described in 3.2, the dataset includes only a subset of the original data, sampled over a specific period and refined through preprocessing.

**What data does each instance consist of?** Our dataset primarily comprises ad text data and related information, including LP texts, keywords, and industry types.

**Is there a label or target associated with each instance?** Yes. See Section 3.

**Is any information missing from individual instances?** N/A.

**Are relationships between individual instances made explicit (for example, users' movie ratings, social network links)?** N/A.

**Are there recommended data splits (for example, training, development/validation, testing)?** Yes. See Section 3.2 and Table 5.

**Are there any errors, sources of noise, or redundancies in the dataset?** The use of automatically generated ad texts by NLG models may introduce unnatural expressions, which can be considered a form of noise.

**Is the dataset self-contained, or does it link to or otherwise rely on external resources (for example, websites, tweets, other datasets)?** The dataset is self-contained for the tasks described in the paper.

**Does the dataset contain data that might be considered confidential (for example, data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?** Data originally intended for internal use but subsequently made public through appropriate channels.

**Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?** No.

**Does the dataset identify any subpopulations (for example, by age, gender)?** No.

**Is it possible to identify individuals (that is, one or more natural persons), either directly or indirectly (that is, in combination with other data) from the dataset?** No.

**Does the dataset contain data that might be considered sensitive in any way (for example, data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)?** No.

### Collection Process

**How was the data associated with each instance acquired? Was the data directly observable (for example, raw text, movie ratings), reported by subjects (for example, survey responses), or indirectly inferred/ derived from other data (for example, part-of-speech tags, model-based guesses for age or language)?** See Section 3.2.

**What mechanisms or procedures were used to collect the data (for example, hardware apparatuses or sensors, manual human curation, 

software programs, software APIs)?See Section 3.2.

If the dataset is a sample from a larger set, what was the sampling strategy (for example, deterministic, probabilistic with specific sampling probabilities)?As described in Section 3.2, we utilized data from a specific period without employing probabilistic sampling. Items that could not be technically included in the dataset were removed.

Who was involved in the data collection process (for example, students, crowdworkers, contractors) and how were they compensated (for example, how much were crowdworkers paid)?Authors and other full-time employees. All workers involved in this research are employed as full-time employees by the company and are compensated at rates above the minimum wage set by local authorities. Additionally, they are provided with basic social security benefits.

Over what timeframe was the data collected?The data collection was conducted in 2023.

Were any ethical review processes conducted (for example, by an institutional review board)?No. Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (for example, websites)?In question directly. Were the individuals in question notified about the data collection?Yes. Did the individuals in question consent to the collection and use of their data?Yes. If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses?N/A. Has an analysis of the potential impact of the dataset and its use on data subjects (for example, a data protection impact analysis) been conducted?N/A.

Preprocessing/cleaning/labeling Was any preprocessing/cleaning/labeling of the data done (for example, discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?Yes. See Section 3.2. Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (for example, to support unanticipated future uses)?Yes. Is the software that was used to preprocess/clean/label the data available?No.

### Uses

Has the dataset been used for any tasks already?Yes. The dataset has been used to evaluate the quality of ad texts at the author's company. Is there a repository that links to any or all papers or systems that use the dataset?Yes. [https://cyberagentailab.github.io/AdTEC](https://cyberagentailab.github.io/AdTEC) What (other) tasks could the dataset be used for?Various tasks using the dataset can be considered, such as language modeling on the acceptable ad texts and hallucination detection with the Ad Consistency dataset. Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?As mentioned in Section 4, the AdPerformance Estimation task masks certain entities to protect information. Consequently, users may need to complete the task with incomplete information, which could affect its usability. Are there tasks for which the dataset should not be used?N/A.

### Distribution

Will the dataset be distributed to third parties outside of the entity (for example, company, institution, organization) on behalf of which the dataset was created?Yes. How will the dataset be distributed (for example, tarball on website, API, GitHub)?[https://cyberagentailab.github.io/AdTEC](https://cyberagentailab.github.io/AdTEC). When will the dataset be distributed?After this paper is accepted, as soon as possible. Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?The dataset is distributed under 

Creative Commons Attribution-NonCommercialShareAlike 4.0 International (CC BY-NC-SA 4.0) License.

**Have any third parties imposed IP-based or other restrictions on the data associated with the instances?** The third party data is not intended for commercial use and is subject to the organization's terms and conditions.

**Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?** No

### Maintenance

Who will be supporting/hosting/maintaining the dataset?The authors will be.

How can the owner/curator/ manager of the dataset be contacted (for example, email address)?By the email address.

Is there an erratum?N/A.

Will the dataset be updated (for example, to correct labeling errors, add new instances, delete instances)?It is possible to update the dataset on the website or codebase to correct errors or delete instances upon request.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (for example, were the individuals in question told that their data would be retained for a fixed period of time and then deleted)?N/A.

Will older versions of the dataset continue to be supported/hosted/ maintained?It depends on the nature of the dataset update. We may inform dataset users through our website or codebase.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?No. We consider that the dataset should be used exclusively for our task. Any further work should be a separate contribution from ours.

## Appendix B Additional Information in Dataset Construction

### Annotation Guidelines

The annotation guidelines for Ad Acceptability, Ad Consistency, and Ad Similarity are shown in Figures 7 and 8.

 

the model implementation. We present the hyperparameters employed during the training of both fine-tuned encoder models and zero-/few-shot settings on LLMs are detailed in Table 10.

### Prompts

The template for the prompt in each task is illustrated in Figure 6. We begin by outlining the purpose and overview of the task, followed by an explanation of the criteria and standards for evaluation. Next, we provide instructions on the response format, and in some cases, additional few-shot examples are included. Tables 13 and 14 show the prompts actually used in the tasks. Terms in curly brackets are variables to be filled in. All variables, except {few_shot_examples}, correspond to the fields described in Section 3. Note that all prompts are translated from Japanese into English for presentation brevity.

### Evaluators

Fine-tuning Setting with Encoder Models.In the fine-tuning settings, models receive ad text and other values \(\mathbf{x}=(x_{i})^{|\mathbf{x}|}_{i=1}\) as input and predict labels \(\mathbf{y}\), where \(x_{i}\) represents the \(i\)-th token of the input sequence. The hyperparameters used are shown in Appendix Table 10. Ad Acceptability and Ad Consistency involve predicting binary label \(\mathbf{y}\in\{0,1\}\) using an MLP from the encoded vector representation \(h^{[\mathtt{CLS}]}\), where \([\mathtt{CLS}]\) is the special token for classification. The inputs of the Ad Acceptability task only use the ad text \(\mathbf{x}^{\mathrm{ad}}\), while Ad Consistency uses \(\mathbf{x}^{\mathrm{ad}}\oplus\mathbf{x}^{\mathrm{LP}}\) as input, where \(\mathbf{x}^{\mathrm{LP}}\) and \(\oplus\) represent the LP text and concatenation by a special token \([\mathtt{SEP}]\), respectively. A\({}^{3}\) Recognition involves predicting all possible indels from a given ad text, \(\mathbf{x}^{\mathrm{ad}}=(x_{i}^{\mathrm{ad}})^{|\mathbf{x}^{\mathrm{ad}}|}_{i=1}\). We adopted the doc-based architecture described in Murakami et al. (2022), where the encoder model was used to obtain the vector representation \(h^{[\mathtt{CLS}]}\). This vector was then fed into an MLP layer, producing a label probability distribution \(\mathbf{m}=\mathtt{Signoid}(\text{MLP}(h^{[\mathtt{CLS}]})\), where \(\mathbf{m}=(m_{k})_{k=1}^{K}\) and \(K\) is 21. This number corresponds to the number of A\({}^{3}\) labels defined by Murakami et al. (2022), as shown in Appendix Table 11. Ad Similarity and Ad Performance Estimation are regression tasks that involve predicting a value of range \(\mathbf{y}\in[1,5]\) and \(\mathbf{y}\in[0,100]\), respectively. Similar to the text classification task, we concatenated all inputs for each task with the special token \([\mathtt{SEP}]\), encoded the input into vector representation \(h^{[\mathtt{CLS}]}\), and then fed the vector into the MLP layer to predict \(\mathbf{y}\).

Zero-/Few-shot Setting with LLMs.Before conducting the experiment, we sampled 100 instances from the development set to adjust the parameters and prompts. The prompts we used are shown in Figure 6. Additionally, we determined the number of shots that performed best in the development set: 3-shot for Ad Similarity and

\begin{table}
\begin{tabular}{l r} \hline \hline
**Parameter** & **Value** \\ \hline \multicolumn{3}{c}{_Fine-tuned Encoder Models_} \\ Learning Rate & {2e-5, 5.5e-5, 2e-6} \\ Seed & 0 \\ Epoch & 30 \\ Early Stopping Patience & 3 \\ Optimizer & Adam \\ Max Sequence Length & 128 \\ \hline \multicolumn{3}{c}{_Zero-/Few-shot LLMs_} \\ Attempts Per an Instance & 5 \\ Temperature & 0.8 \\ Max New Tokens & 64 \\ \hline \multicolumn{3}{c}{_Fine-tuned LLMs_} \\ LoRA alpha & 16 \\ LoRA dropout & 0.1 \\ Bottleneck r & 16 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Hyperparameters used to the fine-tuned encoder and LLM evaluators. Numbers in curly brackets represent the range of possible values.

\begin{table}
\begin{tabular}{l c l c c c} \hline \hline
**Model** & **PreTokenizer (Dictionary)** & **Tokenization Unit** & **Dataset** & **\#Vocab** & **\#Param** \\ \hline Tohoku BERTBASE & MeCab (PADic+NEologd) & BPE & Wikipedia (Ja) & 32K & 111M \\ Tohoku BERLARGE & McCb (PADic+NEologd) & BPE & Wikipedia (Ja) & 32K & 337M \\ Waseda RoBERTBABASE & Juman++ & Unigram LM & Wikipedia (Ja) + CC (Ja) & 32K & 110M \\ Waseda RoBERTALRAGE & Juman++ & Unigram LM & Wikipedia (Ja) + CC (Ja) & 32K & 336M \\ XLM-RoBERTBAASE & — & Unigram LM & Multilingual CC & 250K & 278M \\ XLM-RoBERTBLARGE & — & Unigram LM & Multilingual CC & 250K & 559M \\ \hline \hline \end{tabular}
\end{table}
Table 9: List of trained language models used in the experiment, where CC and Ja represent CommonCrawl and Japanese data, respectively.

 

Ad Performance Estimation task, 2-shot for A\({}^{3}\) Recognition, and zero-shot for the others. We calculated the final score by averaging five runs with different few-shot examples for a single instance.

Fine-tuning Setting with LLM.We fine-tuned ELYZA-7b, one of the open-sourced LLMs used in our study. We utilize QLoRA tuning, which applies the query and value of the attention module, training it for one epoch with the hyperparameters provided in Table 10.

### Additional Results and Discussion on Fine-Tuning LLM Settings

The results of this fine-tuning are detailed in Table 12. By fine-tuning ELYZA\({}_{7\text{b}}\), we observed performance improvements in most tasks as expected. Although it still falls short of fine-tuned encoder models, it outperformed GPT-4 in Ad Acceptability and Ad Consistency tasks. While it did not match GPT-4's performance in A\({}^{3}\)Recognition and Ad Similarity tasks, it achieved performance levels close to human capabilities in Ad Similarity.

The extent of improvement varied across tasks. Notably, in Ad Consistency, the accuracy increased by only 0.064, whereas in Ad Similarity, the Pearson and Spearman correlations improved dramatically to 0.659 and 0.736, respectively. The Ad Acceptability and A\({}^{3}\)Recognition tasks saw moderate improvements. This suggests that pre-training is particularly effective for tasks similar to Ad Consistency, likely because the pre-training data, derived from web crawls, includes general knowledge applicable to Ad Consistency. Conversely, the low scores in zero-/few-shot experiments for Ad Similarity indicate that applying general knowledge from pre-training is challenging. However, fine-tuning appears to effectively map the pre-trained knowledge to the specific task, significantly enhancing performance. For Ad Acceptability and A\({}^{3}\)Recognition tasks, although additional training led to some improvement, the performance remained relatively low, suggesting that the provided training data was insufficient to fully grasp the knowledge required for these tasks.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Evaluator** & **Ad Accept.** & **Ad Consistent.** & **Ad Perform. Est.** & **A\({}^{3}\)Recognition** & **Ad Similarity** \\  & Accuracy/Fl-score & Accuracy/Fl-score & Pearson/Spearman & F1-micro/macro & Pearson/Spearman \\ \hline \multicolumn{8}{c}{_Eine-tuned Encoder Models_} \\ Tohoku BERT\({}_{\mathrm{BASE}}\) & 0.685/0.691 & 0.757/0.504 & 0.437/0.454 & 0.753/0.629 & 0.769/0.803 \\ Tohoku BERT\({}_{\mathrm{LARGE}}\) & 0.711/0.688 & **0.767**/0.552 & **0.480**/**0.497** & 0.774/**0.694** & 0.773/0.807 \\ Waseda BER\({}_{\mathrm{BASE}}\) & 0.615/0.639 & 0.725/0.388 & 0.444/0.454 & 0.641/0.442 & 0.749/0.797 \\ Waseda BER\({}_{\mathrm{LARGE}}\) & 0.598/0.637 & 0.755/0.474 & 0.445/0.457 & 0.663/0.517 & 0.740/0.800 \\ XLM-RoBERT\({}_{\mathrm{BASE}}\) & 0.694/0.677 & 0.743/0.465 & 0.425/0.439 & 0.730/0.542 & 0.846/0.870 \\ XLM-RoBERT\({}_{\mathrm{ALARG}}\) & 0.705/0.690 & 0.758/0.519 & 0.453/0.457 & **0.778**/0.677 & **0.878**/**0.878** \\ \hline \multicolumn{8}{c}{_Zero-/Few-shot LLMs_} \\ CALM2\({}_{7\text{b}}\) & 0.520/0.115 & 0.381/0.472 & 0.006/0.013 & 0.154/0.042 & 0.036/0.036 \\ ELYZA\({}_{7\text{b}}\) & 0.352/0.520 & 0.628/0.771 & 0.003/0.046 & 0.196/0.044 & 0.015/-0.004 \\ GPT-3.5 & 0.369/0.489 & 0.528/0.570 & -0.013/-0.022 & 0.255/0.064 & 0.389/0.385 \\ GPT-4 & 0.325/0.433 & 0.583/0.612 & 0.028/0.073 & 0.417/0.113 & 0.776/0.811 \\ \hline \multicolumn{8}{c}{_Free-tuned LLMs_} \\ ELYA\({}_{7\text{b}}\) & 0.638/0.638 & 0.692/0.694 & 0.240/0.235 & 0.379/0.280 & 0.684/0.740 \\ \hline Human & **0.732**/**0.790** & 0.703**/**0.807** & — & 0.564/0.538 & 0.699/0.765 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Complete version of Table 6 which showing the performance of PLMs and human evaluators on the test set. Bold indicates the best result across all methods.

 



\begin{table}
\begin{tabular}{l r r r r} \hline \hline
**Label** & **Train** & **Dev** & **Test** & **Total** \\ \hline \multicolumn{5}{c}{Ad Acceptability} \\ Acceptable & 15,099 & – & 850 & 15,949 \\ Not acceptable & 20,278 & – & 1,150 & 21,428 \\ Total & 35,377 & – & 2,000 & 37,377 \\ \hline \multicolumn{5}{c}{Ad Consistency} \\ Consistent & 8,708 & – & 620 & 9,328 \\ Not consistent & 20,048 & – & 1,380 & 21,428 \\ Total & 28,756 & – & 2,000 & 30,756 \\ \hline \multicolumn{5}{c}{A\({}^{3}\) Recognition} \\ Special deals & 230 & 63 & 42 & 335 \\ Discount price & 80 & 22 & 16 & 118 \\ Reward points & 50 & 17 & 14 & 81 \\ Free & 283 & 66 & 62 & 411 \\ Special gift & 83 & 20 & 18 & 121 \\ Features & 746 & 178 & 190 & 1,114 \\ Quality & 35 & 17 & 11 & 63 \\ Problem solving & 10 & 4 & 3 & 17 \\ Speed & 95 & 23 & 20 & 138 \\ User-friendliness & 213 & 65 & 32 & 310 \\ Transportation & 53 & 17 & 10 & 80 \\ Limited offers & 36 & 10 & 6 & 52 \\ Limited time & 43 & 12 & 5 & 60 \\ Limited target & 81 & 10 & 18 & 109 \\ First-time limited & 16 & 5 & 4 & 25 \\ Performance & 47 & 13 & 9 & 69 \\ Largest/No.1 & 108 & 11 & 20 & 139 \\ Product lineup & 167 & 38 & 42 & 247 \\ Trend & 69 & 15 & 13 & 97 \\ Others & 117 & 33 & 23 & 173 \\ Story & 73 & 16 & 9 & 98 \\ Total & 2,635 & 655 & 567 & 3,857 \\ \hline \multicolumn{5}{c}{A\({}^{3}\) Recognition (\#Labels per document)} \\
0 & 337 & 94 & 84 & 515 \\
1 & 769 & 198 & 165 & 1,132 \\
2 & 485 & 98 & 100 & 683 \\
3 & 182 & 44 & 47 & 273 \\
4 & 69 & 26 & 10 & 105 \\
5 & 10 & 5 & 3 & 18 \\
6 & 4 & 0 & 1 & 5 \\ Total & 1,856 & 465 & 410 & 2,731 \\ \hline \multicolumn{5}{c}{Ad Similarity} \\ \(1\leq x<2\) & 527 & 66 & 67 & 660 \\ \(2\leq x<3\) & 845 & 105 & 108 & 1,058 \\ \(3\leq x<4\) & 2,739 & 343 & 344 & 3,426 \\ \(4\leq x<5\) & 790 & 99 & 100 & 989 \\ \(5\leq x\) & 79 & 10 & 10 & 99 \\ Total & 4,980 & 623 & 629 & 6,232 \\ \hline \end{tabular}
\end{table}
Table 11: Label distribution of Ad Acceptability, Ad Consistency, A\({}^{3}\) Recognition and Ad Similaritytask. Note that the number of labels in A\({}^{3}\) Recognition does not necessarily equal the number of sentences, because a single document can have multiple labels.





## Ad Acceptability

Based on the following settings, determine whether the input sentence is acceptable as an advertisement expression.

Criteria - Output _acceptable_ if the sentence is coherent, fluent, and easy to read. - Output _unacceptable_ if the sentence is unnatural due to unnecessary and excessive repetition, inappropriate use of symbols, etc.

Formats - Answer with either _acceptable or unacceptable_. - Do not output anything other than your answer.

{few_shot_examples} Input sentence: (ad_text) Output:

Ad Consistency Based on the following settings, determine whether the given input sentence is consistent with the LP text.

Criteria - Output _inconsistent_ if the input sentence contains expressions not included in the LP text. - Output _inconsistent_ if the input sentence outputs different numbers or names from those mentioned in the LP text.

Formats - Answer with either _consistent_ or _inconsistent_. - Do not output anything other than your answer.

{few_shot_examples} Input sentence: (ad_text) LP text: [lp_text] Output:

Ad Performance Estimation Based on the following settings, estimate the performance of the given advertisement information.

Criteria - The good quality of the ad text (whether it is attractive, whether it is effective in the industry) - The relevance of keywords and ad text - The relevance of LP text and ad text

# Formats - Do not output anything other than your answer. - Answer with a number between 0 and 100.

{few_shot_examples} Industry: (industry_type) Keyword: (keyword) Title: (title_1) (title_2) (title_3) Description: (description_1) (description_2) Output: AD Similarity Based on the following settings, rate the advertising similarity of the given two input sentences.

Criteria - Judge based on the similarity in product category/product name and the similarity in appeal axis/persuasive expressions. - Closer to 5 if similar, closer to 1 if not similar.

Formats - Do not output anything other than your answer. - Answer with a decimal number between 1 and 5.

{few_shot_examples} Input sentence 1: (ad_text_1) Input sentence 2: (ad_text_2) Output:

\begin{table}
\begin{tabular}{l l} \hline \hline  & \\ \# Criteria & \\ - Output _acceptable_ if the sentence is coherent, fluent, and easy to read. - Output _ancceptable_ if the sentence is unnatural due to unnecessary and excessive repetition, - inappropriate use of symbols, etc. \\
* Answer with either _acceptable_ or _uncacceptable_.
- Do not output anything other than your answer. \\
* Output _inconsistent_ if the input sentence contains expressions not included in the LP text.
- Output _inconsistent_ if the input sentence outputs different numbers or names from those mentioned in the LP text. \\
* Answer with either _consistent_ or _inconsistent_.
- Do not output anything other than your answer. \\
* The good quality of the ad text (whether it is attractive, whether it is effective in the industry)
- The relevance of keywords and ad text
- The relevance of LP text and ad text \\
* Do not output anything other than your answer.
- Answer with a number between 0 and 100. \\
* The good quality of the ad text (whether it is attractive, whether it is effective in the industry)
- The relevance of keywords and ad text
- The relevance of LP text and ad text \\
* Do not output anything other than your answer.
- Answer with a number between 0 and 100. \\
* Judge based on the similarity in product category/product name and the similarity in appeal axis/persuasive expressions.
- Closer to 5 if similar, closer to 1 if not similar. \\
* Do not output anything other than your answer.
- Answer with a decimal number between 1 and 5. \\
* Judge based on the similarity in product category/product name and the similarity in appeal axis/persuasive expressions.
- Closer to 5 if similar, closer to 1 if not similar. \\
* Do not output anything other than your answer.
- Answer with a decimal number between 1 and 5. \\
* Dayed title = (id_text_1) (id_text_2) (id_text_3) (id_text_4) (id_text_5) (id_text_6) (id_text_7) (id_text_8) (id_text_9) (id_text_10) (id_text_11) (id_text_12) (id_text_13) (id_text_14) (id_text_15) (id_text_16) (id_text_17) (id_text_18) (id_text_19) (id_text_19) (id_text_12) (id_text_13) (id_text_14) (id_text_15) (id_text_16) (id_text_17) (id_text_18) (




\begin{table} \end{tabular}

* 

\end{table}
Table 14: Prompt used in A\({}^{3}\) Recognition task.