Title: Striking Gold in Advertising: Standardization and Exploration of Ad Text Generation
Benchmark: CAMERA (CyberAgent Multimodal Evaluation for Ad Text GeneRAtion)
Summary: CAMERA is a multimodal benchmark for automatic ad text generation (ATG)  in Japanese. The paper presents the first standardization and formalization of the ATG task, and the first ATG benchmark. The dataset was manually annotated, and the benchmark contains automatic and human evaluations. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: automatic ad text generation
Phenomenon defined by authors: Yes
Phenomenon definition: We standardize the ATG (automatic ad text generation) task as follows: Let x be a source document that describes advertised products or services, a a user signal reflecting the user’s latent needs or interests, and y an ad text. ATG aims is to model p(y|a, x). The specific data to be selected for each x, a, and y will be left to future dataset designers
and providers. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: The paper describes speed, trend, and user-friendliness, faithfulness, fluency, and attractiveness as aspects of a good ad text. Faithfulness, fluency, and attractiveness are used in human evaluation, and those sub-elements are reported. 
Task definition: Models are optionally pre-trained on the train split of CAMERA, and then generate an ad text given a landing page OCR text, the landing page layout information, and the landing page bbox image features. The ad text is manually and automatically evaluated. 
Task item definition: A single item would have the landing page (LP) description, the user query, the landing page layout information, the landing page bbox image features, and entity type (time expression, named entity, terms, etc). 
Task source: Real task examples (e.g. GitHub issues)
Task source detail: Task dataset in Japanese
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 872
Dataset size notes: Train/Dev/Test 12395/3098/872
Dataset metadata available: Yes
Dataset metadata detail: Landing page description, user query, landing page layout information, landing page bbox image features, entity type, industry type. 
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM post-processing (extracting answers, reformatting for automated scoring), Distribution (perplexity, calibration, correlation)
Metric statistics: BLEU-4, Rouge-1, BERTScore, Keyword Insertion Rates (KWD), Sentence Length Regulation Compliance Rates (REG), Pearson and Spearman Correlation for Human Evaluation
Metric subscores: Yes
Metric subscores detail: Faithfulness, fluency, and attractiveness have sub-scores in human evaluation
Metric access: Outputs alone
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors define two requirements for ad text: (1) the information provided by the ad text is consistent with the content of the source document; and (2) the information is carefully curated and filtered based on the users’ potential needs. Thus, for a benchmark for ATG, the authors outline two design policies: the benchmark should (1) utilize multimodal information and (2) evaluate by industry domain. The authors tailor CAMERA to fit both design policies and measure both ad text requirements. 
Taxonomy: Business / 
New taxonomy: Specific Application / Profesisonal Domains