# A Novel Framework Leveraging Large Language Models to Enhance Cold-Start Advertising Systems
## 4 Validation Methodology
### Stage 3 and Integrated Framework Validation Through User Study
The top five ads with the highest predicted click-through probabilities were passed to a large language model (LLM) API for further personalization. The LLM dynamically refined the content of the most relevant ad message and provided a brief explanation based on the user profile and context. It also evaluated whether a reranking of the selected ads was necessary. Participants in this condition were shown the final recommended advertisement (post-refinement), a system-generated explanation message, and the four excluded ads that were not selected as optimal. * Condition 2: Baseline (Random Selection) Participants were shown five randomly selected advertisements from the same ad pool. One was randomly labeled as the recommended ad, and the remaining four were presented as excluded. No personalization or explanation was provided. After viewing the website and advertisements, participants completed an online survey (see Appendix A) designed to assess their perceptions, attitudes, and reactions to the ads. The survey was identical across both experimental conditions, with the exception of one question group specific to Condition 1, which addressed the explanation message. The entire experimental session took approximately 10 min per participant. Participants responded to key components using 5-point Likert scales (1 = lowest, 5 = highest). Each component was measured using a group of three related questions to ensure reliability. The measured components included:  
1. Advertisement Relevance: A set of three questions assessing how relevant participants perceived the ad to be. 2. Behavioral Intention: A three-question group evaluating the likelihood of participants engaging with the ad (e.g., clicking on it). 3. Explanation Effectiveness: * Condition 1: Questions focused on whether the provided explanation helped participants understand why the ad was shown and whether it increased their likelihood of engaging with the ad. 4. Comparative Relevance: Three questions assessing the perceived relevance of the displayed ad compared to potential alternatives that were not shown. 5. Message Quality: A question group evaluating the clarity and communicative effectiveness of the ad's message. Additionally, participants were asked to select one or more factors that influence their clicking decisions from a list including relevance/interest, explanation, transparency, advertisement message, design, and brand. Finally, to also evaluate the linguistic clarity of the LLM-generated explanations and compare them with the user study findings, we employed the Flesch-Kincaid readability test. This is a widely used computational metric that assesses how easily a text can be understood by readers. This test is applied programmatically and provides two scores: the Flesch Reading Ease, which rates readability on a scale from 0 to 100 (with higher values indicating easier comprehension), and the Flesch-Kincaid Grade Level, which estimates the U.S. school grade required to understand the text. We applied this test to all LLM-generated explanations in our study and calculated the average to quantitatively assess their accessibility and alignment with our prompt constraints (e.g., professional tone, concise length).