# A Novel Framework Leveraging Large Language Models to Enhance Cold-Start Advertising Systems
## 5 Results and Discussion
### Integrated Framework Evaluation Results  
A detailed statistical analysis was performed on the questionnaire data. Each construct was measured using a group of three items, and the average of those items was used to compute a composite score per participant. Differences between the two experimental conditions were analyzed using the Mann-Whitney U test, due to non-parametric distribution of responses. * Advertisement Relevance: Participants in Survey 1 rated advertisements as significantly more relevant (M = 4.15) than those in Survey 2 (M = 3.25), U = 435.0, \(p=0.014\). * Behavioral Intention: Survey 1 participants reported significantly greater likelihood of clicking on the ad (M = 4.15) compared to Survey 2 (M = 2.69), U = 542.0, \(p<0.001\). * Explanation Effectiveness: Participants in Survey 1, who were shown an explanation, reported higher scores on explanation effectiveness (M = 4.15) than Survey 2 participants (M = 3.23), U = 492.0, \(p=0.0017\). * Comparative Relevance: Survey 1 participants rated the displayed ad as more relevant compared to unseen alternatives (M = 4.00) than Survey 2 participants did (M = 2.69), U = 472.0, \(p=0.0015\). This difference shows that LLM-enhanced reranking produces ad selections users perceive as substantially more relevant than unrefined alternatives. A limitation in evaluating the complete ensembler + LLM system versus the ensembler is that traditional AUC metrics cannot be directly applied, as the LLM operates as a post-processing reranking layer that selects advertisements based on semantic relevance rather than click probability prediction. Since ground-truth click labels correspond to the original dataset interactions and not to the LLM's semantic reranking decisions, the system's final output represents a qualitatively different recommendation paradigm that requires alternative evaluation methodologies beyond standard CTR prediction metrics. * Message Quality: Survey 1 participants rated the clarity of the ad's message higher (M = 4.23) than Survey 2 participants (M = 3.38), U = 548.0, \(p<0.001\). Also, regarding the factors influencing clicking decisions for both conditions, the following results are reported:  
* Relevance/interest: 92.3%;
* Advertisement message: 84.6%;
* Brand 61.5%;
* Design: 53.8%;
* Explanation: 46.2%;
* Transparency: 38.4%. It is worth mentioning that no statistically significant differences were found between groups (\(p>0.05\)), suggesting that the core perception about the factors that influence ad clicking and emotional response remained consistent across conditions. Last, the LLM-generated explanations achieved a Flesch Reading Ease score of 68.73, indicating they are easily understandable by readers with a middle school education. The corresponding Flesch-Kincaid Grade Level of 7.57 suggests the content is suitable for readers around the 7 th to 8 th grade level. These scores confirm that the explanations maintain a balance between clarity and professionalism, aligning with our prompt constraints. This study demonstrates significant differences in user responses to online advertisements based on the advertisement selection process and presentation format. Across five key dimensions--ad relevance, click likelihood, explanation effectiveness, comparative relevance, and message clarity--participants exposed to hybrid model (Survey 1) consistently reported more favorable evaluations than those in the control condition (Survey 2). These results suggest that both the relevance and the presentation context of an advertisement, including the presence of explanations, play critical roles in shaping user engagement and perceptions. The findings align with established theoretical frameworks in personalization and persuasive communication. Specifically, the results support Personalization Theory [40], which argues that tailoring content to individual user preferences enhances satisfaction and engagement.