# A Novel Framework Leveraging Large Language Models to Enhance Cold-Start Advertising Systems
## 4 Validation Methodology
### Stage 3 and Integrated Framework Validation Through User Study  
To validate the LLM-powered enhancement system (Stage 3) and the integrated framework performance, a dedicated website was developed to facilitate comprehensive user interaction and evaluation. Built upon this web-based platform, a systematic user study methodology was designed to evaluate the model's overall performance alongside LLM utilization for semantic reranking, message refinement, and explanation generation capabilities through direct user feedback. A between-subjects comparative survey design was employed to investigate user responses to online advertisements under two distinct presentation conditions. The study received appropriate ethical clearance, and informed consent was obtained from all participants prior to participation. A total of 46 participants were recruited via convenience sampling through social media platforms (Facebook, LinkedIn). Volunteers were invited to participate through anonymous recruitment messages that did not collect or store any personal identifying information. All data collection maintained strict anonymity protocols. No personal identifying information (names, email addresses, IP addresses, or social media profiles) was collected or stored. Participants were assigned random alphanumeric codes for data-tracking purposes. Participants (\(n=46\)) were randomly assigned to one of two experimental conditions using block randomization to ensure equal group sizes (Condition 1: \(n=23\); Condition 2: \(n=23\)). A Randomized Control Trial (RCT) design was implemented to moderate selection bias and support causal interpretation. Eligibility criteria required participants to be active users of the internet or social media platforms and over the age of 18. Comprehensive statistical analyses were conducted to verify equivalent group distribution across key demographic variables:  
Gender Distribution:  
* Condition 1: 54% male (\(n=12\)), 46% female (\(n=11\));
* Condition 2: 58% male (\(n=13\)), 42% female (\(n=10\));
* Chi-square test: \(\chi^{2}\) (\(1\), \(N=46\)) = 0.089, \(p=0.778\). Age Category Distribution:
* 18-24 years: Condition 1 (\(n=5\)), Condition 2 (\(n=4\));
* 25-34 years: Condition 1 (\(n=6\)), Condition 2 (\(n=5\));
* 35-44 years: Condition 1 (\(n=5\)), Condition 2 (\(n=6\));
* 45-54 years: Condition 1 (\(n=4\)), Condition 2 (\(n=5\));
* 55+ years: Condition 1 (\(n=3\)), Condition 2 (\(n=3\));
* Chi-square test: \(\chi^{2}(4\), \(N=46)\) = 0.542, \(p=0.969\). Internal consistency reliability was assessed for all multi-item Likert scale measures using Cronbach's alpha:
* Advertisement Relevance (3 items): \(\alpha=0.89\), 7%;
* Behavioral Intention (3 items): \(\alpha=0.81\), 95%;
* Explanation Effectiveness (3 items): \(\alpha=0.87\), 9%;
* Comparative Relevance (3 items): \(\alpha=0.85\), 8%;
* Message Quality (3 items): \(\alpha=0.88\), 3%. All reliability coefficients exceeded the conventional threshold of 0.70, indicating acceptable to excellent internal consistency for the measurement scales employed in this study. The experiment simulated a realistic online browsing scenario. Each participant was presented with a mock website, with the site type randomly assigned from one of several categories (e.g., food, travel, health). Although each participant was shown a different website type, the structural and visual design of the websites remained identical across all conditions, ensuring experimental control. The website category was clearly displayed to reinforce the simulated context of user interest in a particular domain. Following website exposure, participants were shown five advertisements. The selection and presentation of these advertisements varied by condition:  
* Condition 1: AI-enhanced Recommendation System Advertisements were initially selected from an experimentally curated ad pool using the ensemble prediction model, which estimated the likelihood of ad engagement based on both user-profile data and the assigned website category.