# A Novel Framework Leveraging Large Language Models to Enhance Cold-Start Advertising Systems
## 5 Results and Discussion  
### Transformer-Based Feature Representation and Ensembler Model Evaluation Results  
Following data preprocessing, a systematic evaluation of the transformer-enhanced ensemble approach on the DigiX and Avazu datasets was conducted, focusing on frozen-start advertising scenarios. The experimental design involved two stages as discussed above: optimizing individual state-of-the-art models (DCN-V2, DIFM, FiBiNET, MMOE) and evaluating their transformer-enhanced versions, followed by assessing the weighted ensemble model. To establish robust baselines, we optimized the architectures and hyperparameters of DCN-V2, DIFM, FiBiNET, and MMOE using grid search and iterative testing to maximize AUC and accuracy on both datasets. Common parameters across all models included a batch size of 2086, ReLU activation in hidden layers, sigmoid activation in the output layer, binary cross-entropy (BCE) loss, Xavier weight initialization, and 20 epochs, with learning rates of 0.001 for DigiX and 0.001 for Avazu, using the Adam optimizer (Table 2 below). Model-specific parameters, such as the number of neurons (ranging from 32 to 400), layers (2 to 4), and dropout rates (0.1 to 0.3), were tuned individually to optimize performance, with configurations selected based on validation set performance. Subsequently, each model was modified by replacing traditional shallow embeddings with transformer-based feature representations, incorporating single-head attention and positional encoding to capture semantic relationships in sparse data. To ensure fairness in evaluating these modifications, we applied the same optimized parameters as the baselines, maintaining consistency across experiments. The transformer-enhanced models were evaluated on the same data. Finally, the weighted ensemble model, combining the outputs of the transformer-enhanced DCN-V2, DIFM, FiBiNET, and MMOE, was evaluated to assess its effectiveness against individual models. The ensemble used a softmax-derived weighting scheme (Section 3.2), with weights optimized via grid search to minimize BCE loss on validation sets. The same common parameters were applied to ensure consistency. Starting with the Avazu dataset (Table 3--PI stands for performance improvement over its original counterpart and for ensemble over the best model), the experimental results demonstrate consistent performance improvements across all base models when enhanced with self-attention and positional bias (transformer-based feature extraction). \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **DigiX** & **Avazu** \\ \hline Learning rate & 0.001 & 0.001 \\ Optimizer & Adam & Adam \\ Batch size & 2086 & 2086 \\ Embedding size & 32 & 32 \\ Activation functions & ReLU in hidden layers, Sigmoid in output & ReLU in hidden layers, Sigmoid in output \\ Loss function & BCE & BCE \\ Epochs & 20 with early stopping & 20 with early stopping \\ Weight initialization & Xavier & Xavier \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Common optimal parameters. Notably, AUC ROC scores improved for every model variant, outperforming their original counterparts. For instance, DIFM improved from 0.7179 to 0.7229 (0.69%), and DCN from 0.7192 to 0.7224 (0.45%). While the AUC gains may seem modest (generally under 1%), such improvements are meaningful in large-scale recommendation tasks and especially in frozen-user scenarios. Also, in terms of accuracy, enhanced models again show favorable trends, with MMOE-TR and DCN-TR reaching the highest accuracy values (0.8283 and 0.8273, respectively). The ENSEMBLER model achieved the top AUC ROC (0.7251, 0.3% improvement over the best) and accuracy (0.8286), suggesting that combining models further leverages the benefits. Overall, the results indicate that the TR mechanism contributes positively to both discrimination (AUC) and classification performance (accuracy), supporting its value across different model architectures.