## 2 Background and Related Work
### Background
#### 2.1.1 Language Models and Machine Learning Techniques  
TinyLlama [9] is a small large language model (SLLM) that has fewer than 5 billion parameters. It offers a balance between performance and computational efficiency, making it a promising alternative for scenarios where large models like GPT-3 [10] or LLaMA [11] are impractical due to their extensive memory and computational requirements. TinyLlama's design allows it to maintain a high level of performance in natural language processing tasks while requiring significantly less computational power, making it ideal for applications in environments with limited resources.  
Bidirectional Encoder Representations from Transformers (BERT) is a widely adopted pretrained language model in the field of natural language processing. At its core, BERT learns contextual representations of language through pretraining on large textual corpora. This enables the model to capture the meaning of words by considering bidirectional context, leading to impressive performance across various NLP tasks. BERT's pretraining is based on two objectives: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).  
MLM involves masking a random subset of tokens in the input sequence and training the model to predict the masked tokens based on the surrounding context. This allows the model to develop a deep understanding of the bidirectional context and learn to represent words accordingly. On the other hand, NSP is the task of predicting whether two given sentences are actually consecutive or not. This helps the model understand the relationships between sentences and capture discourse-level information. Through pretraining with MLM and NSP, BERT learns rich and generalized representations of language.  
The pretrained BERT model can be fine-tuned for specific tasks. This process involves further training the model using the weights of the pretrained model as a starting point, with a small learning rate and a limited number of epochs. Fine-tuning allows BERT to adapt its language representations to a particular domain or task. The resulting fine-tuned BERT models have shown impressive performance on a wide range of NLP tasks, including text classification, named entity recognition, question answering, and more. The pretraining and fine-tuning capabilities of BERT have brought significant advancements to the field of natural language processing and continue to be actively researched and utilized.  
XGBoost [12] is an optimized gradient boosting library designed to be highly efficient, flexible, and portable. Widely used in machine learning competitions and real-world applications, XGBoost handles large datasets effectively, delivering accurate predictions. It has been successfully applied in various domains, including recommendation systems,due to its robust performance and scalability, making it a preferred choice for complex data-driven tasks.  
DeepFM [13] is a deep learning-based model that combines the strengths of Factorization Machines (FM) and deep neural networks. It is designed to automatically capture both low- and high-order feature interactions, making it particularly effective in recommendation systems where user-item interactions are complex. DeepFM's architecture integrates the benefits of both linear models and deep learning, leading to superior performance in predicting user preferences.