## 5 Experiments
### Experiment Platform and Data Preprocessing  
We implemented our experiments in PowerLeader PR4865P server with an Intel XEON Gold 6132 CPU and 1T memory. The experiments were developed on Reinforcement Learning architecture in the CentOS 7 system. Importing numpy, matplotlib, math, pandas and time modules helps Python programmers to efficiently perform numerical computations, data analysis, visualization, and handle tasks related to time. Different low coupling modules were implemented in the code to ensure the scalability of the experiments. We used four datasets: a random dataset with a normal distribution of reward (Random data-ND), the Alibaba Tianchi dataset (Alibaba) (Available online: [[https://tianchi.aliyun.com/dataset/dataDetail](https://tianchi.aliyun.com/dataset/dataDetail)]([https://tianchi.aliyun.com/dataset/dataDetail](https://tianchi.aliyun.com/dataset/dataDetail)) (accessed on 15 March 2024)), MovieLens (Available online: [[https://grouplens.org/datasets/movielens](https://grouplens.org/datasets/movielens)]([https://grouplens.org/datasets/movielens](https://grouplens.org/datasets/movielens)) (accessed on 15 March 2024)), and the advertising dataset (Ads-CTR-Optimisation) (Available online: [[https://segmentfault.com/a/1190000018871668](https://segmentfault.com/a/1190000018871668)]([https://segmentfault.com/a/1190000018871668](https://segmentfault.com/a/1190000018871668)) (accessed on 15 March 2024)). The datasets were preprocessed for recommendation as follows before the experiments. Random data-ND simulates normal distribution data by using the normal function in numpy and then obtains a value randomly. The input parameters are the mean and variance of the normal distribution. In these experiments, we established nine normal distributions with different means and variances to represent the nine arms and performed the functions with different parameters according to the selected arms. Alibaba randomly samples the advertising display click logs (26 million records) of 1.14 million users within 8 days from the Taobao website to form the original backbone of the sample. This dataset was preprocessed before use. First, we removed unnecessary columns, such as _user_1D_ (desensitized by user ID), and _pid_ (resource bit), and _noclk_ (1 means no click and 0 means click). Then, we processed the remaining data in rows, removed the advertisements that had not been clicked, that is, the rows where _clk_ (0 means no click and 1 means click) is 0, and then removed the _clk_ column. After processing, _adgroup_id_ was classified. Every 10,000 IDs were grouped into one group and integrated into one arm. The number of advertising data in this group is the reward number of this arm. We expanded the current column; one column represents one arm. Meanwhile, we filled the excess blanks with 0 and set the ID to 1. Finally, we used the sample function in pandas to deal with confusion. This completes the dataset with only 0, 1, and 85 arms. The dataset includes 430,000 rows and 85 columns, but each row has only one arm with a reward value, so the data of this dataset are relatively sparse. Movielens is actually a recommendation system and virtual community website. It was founded by the GroupLens project team at the School of Computer Science and Engineering at Minnesota University. It is a non-commercial experimental site for research purposes. The GroupLens research group has produced a MovieLens dataset based on the data provided by the MovieLens website, which contains multiple movie scoring datasets. After decompressing the file, we obtained three files: movies.dat, ratings.dat, and user.dat. We integrated the data into a table with user ID, movie ID, and score. We specified that when the score is greater than 3, the value of the arm is 1; otherwise, it is 0. Ads-CTR-Optimisation collects the click data of 10 advertising spaces within a certain period of time, with a total number of 10,000. There are multiple advertising spaces for clicking at the same time.