## 4 Experiments
### Results  
In Figure 4, we contrast different approaches with growing budgets for executing black-box attacks. The \"Budget\" labeled X-axis shows the number of queries we train the surrogate model. We set specific policies in place and trained surrogate models accordingly. Following this, we conducted evaluations of the surrogate models using a test dataset. Given the significant variability inherent to the reinforcement learning (RL) algorithm, we conducted 10 trials to average the accuracy across various trained policy models. The policy models were trained with \(K=4\) and a budget of 3000. As the DQN model was initially trained using a budget of 3000, the total budget range represented in this figure spans from 3000 to 13,000. We also integrated all DQN extensions to identify the configuration that achieves optimal performance. The findings reveal that this combination of methods outperforms each method when used individually. Specifically, among the standalone applications, PER demonstrated the highest performance. We also investigate how an increase in budget to train policy models affects the performance of our method. Figure 5 shows the results of the increase in the budget for different trained policy performances. Due to limited accessibility regarding the victim model, we are interested in the limited queries for the evaluation of the surrogate model. We are constrained by an evaluation budget of 1000 queries. It is important to note that we also utilize budgets during the training of the policy models, so the total query budget will be 1000 plus additional budget allocations for the different policy model variants. Across all models, the Prioritized Experience Replay (PER) generally yields higher accuracy compared to the other methods and Double DQN also performs well. We evaluated the performance with various sample sizes of selected images, specifically for \(K=1\) and \(K=3\). However, these configurations did not outperform \(K=4\). Therefore, we present our best performance results achieved with a sample size of \(K=4\). For synthetic data generation methods, the experiments aimed to assess the effectiveness of various techniques in producing samples that enable to training of a model with improved performance for model extraction attacks. Specifically, we looked into the Jacobian-based, Linf-projected Gradient Descent (LinfPGD), Fast Gradient Sign Method (FGSM), Entropy, and a data-free model extraction approach. To ensure a robust and fair analysis, the experiment was repeated 10 times, with the results' mean calculated to account for variability. We also added one DQN method to compare with synthetic data generation methods. Figure 6 illustrating these findings was created to visually compare the performances of the different methods. Among these techniques, PER emerged as the superior method, consistently outperforming the others in terms of achieving higher accuracy in the model extraction process. In synthetic data generation methods, LinfPGD outperforms other methods. Conversely, the data-free method demonstrated significantly lower performance. This outcome suggests that the allocated query budget may have been insufficient for generating realistic and useful images for training purposes. The effectiveness of synthetic data generation in model extraction attacks appears to be highly dependent on the ability to produce quality training data, a criterion where the data-free method fell short under the constraints of a limited query budget. Figure 4: Performance of different DQN policies on surrogate models. Here’s the figure showing the performance of the DoubleDQN (DDQN), Noisy Network, Dueling DQN, Soft update, Emphasizing Recent Experiences (ERE), and Prioritized Experience Replay (PER). Each line represents a different DQN model, with the X-axis indicating training surrogate model budgets from 1000 to 10,000 in 1000 intervals, and the Y-axis displaying the model’s accuracy. Figure 5: Performance of various DQN policy variants on surrogate models as the training budget increases.