## 3 Methods
### Active Learning with Reinforcement Learning for Model Extraction Attack  
Our goal is to choose an optimal subset of images that encapsulates the full range of diversity and complexity found in the larger dataset, enabling the effective training of the surrogate model. This can be considered as an active learning problem. We set the active learning (AL) challenge using a [PERSON] decision process model, drawing inspiration from related studies such as ([PERSON] et al., 2017 [54]; [PERSON] et al., 2020 [55]; [PERSON] et al., 2022 [56]). The policy network \(\pi\) is represented as an agent for reinforcement learning, in detail a deep Q-network [57]. This method allows the model to develop selection strategies by leveraging the outputs from both the victim and surrogate models, guided by data-driven insights. Our approach differs from existing methods in the following ways: the problem we address, our definitions of states, actions, and rewards, and the specific reinforcement learning algorithm we utilize to determine the best policy. We aim to enhance the performance of surrogate models, \(S\), parameterized by \(\theta\), using a smaller number of images from a large dataset. This process is iteratively conducted until a specified threshold is met. At each iteration, \(t\), a policy network \(\pi\), parameterized by \(\phi\), selects a single image from a randomly sampled subset \(U\) for training the surrogate model. The selected images are also processed by the victim model to acquire its outputs. To assess performance, we employ a straightforward metric: determining the percentage of images from the total dataset that are correctly classified. In our methodology, we begin by training the victim model using the entire training dataset, establishing it as the target. We then divide the training data into three distinct segments. A specific portion, designated as \(D_{t}\), is used for pre-training the surrogate models. To mitigate overestimation issues during the policy network's training phase, we initialize ten different surrogate models, all sharing the same architecture. For training the policy network \(\pi\), another subset, \(D_{r}\) is employed to facilitate the active learning process over multiple episodes. This process focuses on refining an acquisition function designed to enhance the surrogate model's performance using a carefully selected set of \(K\) images. We utilize a separate data partition, \(D_{e}\), to evaluate the policy network, where the surrogate models are trained up to the defined budget \(B\). The total query budget is \(K*B\), but for simplicity, we refer to it as \(B\). Figure 2 describes the main workflow of training for the Policy Network and surrogate models. The MDP framework is defined by a sequence of transitions \((s_{t},a_{t},r_{t+1},s_{t+1})\). Within this framework, for any given state \(s_{t}\in\mathcal{S}\), the agent can choose an action \(a_{t}\in\mathcal{A}\), which involves selecting specific images from the unlabeled set \(U\). Each action is associated with the selection of a particular image. The agent then receives a reward \(r_{t+1}\), calculated based on the feedback from both the surrogate and victim models. It is important to note that the definitions of states and actions are not contingent on the specific architecture of the surrogate models. Our goal is to develop a policy that enhances sample selection to improve the accuracy of the surrogate model. To achieve this, we utilize a DQN architecture, employing transitions from an experience replay memory \(\varepsilon\) to train the policy \(\pi\). We initiate the process by assigning a set of initial weights \(\theta_{0}\) to the ten surrogate models that have been trained using the dataset \(D_{t}\). The following steps are performed during each iteration \(t\):  
1. Randomly sampled subset \(U\) from \(D_{r}\). 2. The state \(s_{t}\) is computed from the feature extractor by given \(U\). 3. Using an \(\varepsilon\)-greedy policy, the policy agent chooses action \(a_{t}\).