SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B. RL policy
Figure 8: Greedy and RL policies. A: greedy action policy given by Eq. (31) with  $v_{1}$  and  $v_{2}$  as in (37) with  $P = 1$ , for the linear TSA problem with one-dimensional input. The  $x$ -axis shows values of the input, while  $\rho$  represents the student state expressed as  $\omega^{\mathrm{s}} = w^{*}(1 - \rho) + w^{\mathrm{t}}\rho$ . B: best TD3 policy function found for this problem. Parameters:  $C = 1$ ,  $\eta = 0.02$ ,  $a \in [0,1]$ ,  $w^{\mathrm{t}} = 1$ ,  $w^{*} = -1$ .
Figure 9: Convergence of RL agents. A, top panel: average steady-state running cost of the TD3 agent vs training episodes, for attacks on the linear regression model. Training episodes have length of  $4 \times 10^{4}$  SGD steps, and policy updates are performed every 100 steps. The dotted line shows the average steady-state running cost of the greedy algorithm. The blue dot indicates the best performance achieved by the TD3 agent. Bottom panel: example of actions performed by the greedy attacker (dotted line) and by the best TD3 agent (blue line) on the same data stream. B, C: same quantities as shown in A, for attacks on the sigmoidal perceptron and 2-layer NN. For the latter case, the TD3 agent observed the read-out weight layer only. The best TD3 agents were employed for the comparison shown in Fig. 3. Parameters:  $P = 1$ ,  $C = 1$ ,  $D = 10$ ,  $\eta = 0.02 \times D$ ,  $a \in [-2,3]$ .
B.
Figure 10: Comparison of Adam and SGD dynamics. A: average relative distance vs training step  $\mu$  for a logistic regression model classifying MNIST digits 1 and 7, under label-inversion greedy attacks, for  $C = 1$ . Solid and dashed lines correspond to training via Adam and SGD, respectively. Average performed over 100 data streams. B: average steady-state relative distance as a function of the cost of actions  $C$  and batch size  $P$ , obtained for the experiment described in panel A, and using the Adam optimizer. The orange shaded area shows the SGD-based theoretical estimate (Eq. (13)) for  $P$  in the range [1, 10]. Averages performed over 10 data streams of  $10^{4}$  batches, and over the last 1000 steps in each stream. Parameters:  $D = 10$ ,  $\eta = 0.1$  (SGD),  $\eta = 0.01$  (Adam),  $a \in [0,1]$ .