SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: Active Testing While Learning (ATL)  
Input : Total number of quizzes:  $T$ , Active learning/testing/feedback budget:  $N_{AL}$ ,  $N_{AT}$ ,  $N_{FB}$  Unlabeled pool:  $S_{U}$ , Active learning model at quiz  $t$ :  $f_{\theta_t}(\mathbf{x})$ , AL sampling strategy:  $a: f_{\theta_t}(\mathbf{x}) \times S_U \to \mathbb{R}$ , Learning objective  $\mathcal{L}$ :  $\mathcal{L}: f_{\theta_t}(\mathbf{x}) \times y \to \mathbb{R}$ , Annotation method:  $h: \mathbf{x} \to y$   
Output: Quiz samples  $\mathcal{Q} = \{\mathcal{Q}_1, \mathcal{Q}_2, \dots, \mathcal{Q}_T\}$ , Quiz results:  $R_Q = \{R_{Q_1}, R_{Q_2}, \dots, R_{Q_T}\}$ , Annotated training dataset:  $S_L$   
1  $S_L = \{\}$  // Active Learning Samples  
2  $\mathcal{Q} = \{\}$  // Quiz Selection Criterion  
3 for  $t = 1$  to  $T$  do  
4 for  $i = 1$  to  $N_{AL}$  do  
5  $\begin{array}{c} \mathbf{x}_i = \operatorname*{argmax}_{\mathbf{x}_i} a(f_{\theta_t}, S_U) \\ S_L = S_L \cup \{\mathbf{x}_i, h(\mathbf{x}_i)\} \\ S_U = S_U \setminus \{\mathbf{x}_i\} \end{array}$  // Active Learning Starts  
6  
7  
8 Estimate  $R_\theta^{\mathrm{multi}}$  using eq(5) // Active Testing Starts  
9 Compute test distribution  $q(\cdot)$  using eq(4)  
10  $\mathcal{Q}_t = \{\}$   
11 for  $i = 1$  to  $N_{AT}$  do  
12 Select test sample  $\mathbf{x}_i \sim q(\mathbf{x}_i, R_\theta^{\mathrm{multi}}, S_U)$   
13  $\mathcal{Q}_t = \mathcal{Q}_t \cup \{\mathbf{x}_i\}$   
14  $S_U = S_U \setminus \mathcal{Q}_t$   
15 for  $i = 1$  to  $N_{FB}$  do  
16 Choose  $\mathbf{x}^*$  using eq(9) // Active Feedback Starts  
17  $S_L = S_L \cup \{\mathbf{x}^*\}$   
18  $\mathcal{Q}_t = \mathcal{Q}_t \setminus \{\mathbf{x}^*\}$  // Active Feedback Ends  
19 Compute  $\widehat{R}_{\mathcal{Q}_t}$  using eq(6)  
20  $R_Q = R_Q \cup \{\widehat{R}_{\mathcal{Q}_t}\}$   
21  $\mathcal{Q} = \mathcal{Q} \cup \mathcal{Q}_t$  // Active Testing Ends
Source code. The data and source code for replicating the results are provided in this link: https://github.com/ritmininglab/ATL.git