SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As an arbitrary bijection from  $\mathbf{z}$  to  $\mathbf{G}$  doesn't have any extra information to improve the coding efficiency, Equation (4) provides an upper bound of the minimal Kolmogorov complexity.
On the contrary, as each compositional mapping can be represented by an element in  $S_v^m \rtimes S_m$ , we can encode the mappings more efficiently. Specifically, we need to first use  $m$  sequences with length  $v$ , i.e.,  $\operatorname{Perm}(\{1, 2, \dots, v\})$ , to represent the assignment of "words" for each  $z_i$ . After that, we need one sequence of length  $m$ , i.e.,  $\operatorname{Perm}(\{1, 2, \dots, m\})$  to encode the assignment between  $z_i$  and  $G_j$ . The corresponding Kolmogorov complexity is then bounded as
$$
\mathcal {K} (\operatorname {c o m p}) \leq v \cdot \log_ {2} v + m \cdot \log_ {2} m, \tag {5}
$$
Although this is only an upper bound, by a counting argument most such mappings must have a complexity no less than, say, a constant multiple of that bound.
To compare the Kolmogorov complexity, we can define a ratio as  $\gamma \triangleq \frac{\mathcal{K}(\mathrm{bijection})}{\mathcal{K}(\mathrm{comp})}$ . Obviously, when  $m \leq v$ ,  $\gamma \geq \frac{v^{m-1} \cdot m}{2}$ , which is larger than 1 as long as  $m, v \geq 2$ . When  $m > v$ ,  $\gamma \geq \frac{v^m \log_2 v}{2 \log_2 m}$ , which is also larger than 1 when  $m, v \geq 2$ .
Actually, there might be some mappings that are not purely compositional or holistic. For example, we can have a mapping with  $z_{i \leq 10}$  sharing the reused rules while other  $z_{i > 10}$  doesn't. Then this type of mapping can be represented by an element in  $S_v^{10} \rtimes S_{10} \rtimes S_{v^{m - 10}}$ . As a mapping in this subset shares 10 common rules, its Kolmogorov complexity is between  $\mathcal{K}(\text{bijection})$  and  $\mathcal{K}(\text{comp})$ . Intuitively, for all bijections, smaller  $\mathcal{K}(\cdot)$  means higher compressibility and higher compositionality.
B.2 Regularize the Kolmogorov complexity using iterated learning
From the analysis in Appendix A.4 and B.1, we know that finding mappings with lower Kolmogorov complexity is the key to generalizing well compositionally. From existing works in cognitive science, we know iteratively introducing new agents to learn from old agents can impose the compressibility pressure and hence make the dominant mapping more compact after several generations [42]. Although iterated learning reliably prompts the emergence of compositional mapping in lab experiments, directly applying it to deep learning is not trivial: as we are not sure whether the preference for compositionality still exists for the neural agents. Hence in this subsection, we study a simple overparameterized linear model on a 0/1 classification task to show that iterated learning can indeed introduce a non-trivial regularizing effect. Combining with the fact that mappings with lower Kolmogorov complexity are more likely to capture the ground truth generating mechanism and hence generalize better [71], we can conclude that iterated learning is helpful for compositional generalization problems.
Consider a general supervised learning problem, in which we want to learn a mapping  $f \in \mathcal{F} : \mathcal{X} \to \mathcal{Y}$  that could approximate the underlying relationship between random variables  $X$  and  $Y$ . As we usually have a finite number of training samples and the space of all possible mappings is large, the model could just remember all  $(\mathbf{x}, y)$  pairs in the training set to achieve a perfect training performance. To avoid this trivial solution, we usually expect the optimal  $f^*$  to have specific properties, e.g., smoothness or Lipschitz continuousness, etc. Hence usually, we want to optimize a problem with a corresponding regularization term:
$$
f ^ {*} \triangleq \arg \min  _ {f \in \mathcal {F}} R (f) \quad \text {s . t .} \quad \frac {1}{N} \sum_ {n} \| f (\mathbf {x} _ {n}) - y _ {n} \| _ {2} ^ {2} \leq \epsilon , \tag {6}
$$
where  $R: \mathcal{F} \to \mathbb{R}$  is regularizing  $f$  and  $\epsilon$  is the training loss tolerance. This regularization term is usually the inner product of  $f$  on the functional space, i.e.,  $R(f) = \| f\|_{\mathcal{H}}$ , where  $\mathcal{H}$  is a reproducing kernel Hilbert space (RKHS) determined by some kernel function  $\kappa (\cdot ,\cdot)$ . For example, if we consider  $\forall (x,y)\in [0,1]^2$  and  $\kappa (x,y) = \min (x,y)$ , then  $\| f\|_{\mathcal{H}} = \| f^{\prime}\|_{[0,1]^{2}}$  [64]. In other words, the regularizer will penalize functions with higher first-order derivatives.
In order to make the analysis generalize to other properties of  $f$ , we define a linear differential operator  $L$  as  $[Lf] \triangleq \int_{\mathcal{X}} u(\mathbf{x}, \cdot)f(\mathbf{x}) \, \mathrm{d}\mathbf{x}$ , where  $u(\cdot, \cdot)$  is a kernel function. Then, the regularization term is:
$$
R (f) = \| f \| _ {\mathcal {H}} = \langle f, f \rangle_ {\mathcal {H}} = \langle L f, L f \rangle_ {\mathcal {X} ^ {2}} = \int_ {\mathcal {X}} \int_ {\mathcal {X}} u (\mathbf {x}, \mathbf {x} ^ {\dagger}) f (\mathbf {x}) f (\mathbf {x} ^ {\dagger}) d \mathbf {x} d \mathbf {x} ^ {\dagger} \tag {7}
$$
Substituting this definition back to Equation 6 and then applying the Karush-Kuhn-Tucker (KKT) conditions, the closed-form solution for this optimization problem is (i.e., Proposition 1 in [53]):
$$
f ^ {*} (\mathbf {x}) = g _ {\mathbf {x}} ^ {\top} (c I + G) ^ {- 1} \mathbf {Y}, \tag {8}
$$
where  $c$  is a bounded constant,  $\mathbf{Y} = [y_1|\ldots |y_N]^\top$  is the stacked training labels. The matrix  $G\in \mathbb{R}^{N\times N}$  and its vector  $g_{\mathbf{x}}\in \mathbb{R}^{N\times 1}$  is defined as:
$$
G [ j, k ] \triangleq \frac {1}{N} g (\mathbf {x} _ {j}, \mathbf {x} _ {k}); \quad g _ {\mathbf {x}} [ k ] \triangleq \frac {1}{N} g (\mathbf {x}, \mathbf {x} _ {k}). \tag {9}
$$
The  $g(\mathbf{x},\mathbf{q})$  is the Green's function [63] of this operator and is defined by:
$$
\int_ {\mathcal {X}} u (\mathbf {x}, \mathbf {x} ^ {\dagger}) g (\mathbf {x}, \mathbf {q}) \mathrm {d} \mathbf {x} ^ {\dagger} = \delta (\mathbf {x} - \mathbf {q}), \tag {10}
$$
where  $\delta$  is the Dirac delta function. Following the definition of Green's function, we know  $G$  is positive definite and hence decompose it as:
$$
G = V ^ {\top} D V, \tag {11}
$$
where  $D = \mathrm{diag}([d_1,\dots ,d_N])$  is determined by its eigenvalues and  $V$  contains  $N$  corresponding eigenvectors. Now, we can stack the model's prediction for different input samples  $\mathbf{x}_n$  and get the vector form solution of problem 6:
$$
\mathbf {f} ^ {*} \triangleq \left[ f ^ {*} \left(\mathbf {x} _ {1}\right) \right| \dots | f ^ {*} \left(\mathbf {x} _ {k}\right) ] ^ {\top} = G ^ {\top} (c I + G) ^ {- 1} \mathbf {Y} = V ^ {\top} D \left(c I + G\right) ^ {- 1} V \mathbf {Y} \tag {12}
$$
With this solution, following the settings in [53], we can explain where the compressibility pressure (the one that favors mappings with lower Kolmogorov complexity) comes from. Specifically, the optimal model for the first generation is  $\mathbf{f}_0^* = V^\top D(c_0I + G)^{-1}V\mathbf{Y}_0$ . Then in the following generations, the model in generation  $t$  will learn from the predictions of the model in the previous generation. As the problem is identical (the only difference is the labels) for different generalizations, we can have the following recursion formulas:
$$
\mathbf {f} _ {t} ^ {*} = V ^ {\top} D \left(c _ {t} I + G\right) ^ {- 1} V \mathbf {Y} _ {t} \quad \text {a n d} \quad \mathbf {Y} _ {t} = \mathbf {f} _ {t - 1} ^ {*}. \tag {13}
$$
Solving this yields the expression of the labels in the  $t$ -th generation:
$$
\mathbf {Y} _ {t} = V ^ {\top} A _ {t - 1} V \mathbf {Y} _ {t - 1} = V ^ {\top} \left(\prod_ {i = 0} ^ {t - 1} A _ {i}\right) V \mathbf {Y} _ {0}, \tag {14}
$$
where  $A_{t} \triangleq D(c_{t}I + D)^{-1}$  is a  $N \times N$  diagonal matrix. Substituting this back to Equation 12, we finally obtain the following expression:
$$
f _ {t} ^ {*} (\mathbf {x}) = g _ {\mathbf {x}} ^ {\top} V ^ {\top} D ^ {- 1} \left(\prod_ {i = 0} ^ {t} A _ {i}\right) V \mathbf {Y} _ {0} \tag {15}
$$
$$
\begin{array}{l} \mathbf {f} _ {t} ^ {*} = G V ^ {\top} D ^ {- 1} \left(\prod_ {i = 0} ^ {t} A _ {i}\right) V \mathbf {Y} _ {0} \\ = V ^ {\top} \left(\prod_ {i = 0} ^ {t} A _ {i}\right) V \mathbf {Y} _ {0}. \tag {16} \\ \end{array}
$$
From this solution, the model's prediction at  $t$ -th generation can be considered as a weighted combination of transformed  $\mathbf{Y}_0$ . The matrix  $V$  will first map  $\mathbf{Y}_0$  to a space determined by the Green's function. Different dimensions of this space are then rescaled by a diagonal matrix  $\prod_{i=0}^{t} A_i$ . After that, the vector is transformed back to the origin space by multiplying  $V^\top$ . Among these terms,  $\prod_{i=0}^{t} A_i$  is the only one that depends on  $t$ . Recall the definition of  $A_t = D(c_t I + D)^{-1}$ , we can conclude that  $\prod_{i=0}^{t} A_i$  is also a diagonal matrix where each entry has the form like  $\prod_{t} \frac{d_j}{c_t + d_j}$  (here  $d_j$  is the  $j$ -th eigenvalue of  $G$ ). As  $c_t > 0, \forall t$ , as stated in [53], all the diagonal entries will gradually decrease when  $t$  grows. The dimensions with smaller  $\prod_{t} \frac{d_j}{c_t + d_j}$  decrease faster, and vice versa. Recall
the role played by this diagonal matrix, we can imagine that the number of active bases in  $V$  is decreasing when  $t$  grows, which is a strong and unique inductive bias (i.e., compressibility pressure) introduced by this recursive training fashion<sup>9</sup>.
Now, we can link these theoretical analyses to the Kolmogorov complexity and compressibility pressure. The crux is the understanding of "active bases". Consider a toy example where  $\mathbf{G} = [G_1, G_2]$ , where each  $G_i$  has 4 possible values (there are  $N = 16$  different objects). Then, to memorize these 16 samples, the model needs 16 bases like "S 00 → blue circle". However, for compositional mappings, only 9 bases are enough<sup>10</sup>: because the model reuse some rules.
In summary, when  $t$  is small, there is no preference for compositional mappings because the number of active bases is large enough to remember most of the training samples. As  $t$  increases, the model then needs to be clever enough to reuse some bases, where the structure of the mapping emerges. If  $t$  is too large, where the compressibility pressure is too strong, the model will degenerate into a very naive solution, which is harmful for generalization (hence we need the interaction phase in iterated learning, discussed later).
C Experiments on Controlled Vision dataset
dSprites
3dShapes
MPI3D-Real
Figure 6: The toy vision datasets and a train/test split example. All images have size  $64*64$ . Pixels in dSprite are binary values while those in 3dShape and MPI3D contain 3 channels.
Trai
n split
#
Test split
O
Others
Table 5: Vision datasets considered in this paper. The numbers in the parentheses represent how many different values of the attribute. The last column means how many different samples we select for each G. Hence the number of samples in both training and test sets of these three datasets would be 9000, 8000, and 7200.
G1	G2	G3	G4	|G|	O	# per G
dSprites	shape (3)	scale (6)	pos-x (10 out of 32)	pos-y (10 out of 32)	1600	orientation (40)	5
3dShape	floor hue (10)	wall hue (10)	object hue (10)	object scale (8)	8000	object shape (4) orientation (15)	1
MPI3D	object color (6)	object shape (6)	horizontal x (10 out of 40)	vertical y (10 out of 40)	3600	size (2) camera (3) background (3)	2
Data generating factors In this paper, we conduct experiments on three vision datasets, i.e., dSprites [52], 3dShapes [9], MPI3D-real [23], where the ground truth  $\mathbf{G}$  are given. The summary and examples of these datasets are provided in Table 5 and Figure 6.
Here we specify how to split the dataset and generate the downstream labels using 3dShapes as an example. We denote the hue of the floor, wall, and object as  $G_{1}$ ,  $G_{2}$ , and  $G_{3}$ , respectively; each has 10 possible values, linearly spaced in [0, 1]. The object scale,  $G_{4}$ , has 8 possible values linearly spaced in [0, 1]. The remaining two factors, object shape (4 possible values) and object orientation (15 possible values), are treated as other factors and merged into  $\mathbf{O}$ . Data augmentation methods such as adding Gaussian noise, random flipping, and so on, are also merged into  $\mathbf{O}$ . Under this setting, the universe of  $\mathbf{G}$ , i.e.,  $\mathcal{G}$ , has 8000 different values, which is further divided into  $\mathcal{G}_{train}$  and  $\mathcal{G}_{test}$ . For
<sup>9</sup>The authors in [53] also prove that such a regularization cannot be achieved by other forms of regularizations. <sup>10</sup> Specifically, we need one basis like “S → z1 z2”, four bases like “z0 i → some color”, and four bases like “z1 j → some shape”. Please refer to Table 4.
the sys-gen problem, we assume  $\mathcal{G}_{train} \cap \mathcal{G}_{test} = \emptyset$ . One measure of the difficulty of the problem is the split ratio,  $\alpha = |\mathcal{G}_{train}| / |\mathcal{G}|$ ; smaller  $\alpha$  generally means a more challenging problem.
Data generating mechanisms For the training set, we first select  $\mathbf{G} \in \mathcal{G}_{train}$ , and then generate multiple input signals using  $\mathbf{x} = \mathrm{GenX}(\mathbf{G}, \mathbf{O})$ , where  $\mathbf{O}$  is uniformly random. The  $\mathbf{x}$  in the test set is generated in a similar way, but without including data augmentation in  $\mathbf{O}$ . Labels for all pairs in the dataset are generated by  $\mathbf{y} = \mathrm{GenY}(\mathbf{G}, \epsilon)$ . The main downstream task we study here is regression:  $\mathrm{GenY}(\mathbf{G}, \epsilon) = \mathbf{a}^{\top} \mathbf{G} + \epsilon$ , where  $\mathbf{a} = [a_1, \dots, a_m]^{\top}$  is a column vector and all  $a_i$  are chosen from  $[0, 1]^{11}$ . These examples assume  $\mathbf{y}$  is generated by a simple combination of different  $G_i$ , where recovering the generating factors is necessary to generalize well compositionally.
Model and training settings The model structure for this section is illustrated in Figure 2. We consider using a randomly initialized 4-layer CNN for dSprites and ReNet18 [31] for 3dShapes and MPI3D. Unless otherwise specified, we consider a linear head  $g(\mathbf{z})$ , and a typical  $\mathcal{L}_{ds}$ , i.e., cross-entropy loss for classification and mean square error loss for regression. The networks are optimized using a standard SGD optimizer with a learning rate of  $10^{-3}$  and a weight decay rate of  $5 * 10^{-4}$ . Actually, we find the results are insensitive to these settings.
C.2.1 Some interesting observations
Figure 7: Left to right: 1.) zoom in on the curves of training and test loss; 2.) demonstration of what will happen to test loss when replacing the sampling by argmax in the imitation phase; 3.) how the average predicting entropy, i.e.,  $\mathbb{E}[H(\bar{z}_l)]$ , changes; 4.) histograms of predicting probabilities in different generations. All panels come from experiments of SEM-IL on 3dShapes.
See the first panel in Figure 7, which demonstrates the training and testing loss when training a model using SEM-IL. In the first generation, we see the training loss is always smaller than the test one. The test loss then plateaus after some epochs, which matches our expectations. However, in the following generations, the test loss would decrease faster than the training one at the beginning of the interaction phase, which is quite counter-intuitive. We would explore why this happens and whether it is a sign of increased topological similarity in the future.
Another observation is about the sampling mechanism applied in the imitation phase. Remember in Algorithm 1, the pseudo labels used in the imitation phase are sampled from the teacher's prediction  $\bar{z}_l$ . Hence if the teacher is confident in some attributes, the generated labels would be consistent in different epochs, and vice versa. From Figure 4, we provide the scatter plot of the correlation between the teacher's confidence and the student's learning speed. Here we verify this hypothesis through an ablation study. Specifically, we replace the sampling procedure with an argmax function, i.e., the teacher always provides the label with the largest predicting probability regardless of its confidence. As illustrated in the second panel in Figure 7, the test performance of the argmax-case is much worse than the standard SEM-IL method.
The benefits introduced by sampling pseudo labels can also be interpreted as we are making self-adapting  $\tau$  for different input samples in SEM. In the origin SEM, we only have one  $\tau$  to control the average entropy of the backbone's prediction (lower  $\tau$  leads to peakier predicting distributions, and vice versa). However, a model trained using SEM-IL equivalently has different  $\tau$  for different  $\mathbf{x}$
We also tried a simple non-linear mapping from  $\mathbf{G}$  to  $y$ , i.e.,  $\mathrm{GenY}(\mathbf{G}, \epsilon) = a_1 \cdot G_1 + a_2 \cdot G_2 + a_3 \cdot G_4 G_3 + \epsilon$  and a multi-task scenario, where  $\mathbf{a} \in \mathbb{R}^{|y| \times m}$  is a matrix. The resulting trends in these settings are quite similar.
To understand this, we can compare the entropy of  $h_1(\mathbf{x})$  and  $h_1(\mathbf{x}')$ , where the teacher is confident in  $(\mathbf{x}, \mathbf{g}_1 = 0)$  and less confident in  $(\mathbf{x}', \mathbf{g}_1 = 0)$ . Then during imitation, the student will remember  $(\mathbf{x}, \mathbf{g}_1 = 0)$  much faster than  $(\mathbf{x}', \mathbf{g}_1 = 0)$  and hence assign  $\bar{z}_1 = 0$  higher probability when the input is  $\mathbf{x}$ . On the contrary, for the input  $\mathbf{x}'$ , as the student might receive different corresponding pseudo labels during imitation, it would assign a lower probability for  $\bar{z}_1 = 0$  given  $\mathbf{x}'$ . As a result, the entropy  $H(\bar{z}_1 \mid \mathbf{x})$  and  $H(\bar{z}_1 \mid \mathbf{x}')$  of the student model would be very different as the teacher have different confidence when generating the pseudo labels, which is equivalent as automatically selecting different  $\tau$  for  $\mathbf{x}$  and  $\mathbf{x}'$ .
The last two panels in Figure 7 demonstrate how the entropy of the model's prediction on  $\mathbf{G}$  changes in different generations. (Remember the output of the backbone after SEM, i.e.,  $\mathbf{z} = [\bar{z}_1,\dots ,\bar{z}_m]$ , are  $m$  simplicial vectors with length  $v$ .) From the figures, we see the entropy gradually decreases as the training goes on, which means the model is becoming more and more confident in its predictions on average. However, the last panel shows that there are still many unconfident predictions even after converging on the downstream task performance: we speculate that these factors contain little information on  $\mathbf{G}$  as they might have high entropy.
C.2.2 Influence of task difficulty
As the ground-truth generating factors are accessible for these vision datasets, we could explore how the difficulty of the task, i.e.,  $\alpha$ , influence the performance gap among different methods. As shown in Table 6, SEM-IL brings a significant enhancement when  $\alpha$  is not too big nor too small. When  $\alpha$  is big, the test loss might be very small and there is no room to make considerable improvement. On another extreme, if  $\alpha$  is too small, some attributes might be never observed during training, which is hard for the model to extract correct generating factors. Remember we expect to generalize to "red circle" by knowing the concept of red and circle from other combinations: if there is only "blue" and "box" in the training set, it is impractical for a model to learn such a concept (maybe the model can extrapolate, but that is another topic). We speculate most real tasks are in a relatively small  $\alpha$  regime, as the generating factors and their possible values can be very large. Please note that as the train/test split and GenY all depend on the random seeds, the variance of the numbers in these tables could be large. However, we observe the SEM-IL consistently outperforms other methods under each task generated by different random seeds.
Table 6: Relative improvement comparison for different  $\alpha$  on three datasets. We report the average and standard error of 4 different runs.  $\Delta_{1}$  is calculated by SEMIL-Baseline, while  $\Delta_{2}$  is calculated by SEMIL-Baseline. The test MSE for different settings are the numbers multiplying  $10^{-3}$ . Note that the numbers of different datasets are not comparable, as the model structure and generating factors of them are different. When  $\alpha$  is too small, the model fails to converge on MPI3D, which means MPI3D might be a more challenging dataset.
α	0.8	0.5	0.2	0.1	0.02
3dShapes	Baseline	3.778±0.792	7.902±2.000	28.01±11.75	57.87±9.852	355.5±136.0
NIL-only	3.866±0.733	7.536±1.966	33.18±15.16	56.46±12.54	330.5±183.1
SEM-only	2.531±0.742	5.15±0.415	21.41±5.274	55.48±15.76	292.7±148.0
SEM-IL	0.633±0.117	1.27±0.112	5.165±0.697	17.52±3.103	221.0±122.5
Relative Δ1	0.8324	0.8396	0.8156	0.6972	0.3783
Relative Δ2	4.968	5.236	4.423	2.303	0.6086
MPI3D	Baseline	45.42±10.97	61.95±17.80	125.9±29.30	234.0±34.94	Not Converge
NIL-only	43.38±14.03	57.34±17.35	110.8±36.06	203.3±73.62	Not Converge
SEM-only	42.91±10.05	57.69±18.34	116.5±38.50	204.1±72.68	Not Converge
SEM-IL	31.20±8.053	40.33±12.12	73.43±22.63	137.8±67.36	Not Converge
Relative Δ1	0.313	0.349	0.417	0.411	-
Relative Δ2	0.456	0.536	0.714	0.698	-
dSprites	Baseline	0.172±0.145	7.906±2.309	109.2±10.28	313.5±28.47	839.7±73.85
NIL-only	0.136±0.123	3.678±0.869	56.42±7.169	241.0±24.68	630.2±32.53
SEM-only	0.126±0.119	7.667±1.937	108.8±8.563	315.2±23.77	658.0±82.52
SEM-IL	0.085±0.042	2.487±0.874	40.11±9.726	213.1±38.11	596.8±41.94
Relative Δ1	0.506	0.685	0.633	0.320	0.289
Relative Δ2	1.023	2.179	1.722	0.471	0.407
C.3 Other baselines from disentangled representation learning
Most related works consider sys-gen as an NLP or emergent language problem rather than a general representation learning problem, so to the best of our knowledge, there are no specific advanced baselines for this concrete problem. The most related works are some VAE-based methods in disentanglement learning. In this part, we re-implement  $\beta$ -VAE and compare them with the baseline method in our setting. Specifically, we first pre-train the encoder of VAE on the same training set and then attach a task head for the downstream task to the "  $\mu$ -part" of the encoder's prediction (note that the encoder will output "  $\mu$ -part" and "  $\sigma$ -part" together). We observe that the VAE-based method performs worse than the baseline method, even though they seem to recover some disentangled factors when conducting latent traversal. This observation is consistent with the findings in [65] and [81], where the authors claim that the disentangled representations are incapable of reliably generalizing to new conceptual combinations. We also speculate that the challenging requirement of the comp-gen problem, i.e.,  $\mathcal{G}_{train} \cap \mathcal{G}_{test} = \emptyset$ , exacerbates this: there will not be enough variations in  $\mathbf{x}$  to make the VAE model capture the latent vectors precisely. However, as VAE is also an encoder-decoder system, it is possible to combine SEM-IL with it, which is left for our future work.
D Experimental Settings and More Results on Molecular Graph Dataset
(a). The standard scaffold split
Step3: order all training  $\mathbf{x}$  by the score
Step4: prune  $\eta \%$  training samples with highest scores
(b). A more challenging comp-gen split
Figure 8: Left: an example of the scaffold split, the figures are copied from [79]. Right: the procedure of a more challenging few-shot split used in Table 1.
Step2: calculate the similarity score for each x in training set using the fingerprint vector