SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 1 is a straightforward extension of Theorem 1, in which we consider multiple training tasks instead of a single training task. In Corollary 1, the generalization error on a novel task  $\tau_j^{\prime}$  is bounded partially by the average distance between task  $\tau_j^{\prime}$  and  $n$  training tasks. Note that the distance  $d_{\theta}(\tau_i,\tau_j^{\prime})$  used in the bound is model-related. Next, we further derive the relationship between model-related distance and model-agnostic distance as follows.
Theorem 2. With the same notation and assumptions as in Corollary 1, assume that the conditional distribution  $p(x|a^l)$  is task agnostic. If the number of labeled samples  $m_i$  in  $n$  training tasks and the number of labeled samples  $m_j'$  in novel task  $\tau_j'$  tend to be infinite, the following equality holds:
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} d _ {\theta} \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right) \leq \frac {1}{n} \sum_ {i = 1} ^ {n} d \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right). \tag {7}
$$
Theorem 2 shows that when the number of labeled samples in  $n$  training task and a novel task tends to be infinite, the model-related average distance can be bounded by the average TAD. Based on the universal approximation theorem [21, 15] and the same assumption as in Theorem 2, if  $g_{\phi} \circ f_{\theta}$  is a complex enough network, the second and fourth terms on the r.h.s. of Eq. (6) in Corollary 1 will converge to 0, the first term  $\hat{\epsilon}(f_{\theta}, \tau_{i=1}^{n})$  and  $\frac{1}{n} \sum_{i=1}^{n} \lambda_{i}$  will converge to Bayes risk [13]. We assume the Bayes risk is small enough, so that it can be ignored. In this case, the generalization error on novel task  $\tau_{j}'$  is only bounded by  $\lambda_{j}'$  and the average TAD, i.e.,  $\epsilon(f_{\theta}, \tau_{j}') \leq \lambda_{j}' + \frac{1}{n} \sum_{i=1}^{n} d(\tau_{i}, \tau_{j}')$ . Note that the average TAD is independent of models, thus we can rely solely on the TAD metric to measure the adaptation difficulty of each novel task, without the need for training any models.
5 Computing the Model-Agnostic Task Distance
5.1 Distance Approximation
In this section, we discuss practical considerations in computing the TAD on real data, where the attributes often take on discrete and finite values. For continuous and infinite attributes, it is possible to divide their values into many segments or discrete parts. Therefore, we denote  $V^l$  as a finite set of possible values of attribute  $a^l$ , and re-express the TAD in Eq. (2) as
$$
d \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right) = \frac {1}{2 L | M |} \sum_ {e _ {k t} \in M} \sum_ {l = 1} ^ {L} \sum_ {v \in V ^ {l}} \left| P _ {a ^ {l} | y _ {k}} ^ {i} [ v ] - P _ {a ^ {l} | y _ {t}} ^ {j} [ v ] \right|. \tag {8}
$$
Computing the above distance is challenging as it requires finding the maximum matching  $M$  with minimum weights, which is a combinatorial optimization problem. The Hungarian algorithm [25] is commonly used to solve the matching problem, but it is computationally expensive. Due to the high computational cost, we do not calculate this distance directly but estimate the approximation instead:
$$
d \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right) \approx \frac {1}{2 L C} \sum_ {l = 1} ^ {L} \sum_ {v \in V ^ {l}} \left| \sum_ {k = 1} ^ {C} P _ {a ^ {l} | y _ {k}} ^ {i} [ v ] - \sum_ {t = 1} ^ {C} P _ {a ^ {l} | y _ {t}} ^ {j} [ v ] \right|, \tag {9}
$$
where we assume all tasks have  $C$  categories, which holds in most FSL settings. In Eq. (9), we simplify the comparison of attribute conditional distributions between two tasks by replacing the individual differences with the average difference. Through the above approximation, we can avoid seeking the minimum weight perfect matching  $M$  and estimate the TAD efficiently.
5.2 Attributes Auto-Annotation
As stated before, our proposed theory and TAD metric rely on category-level attribute annotations that are possibly unavailable for some few-shot benchmarks, like miniImageNet [48]. To address this issue, we propose a solution to auto- annotate attributes using a pretrained CLIP [36] model, which has demonstrated impressive zero-shot classification ability. We first pre-define 14 attribute labels based on pattern, shape, and texture. Then we create two descriptions for each attribute, such as "a photo has shape of round" and "a photo has no shape of round", which are used as text inputs for CLIP. We formulate the annotation problem as 14 attribute binary classification problems for each image using CLIP model. After that, we gather the attribute predictions of all images within the same category, which provides rough category-level attribute information while greatly reduces the cost of attribute annotations. With the above auto-annotation process, we can obtain the category-level attributes for miniImageNet in just 4 minutes.
In this section, we conduct experiments to validate our theoretical results and explore the generality of the proposed TAD metric in measuring task adaptation difficulty. Firstly, we evaluate our TAD metric in a scenario where human-annotated attributes are available and the training/novel tasks are sampled from the same dataset but with different categories. Next, we test the TAD in a more general and challenging scenario where human-annotated attributes are not available and the training tasks are constructed by sampling categories from different datasets.
Datasets. We choose three widely used benchmarks: (1) CUB-200-2011 (CUB) [49]: CUB is a fine-grained dataset of birds, which has 200 bird classes and 11,788 images in total. We follow [23] to split the dataset into 100 training classes, 50 validation classes and 50 test classes. As a fine-grained dataset, CUB provides part-based annotations, such as beak, wing and tail of a bird. Each part is annotated by a bounding box and some attribute labels. Because the provided attribute labels are noisy, we denoise them by majority voting, as in [24]. After the pre-processing, we acquire 109 binary category-level attribute labels. (2) SUN with Attribute (SUN) [34]: SUN is a scene classification dataset, which contains 14,340 images for 717 scene classes with 102 scene attributes. In following experiments, We split the dataset into 430/215/72 classes for training/validation/test, respectively. (3) miniImageNet [48]: miniImageNet is a subset of ImageNet consisting of 60,000 images uniformly distributed over 100 object classes. Note that, different with CUB and SUN, miniImageNet does not provide the attribute annotations, which means category-level attribute annotations are not available. Following [10], we consider the cross-dataset scenario from miniImageNet to CUB, where we use 100 classes of miniImageNet as training classes, and the 50 validation and 50 test classes from CUB.
Attribute Prototypical Network. Our theoretical analysis is based on a specific meta-learning framework with attribute learning (proposed in Sec. 4.2). Thus, we first instantiate a simple model under that framework as an example to verify our theory. Specifically, we adopt a four-layer convolution network (Conv-4) with an additional MLP as the meta-learner  $f_{\theta}$ . The convolutional network extracts feature representations from images, then the MLP takes features as input and predicts attribute labels. For base-learner  $g_{\phi_i}$  parameterized by  $\phi_i$ , we simply choose an non-parametric base-learner like ProtoNet [42], which takes the attributes outputted by  $f_{\theta}$  as input to calculate cosine distance between test samples and attribute prototypes, then predicts the target label. We call this method as Attribute Prototypical Network (APNet). We train APNet by simultaneously minimizing the attribute classification loss and the few-shot classification loss.
Other FSL Methods. Besides of APNet, we choose five classical FSL methods in the following experiments, since they cover a diverse set of approaches to few-shot learning: (1) Matching Network (MatchingNet) [48], (2) Prototypical Network (ProtoNet) [42], (3) Relation Network (RelationNet) [45], (4) Model Agnostic Meta-Learning (MAML) [17] and (5) Baseline++ [10]. Note that these FSL methods, unlike the above APNet, do not use attribute annotations during training. See more implementation and experimental details in the Appendix.
6.2 Task Attribute Distance with Human-annotated Attributes
Approximate results with limited samples. According to Theorem 2, if the number of labeled samples in tasks tends to be infinite, the TAD serves as a metric to measure the adaptation difficulty on a novel task. However, it is impossible that each task contains infinite labeled samples in real-world applications. Thus, we try to empirically verify the approximate theoretical result with limited labeled samples for each task. We train our APNet on CUB and SUN dataset, then use the provided attribute annotations to calculate the average distance  $\frac{1}{n}\sum_{i=1}^{n}d(\tau_i,\tau_j')$  between each novel task  $\tau_j'$  and  $n$  training tasks  $\{\tau_i\}_{i=1}^n$ , respectively. For simplicity, we sample  $n = 100,000$  training tasks to estimate the distances to 2, 400 novel tasks. Following  $N$ -way  $K$ -shot setting, each task only contains  $N*K$  labeled samples for training or fine-tuning models. Fig. 4 shows the task distance and the corresponding accuracy on novel tasks for APNet. We can observe in Fig. 4 that as the distance increases, the accuracy of APNet decreases in both 5-way 1-shot and 5-way 5-shot settings on CUB and SUN. These results verify that TAD can characterize model's generalization error on each novel task and measure the task adaptation difficulty effectively, even with limited labeled samples for
Figure 2: Accuracy of APNet in terms of the average task distance. (a)-(b) 5-way 1-shot and 5-shot on CUB. (c)-(d) 5-way 1-shot and 5-shot on SUN. In (a)-(d), each gray point denotes the average accuracy for all points in a distance interval, and the error bar denotes the confidence interval at  $95\%$  confidence level. The red dashed line is a fitted line, which shows the tendency of gray points.
Figure 3: Accuracy of different methods in terms of the average task distance. From left to right, 5-way 1-shot and 5-shot on CUB/SUN. Each point denotes the average accuracy in a distance interval.
training and fine-tuning models. Note that the confidence interval in Fig. 4 is much larger for the last a few points. We argue this is because these distance intervals contain fewer novel tasks, thus the average accuracy of novel tasks in the interval is more easily affected by random factors.
The number of labeled samples. Additionally, in Fig. 4, when comparing 1-shot and 5-shot results with the same distance interval, we find that the increase of accuracy varies at different distance intervals. For instance, on the CUB dataset (comparing Fig. 2(a) with Fig. 2(b)), when the task distance is 0.14, APNet shows an improvement of approximately  $10\%$  in accuracy for the 5-shot setting over the 1-shot setting, whereas it shows an improvement of around  $15\%$  for the distance of 0.20. This suggests that increasing the number of labeled samples is more effective for harder tasks. One possible explanation is that as the task distance increases, less knowledge can be transferred from training tasks, making it harder for models to adapt to the novel task. Hence, more information from the novel task is required for adaptation, and the model's performance can get more benefit from additional labeled samples.
Other FSL models. We have shown that the average TAD can effectively reflect the task adaption difficulty for APNet. A natural question is whether the calculated task distance can be directly applied to other FSL methods, even if they do not follow the specific meta-learning framework and do not use the attributes during training. We try to empirically explore this question. With the same distance estimation and experimental setting, we conduct experiments with five different FSL methods on CUB and SUN. Fig. 3 shows the task distance and the corresponding accuracy of 2,400 novel test tasks for them. We observe similar results that with the increase of task distance, the accuracy of all FSL models tends to decrease. This indicates that even though these FSL methods do not use the attribute annotations during training, they implicitly learn mixture of attributes in their representations. Therefore, attribute-based distance can still reflect the task adaptation difficulty for them. These results demonstrate the generality of the proposed TAD metric, and provide some insight into the transferable knowledge that different FSL models have learned.
6.3 Task Attribute Distance with Auto-annotated Attributes
Cross-dataset generalization. Another interesting question is how to utilize the proposed TAD metric in situations where human-annotated attributes are either unavailable or expensive to obtain. One classic example is the cross-domain few-shot learning problem, since it requires the annotation
(b)
Figure 4: Accuracy of different methods in terms of the average task distance with (a) all attributes, (b) removal of pattern attributes, (c) removal of shape attributes, (d) removal of texture attributes. The experiment is conduct in 5-way 5-shot setting.
of distinct datasets with a common attribute set. To address this challenge, we use the auto-annotation method described in previous section to annotate the attributes for both miniImageNet and CUB. We then train the above FSL models on the cross-dataset scenario from miniImageNet to CUB, and estimate the average task distance with the auto-annotated attributes. Fig. 4(a) illustrates the distance and corresponding accuracy of 2,400 novel test tasks with the auto-annotated attributes. We can find that, with the increase of task distance, the accuracy of different models tends to decrease. This phenomenon is consistent with our previous findings and shows the proposed TAD metric still works with auto-annotated attributes. Besides, we selectively remove some attributes to explore the influence of them in the distance-accuracy curve. Figs. 4(b) to 4(d) show that the exclusion of pattern, shape, and texture attributes exhibits varying degrees of influence on the decreasing tendency. Notably, we discover that texture attributes are more importance than others, as indicated by the more pronounced fluctuations in the curve.
Figure 5: 1-shot results of test-time intervention (miniImageNet  $\rightarrow$  CUB). We only show the accuracy of the intervened tasks.
100 classes	64 classes	Δ
MatchingNet	50.0	51.7	-1.7
ProtoNet	53.7	52.6	1.1
RelationNet	52.6	50.3	2.3
MAML	51.6	49.7	1.9
Baseline++	52.4	55.5	-3.1
Table 1: Comparison of 5-shot accuracy for different methods, when training on selected 64 classes instead of all 100 classes.  $\Delta$  represents the difference in accuracy between the two cases. The best accuracy of each method is marked in bold.
Test-time Intervention. With the proposed TAD metric, we can quantify the adaptation difficulty of novel test tasks without training and adapting a model. This makes sense in real-world scenarios, as we can identify more challenging tasks before training and take interventions to improve a model's performance on those tasks. In this part, we explore a simple test-time intervention operation that supplies more labeled samples for harder novel tasks. To construct the intervention experiment, we first calculate the distances between each novel task and  $n$  training tasks, as done in previous experiments. We manually set a constant threshold value, denoted as  $r$ , to identify the harder tasks that exhibit large distances and generally yield low accuracy. Once we identify these challenging tasks, we intervene by providing additional labeled samples, which contain more task-specific information. Specifically, in the 5-way 1-shot setting, we offer 4 extra labeled samples for each intervened task, which can be seen as the additional annotated samples. We set the distance threshold  $r = 0.29$  for the cross-dataset scenario. With the above threshold values, we only need to intervene with a small subset of novel tasks (approximately  $5\% - 7\%$ ), which greatly reduces the cost of sample annotation. The results of the test-time intervention can be seen in Fig. 5. As illustrated in Fig. 5, the test-time intervention can significantly improve the performance of different FSL models on the challenging novel tasks. Note that we are not proposing a novel intervention method, but demonstrating a practical application of the proposed TAD metric.
Training with partial versus all tasks. Using the proposed TAD, we can also investigate whether training on all available training tasks leads to better generalization compared to training on only the most related ones. While it is generally expected that training with more tasks helps to improve generalization, it remains a question whether this holds for heterogeneous data (e.g. cross-dataset scenario). To explore this, we follow the cross-dataset experiment setup from miniImageNet to CUB, and select these less related training tasks based on their calculated task distances. We collect the classes in the selected tasks, compute the frequency of each class, and remove the most frequent 36 classes (out of a total of 100 classes in miniImageNet). We then reconstruct training tasks based on the remaining 64 classes in miniImageNet and retrain five FSL models. The results are presented in Tab. 1. It can be observed that incorporating more heterogeneous data that is less related to the novel tasks may not lead to much improvement ( for ProtoNet, RelationNet and MAML), but could even result in performance degradation ( for MatchingNet and Baseline++) This result may inspire future research on how to better utilize heterogeneous data to improve the performance of models in the cross-domain few-shot learning problem.
We propose a novel distance metric, called Task Attribute Distance (TAD), in FSL to quantify the relationship between training and novel tasks, which only relies on the category-level attribute annotations. We present a theoretical analysis of the generalization error bound on a novel task with TAD, which connects task relatedness and adaptation difficulty theoretically. Our experiments demonstrate TAD can effectively reflect the task adaptation difficulty for various FSL methods, even if some of them do not learn attributes explicitly or human-annotated attributes are not available. We also present two potential applications of our TAD metric: intervening a small set of difficult novel tasks before training, and investigating the benefits of heterogeneous training tasks. We believe our theoretical and empirical analysis can provide more insight into few-shot learning and related areas.
Limitations and broader impact. Our theory and proposed TAD metric assume that attributes are conditionally independent without considering the correlations that may exist between them. Furthermore, identifying which attributes are critical in the distance metric is still an open question. However, we believe that our work lays a solid foundation for measuring task relatedness and adaptation difficulty of novel tasks, which offers a starting point for further research in few-shot learning and related areas.
This work is partially supported by National Key R&D Program of China no. 2021ZD0111901, and National Natural Science Foundation of China (NSFC): 61976203 and 62276246.
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In ICCV, 2019.
[2] Arman Afrasiyabi, Hugo Larochelle, Jean-François Lalonde, and Christian Gagné. Matching feature sets for few-shot image classification. In CVPR, 2022.
[3] Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICML, 2019.
[4] Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In ICML, 2018.
[5] Antreas Antoniou, Harri Edwards, and Amos Storkey. How to train your MAML. In ICLR, 2019.
[6] Sébastien Arnold, Guneet Dhillon, Avinash Ravichandran, and Stefano Soatto. Uniform sampling over episode difficulty. 2021.
[7] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, 2009.
[8] Tianshi Cao, Marc T Law, and Sanja Fidler. A theoretical analysis of the number of shots in few-shot learning. In ICLR, 2020.
[9] Susan Carey and Elsa Bartlett. Acquiring a single new word. Papers and Reports on Child Language Development, 15:17-29, 1978.
[10] Weiyu Chen, Yencheng Liu, Zsolt Kira, Yuchiang Frank Wang, and Jiabin Huang. A closer look at few-shot classification. In ICLR, 2019.
[11] Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In ICCV, 2021.
[12] Nan Ding, Xi Chen, Tomer Levinboim, Sebastian Goodman, and Radu Soricut. Bridging the gap between practice and PAC-Bayes theory in few-shot meta-learning. In NeurIPS, 2021.
[13] Geoff Dougherty. Pattern recognition and classification: an introduction. Springer Science & Business Media, 2012.
[14] Simon Shaolei Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In ICLR, 2021.
[15] András Faragó and Gábor Lugosi. Strong universal consistency of neural network classifiers. IEEE Transactions on Information Theory, 39:1146-1151, 1993.
[16] Nanyi Fei, Yizhao Gao, Zhiwu Lu, and Tao Xiang. Z-score normalization, hubness, and few-shot learning. In ICCV, 2021.
[17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
[18] Alison L Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International statistical review, 70:419-435, 2002.
[19] Jiechao Guan and Zhiwu Lu. Fast-rate PAC-Bayesian generalization bounds for meta-learning. In ICML, 2022.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
[21] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2:359-366, 1989.
[22] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In NeurIPS, 2019.
[23] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In ICCV, 2021.
[24] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, 2020.
[25] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2:83-97, 1955.
[26] Cat Phuoc Le, Juncheng Dong, Mohammadreza Soltani, and Vahid Tarokh. Task affinity with maximum bipartite matching in few-shot learning. In ICLR, 2022.
[27] SuBeen Lee, WonJun Moon, and Jae-Pil Heo. Task discrepancy maximization for fine-grained few-shot classification. In CVPR, 2022.
[28] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv, 2017.