SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minyang Hu $^{1,2}$ , Hong Chang $^{1,2}$ , Zong Guo $^{1,2}$ , Bingpeng Ma $^{2}$ , Shiguang Shan $^{1,2}$ , Xilin Chen $^{1,2}$
<sup>1</sup> Institute of Computing Technology, Chinese Academy of Sciences
{minyang.hu, zong.guo}@vipl.ict.ac.cn, {changhong, sgshan, xlchen} @ict.ac.cn, bpma@ucas.ac.cn
Few-shot learning (FSL) aims to learn novel tasks with very few labeled samples by leveraging experience from related training tasks. In this paper, we try to understand FSL by exploring two key questions: (1) How to quantify the relationship between training and novel tasks? (2) How does the relationship affect the adaptation difficulty on novel tasks for different models? To answer the first question, we propose Task Attribute Distance (TAD) as a metric to quantify the task relatedness via attributes. Unlike other metrics, TAD is independent of models, making it applicable to different FSL models. To address the second question, we utilize TAD metric to establish a theoretical connection between task relatedness and task adaptation difficulty. By deriving the generalization error bound on a novel task, we discover how TAD measures the adaptation difficulty on novel tasks for different models. To validate our theoretical results, we conduct experiments on three benchmarks. Our experimental results confirm that TAD metric effectively quantifies the task relatedness and reflects the adaptation difficulty on novel tasks for various FSL methods, even if some of them do not learn attributes explicitly or human-annotated attributes are not provided. Our code is available at https://github.com/hu-my/TaskAttributeDistance.
Learning in human biological system is highly efficient compared to artificial systems. For example, only one example is enough for a child to learn a novel word [9], while thousands of training samples are needed for deep learning models. This learning efficiency comes from the past experiences accumulated by human brain. Inspired by human learning capability, Few-Shot Learning (FSL) aims to learn novel tasks with very few samples by leveraging experience from related training tasks.
Despite the great progress made in FSL during the past decade [48, 42, 17, 45, 22, 10, 16, 50, 27], there is less exploration on what it means for training tasks to be related. Some prior studies [48, 17, 45] focus on constructing related training tasks by sampling categories within a dataset. Recent works [46, 33] consider training tasks sampled from different datasets, known as cross-domain few-shot learning. The latter is considered as a more challenging problem, partly because the category gaps across datasets are usually larger, resulting in training tasks less related to novel tasks. Although this follows the intuition, two natural questions arise: (1) How to quantify the relationship between training and novel tasks? (2) How does the relationship affect the adaptation difficulty on novel tasks for different FSL models?
To quantify the relationship between tasks, one approach is to measure the category relationship in a common representation space. With properly learned representation function, various distribution divergence metrics, like EMD [39] and FID [20], can be used to calculate the distance between feature distributions of different categories, as well as tasks over them. The main obstacle to this
solution is how to learn a good representation function for novel categories with only a few labeled samples. Previous works [32, 33] assume that the representation function learned from training categories can be directly applied to novel categories, but this assumption may not hold due to the large category gaps. Some recent studies [1, 26] utilize the Fisher Information Matrix to quantify the task relatedness without assuming a common representation function. However, this approach has a heavy burden of computing Hessian matrix. Besides, the calculated task distances are highly dependent on the learned model, making them difficult to apply to other FSL models.
To investigate the influence of task relatedness on the difficulty of adapting models to novel tasks, existing studies [37, 40, 32, 33] often make an empirical assumption that a novel task with large distances to the training tasks will be hard to adapt to. With this assumption, some works [40, 32, 33] employ specific distance metrics to quantify the task relatedness, and then construct more challenging benchmarks to explore the generalization ability of different models. Different with the these works, [26] select the most related training data to improve the episodic fine-tune process for FSL models based on an asymmetric task relatedness measure. Despite their empirical success, all the aforementioned works lack theoretical analysis, leaving the connection between task relatedness and task adaptation difficulty not formally clarified.
In this work, we try to overcome the existing obstacles and answer the two questions via attributes. Firstly, we introduce Task Attribute Distance (TAD) as a metric to quantify the task relatedness. Our intuition lies in the attribute compositionality hypothesis (shown in Fig. 1): each category can be represented as a composition of attributes, which are reusable in a huge assortment of meaningful compositions. TAD formulates this intuition by first measuring the total variation distance between attribute distributions of training and novel categories, and then finding the maximum matching with minimum weight of a bipartite graph. Unlike other metrics, TAD only relies on the attribute conditional distributions, making it independent of models and the number of samples in novel tasks. Secondly, we utilize the TAD metric to establish a theoretical connection
Figure 1: The categories and attribute sets of three bird images. Each category can be represented as a composition of some attributes, which act as a relationship bridge between different categories.
between task relatedness and task adaptation difficulty in FSL. We provide theoretical proof to verify that TAD contributes to the generalization error bound on a novel task, at different training settings. Importantly, our theoretical results indicate that the adaptation difficulty on a novel task can be efficiently measured by relying solely on the TAD metric, without the need for training any models.
We conduct experiments on three benchmark datasets to validate our theoretical results and explore the generality of TAD metric. To this end, we consider two scenarios that the human-annotated attributes are provided or not available. For the latter case, we propose an efficient way to auto- annotate attributes. Experimental results show that as the distance between novel and training tasks increases, the performance of various FSL models on novel tasks decreases. With the TAD metric, we also present two potential applications: intervening a small set of difficult novel tasks before training, and investigating the benefits of heterogeneous training tasks for different models.
- We propose Task Attribute Distance (TAD) metric in FSL to quantify the relationship between training and novel tasks, which is model-agnostic and easy to compute.
- We provide a theoretical proof of the generalization error bound on a novel task based on TAD, which connects task relatedness and adaptation difficulty theoretically.
- We conduct experiments to show TAD can effectively reflect the adaptation difficulty on novel tasks for various FSL methods with human-annotated and auto-annotated attributes.
Few-shot learning. Most recent few-shot learning (FSL) approaches are based on meta-learning framework. These meta-learning based methods can be roughly categorized into two categories. The
metric-based approach [48, 42, 45, 22, 3, 16, 2, 50] aims to learn a cross-task embedding function and predict the query labels based on the learned distances. The optimization-based approach focuses on learning some optimization state, like model initialization [17, 44] or step sizes [28, 5], to rapidly update models with very few labeled samples. Except for these meta-learning methods, transfer-based approaches [10, 11] have recently shown that standard transfer learning procedure of early pre-training and subsequent fine-tuning is a strong baseline for FSL with deep backbone networks.
Meta-learning theory. In parallel to the empirical success of meta-learning methods, a series of theoretical works study how meta-learning utilizes the knowledge obtained from the training task data and generalizes to the novel test tasks. Many works [35, 4, 38, 12, 19] give a generalization error bound on novel tasks from the PAC-Bayesian perspective [30, 31]. These works often assume that each task is sampled from a meta task distribution. Under such assumption, the generalization error bound on novel tasks can be reduced with increasing number of training tasks and training samples [4]. Some recent works replace the meta task distribution assumption with other conditions. For example, [8, 14, 47] assume a common representation function between different tasks, then they give the sample complexity bounds for the expected excess risk on a novel task. However, the above works do not quantify the relationship between training and novel tasks, and seldom explore the adaptation difficulty on novel tasks. In this paper, we propose the TAD as a metric to quantify the task relatedness, and provide a new generalization error bound on a novel task.
Task Difficulty. Exploring the difficulty of few-shot tasks is an important research question in FSL, and has been explored from two aspects: measure the difficulty of (1) training tasks and (2) novel tasks. For training tasks, many previous works [44, 29, 43, 6] have attempted to measure their difficulties based on a model's output, such as negative log-likelihood or accuracy, and use this information to sample different tasks to train the model. The sampling strategy based on training task difficulty is similar to the area of hard example mining [41] or curriculum learning [7] that trains a model according to a specific order of samples to improve its generalization ability. For novel tasks, recent works [1, 37, 32, 40, 26, 33] try to measure the adaptation difficulty of novel tasks based on the relationship between novel and training tasks. Task2vec [1] encodes each novel task into an embedding based on the Fisher Information Matrix, and uses the norm of embedding to reflect task difficulty. Other works utilize specific distribution divergence metrics, such as EMD and FID, to quantify the task relatedness, then select the most related training data to fine-tune models [26] or construct more challenging benchmarks to explore the generalization of models [37, 40, 33]. Closer to our work, [37] introduces a transferability score to quantify the transferability of novel tasks via attributes and explore the generalization of FSL models at different transferability scores. However, different with our work, [37] aims to investigate the benefits of self-supervised pretraining with supervised finetuning in the few-shot attribute classification context.
In few-shot learning problem, a model observes  $n$  different training tasks  $\{\tau_i\}_{i=1}^n$  with each task represented as a pair  $\tau_i = (\mathcal{D}_i, S_i), 1 \leq i \leq n$ .  $\mathcal{D}_i$  is an unknown data distribution over the input space  $\mathcal{X}$  and label space  $\mathcal{Y}_i$ .  $S_i = \{(x_k, y_k)\}_{k=1}^{m_i}$ ,  $(x_k, y_k) \sim \mathcal{D}_i$  represents an observed training set drawn from  $\mathcal{D}_i$ . With a model trained on the  $n$  training tasks, our target is to adapt and evaluate it on  $t$  novel test tasks  $\tau_j' = (\mathcal{D}_j', S_j')$ ,  $1 \leq j \leq t$ .  $\mathcal{D}_j'$  is an unknown data distribution over  $\mathcal{X}$  and  $\mathcal{Y}_j'$ , and  $S_j'$  is a labeled dataset drawn from  $\mathcal{D}_j'$ . Note that, in few-shot learning, the labeled data for each category is very limited, and the categories in training tasks  $\{\tau_i\}_{i=1}^n$  will not appear in novel tasks  $\{\tau_j'\}_{j=1}^t$ , which means  $\mathcal{Y}_i \cap \mathcal{Y}_j' = \emptyset, \forall i \in \{1, ..., n\}, j \in \{1, ..., t\}$ .
4 Generalization Analysis of Attribute-based Few-shot Learning
The key to answering the two questions we raised above lies in a proper metric that quantifies task relatedness and contributes to the generalization error on a novel task. In this section, we begin by introducing the Task Attribute Distance (TAD), which first measures the category relationship through attribute conditional distributions, based on which the relationship between tasks is measured as well. Then, to facilitate our theoretical analysis, we introduce a meta-learning framework with attribute learning. Finally, we provide theoretical proof that establishes a connection between the proposed TAD metric and the generalization error bound on a novel task.
4.1 Model-Agnostic Distance Metric via Attributes
To measure the relationship between two categories  $y_{k}$  and  $y_{t}$ , a natural idea is to measure the divergence between class-conditional distributions  $p(x|y_k)$  and  $p(x|y_t)$  by a divergence measure, like the  $L_{1}$  or total variation distance (TV) [18]. Unfortunately, this idea is frequently unfeasible, because the set of measurable subsets under distributions  $p(x|y_k)$  and  $p(x|y_t)$  is often infinite, making it hard to compute the distance from finite samples.
In this paper, we introduce attributes to overcome the above limitation because the values of attributes are usually discrete and finite. The basic idea is based on the attribute compositionality hypothesis presented in the introduction section. Specifically, the distance between two categories can be measured by the difference between the corresponding attribute conditional distributions. Follow the above idea, let  $\mathcal{A}$  be the attribute space spanned by  $L$  attribute variables  $\{a^l\}_{l = 1}^L$ , we define the distance between two categories  $y_{k}$  and  $y_{t}$  on  $L$  attribute variables as:
$$
d \left(y _ {k}, y _ {t}\right) = \frac {1}{L} \sum_ {l = 1} ^ {L} d _ {T V} \left(p \left(a ^ {l} \mid y _ {k}\right), p \left(a ^ {l} \mid y _ {t}\right)\right) = \frac {1}{L} \sum_ {l = 1} ^ {L} \sup  _ {B \in B ^ {l}} \left| P _ {a ^ {l} \mid y _ {k}} [ B ] - P _ {a ^ {l} \mid y _ {t}} [ B ] \right|, \tag {1}
$$
where  $\mathcal{B}^l$  is the set of measurable subsets under attribute conditional distributions  $p(a^l |y_k)$  and  $p(a^{l}|y_{t})$ , and  $\mathcal{B}^l$  is a finite set if the values of variable  $a^l$  are finite.
Based on the distance between categories, we then define the distance metric between training and novel test tasks. To this end, we represent the set of categories in training task  $\tau_{i}$  and the set of categories in novel task  $\tau_{j}^{\prime}$  as two disjoint vertex sets respectively, and construct a weighted bipartite graph  $G$  between the two vertex sets, where each node denotes a category and each edge weight represents the distance between two categories in  $\tau_{i},\tau_{j}^{\prime}$  respectively. Let  $M = \{e_{kt}\}$  denote a maximum matching of  $G$ , which contains the largest possible number of edges and any two edges do not connect the same vertex. We choose  $M$  with minimum weights and define the task distance as
$$
d \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right) = \frac {1}{| M |} \sum_ {e _ {k t} \in M} d \left(y _ {k}, y _ {t}\right) = \frac {1}{L | M |} \sum_ {e _ {k t} \in M} \sum_ {l = 1} ^ {L} d _ {T V} \left(p \left(a ^ {l} \mid y _ {k}\right), p \left(a ^ {l} \mid y _ {t}\right)\right), \tag {2}
$$
where  $|M|$  is the number of edges in matching  $M$ . From this definition, if the two tasks  $\tau_i, \tau_j'$  are identical, the task distance  $d(\tau_i, \tau_j')$  is equal to 0. We call the task distance  $d(\tau_i, \tau_j')$  as the Task Attribute Distance (TAD). Additionally, the TAD is invariant to the permutation of categories in tasks. In other words, modifying the numeric order of categories in  $\tau_i$  or  $\tau_j'$  does not affect their distance  $d(\tau_i, \tau_j')$ . To emphasize the attribute conditional distributions with respect to different tasks, we hereinafter add task indexes on the distribution and probability notations, like  $p^i(a^l | y_k)$  and  $P_{a^l | y_k}^i[B]$ , although the true distributions are task agnostic.
4.2 A Specific Few-Shot Learning Framework
To facilitate the subsequent theoretical analysis, we consider a meta-learning framework with attribute learning. In this framework, a meta-learner  $f_{\theta} : \mathcal{X} \to \mathcal{A}$  parameterized by  $\theta$  learns the mapping from a sample  $x \in \mathcal{X}$  to attributes  $a \in \mathcal{A}$ , and a task-specific base-learner  $g_{\phi_i} : \mathcal{A} \to \mathcal{Y}_i$  parameterized by  $\phi_i$  learns the mapping from attributes  $a \in \mathcal{A}$  to a class label  $y \in \mathcal{Y}_i$  for training task  $\tau_i$ . During training,  $f_{\theta}$  is meta-learned from  $n$  different training tasks. To adapt to a novel task  $\tau_j'$ , we fix  $f_{\theta}$  and train a new base-learner  $g_{\phi_j'}$  parameterized by  $\phi_j'$  using few labeled samples  $S_j'$ .
In the following theoretical deduction, we will denote  $p_{\theta}(a,y) \triangleq p(f_{\theta}(x),y)$  as a joint distribution over  $\mathcal{A} \times \mathcal{V}$  induced by the meta-learned mapping  $f_{\theta}$ . Further, with the induced conditional distribution  $p_{\theta}(a|y)$ , we can compute the distance between task  $\tau_{i}$  and  $\tau_{j}'$  as
$$
d _ {\theta} \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right) = \frac {1}{L | M |} \sum_ {e _ {k t} \in M} \sum_ {l = 1} ^ {L} d _ {T V} \left(p _ {\theta} ^ {i} \left(a ^ {l} \mid y _ {k}\right), p _ {\theta} ^ {j} \left(a ^ {l} \mid y _ {t}\right)\right). \tag {3}
$$
Note that  $d(\tau_i, \tau_j')$  (as defined in Eq. (2)) computed based on  $p(a|y)$  is the model-agnostic distance metric, while  $d_{\theta}(\tau_i, \tau_j')$  is a model-related one since it relies on the meta-learned mapping  $f_{\theta}$ .
4.3 Theoretical Analysis on Generalization
As for the above meta-learning framework, we provide theoretical analysis of the generalization error on a novel task, in terms of the proposed TAD metric. We define the generalization error and empirical error of the meta-learner  $f_{\theta}$  on novel task  $\tau_j'$  as
$$
\epsilon (f _ {\theta}, \tau_ {j} ^ {\prime}) = \mathbb {E} _ {(x, y) \sim \mathcal {D} _ {j} ^ {\prime}} [ \mathbb {I} (g _ {\phi_ {j} ^ {\prime}} (f _ {\theta} (x)) \neq y) ], \quad \hat {\epsilon} (f _ {\theta}, \tau_ {j} ^ {\prime}) = \frac {1}{m _ {j} ^ {\prime}} \sum_ {k = 1} ^ {m _ {j} ^ {\prime}} \mathbb {I} (g _ {\phi_ {j} ^ {\prime}} (f _ {\theta} (x _ {k})) \neq y _ {k}),
$$
where  $\phi_j^\prime$  is the parameters of  $g_{\phi}$  learned from labeled samples  $S_j^\prime$  given  $f_{\theta}$ , and  $m_j^\prime = |S_j^\prime|$  is the number of labeled samples in  $S_j^\prime$ .  $\mathbb{I}$  denotes the indicator function. Similarly, we can define the generalization error  $\epsilon (f_{\theta},\tau_i)$  and empirical error  $\hat{\epsilon} (f_{\theta},\tau_{i})$  for training task  $\tau_{i}$ . With these definitions, we will proceed by introducing an auxiliary lemma, and then stating our main theoretical results. All detailed proofs of the following lemma and theorems are in the Appendix.
Lemma 1. Let  $\mathcal{A}$  be the attribute space,  $L$  be the number of attributes. Assume all attributes are independent of each other given the class label, i.e.  $p(a|y) = \prod_{l=1}^{L} p(a^l | y)$ . For all  $a_i \in \mathcal{A}$  and any two categories  $y_k, y_t$ , the following inequality holds:
$$
\sum_ {a _ {i} \in \mathcal {A}} | p (a _ {i} | y _ {k}) - p (a _ {i} | y _ {t}) | \leq d (y _ {k}, y _ {t}) + \Delta , \tag {4}
$$
where  $d(y_{k},y_{t})$  is the distance as defined in Eq. (1) and  $\Delta = \sum_{a_i\in \mathcal{A}}\frac{1}{2L}\sum_{l = 1}^{L}(p(a_i^l |y_k) + p(a_i^l |y_t))$
Lemma 1 says that under the attribute conditional independence assumption, the distance between two attribute conditional distributions  $p(a|y_k)$ ,  $p(a|y_t)$  over attribute space  $\mathcal{A}$  is bounded by the defined distance  $d(y_k, y_t)$  and a non-negative term  $\Delta$ . This result enables us to make use of the defined TAD metric in deriving the generalization error bound, leading to the following theoretical results.
Theorem 1. With the same notation and assumptions as in Lemma 1, let  $\mathcal{H}$  be the hypothesis space with VC-dimension  $d$ ,  $f_{\theta}$  and  $g_{\phi}$  be the meta-learner and the base-learner as introduced in Sec. 4.2 respectively. Denote  $g_{\phi^*}$  as the best base-learner on some specific tasks given a fixed meta-learner  $f_{\theta}$ . For any single training task  $\tau_i = (\mathcal{D}_i, S_i)$  and a novel task  $\tau_j' = (\mathcal{D}_j', S_j')$ , suppose the number of categories in the two tasks is the same, then with probability at least  $1 - \delta$ ,  $\forall g_{\phi} \circ f_{\theta} \in \mathcal{H}$ , we have
$$
\epsilon \left(f _ {\theta}, \tau_ {j} ^ {\prime}\right) \leq \hat {\epsilon} \left(f _ {\theta}, \tau_ {i}\right) + \sqrt {\frac {4}{m _ {i}} \left(d \log \frac {2 e m _ {i}}{d} + \log \frac {4}{\delta}\right)} + d _ {\theta} \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right) + \Delta^ {\prime} + \lambda , \tag {5}
$$
where  $\lambda = \lambda_{i} + \lambda_{j}^{\prime}$  is the generalization error of  $g_{\phi^{*}}$  and  $f_{\theta}$  on the two tasks, i.e.,  $\lambda_{i} = \mathbb{E}_{(x,y)\sim \mathcal{D}_{i}}[\mathbb{I}(g_{\phi_{i}^{*}}(f_{\theta}(x))\neq y)],\lambda_{j}^{\prime} = \mathbb{E}_{(x,y)\sim \mathcal{D}_{j}^{\prime}}[\mathbb{I}(g_{\phi_{j}^{\prime *}}(f_{\theta}(x))\neq y)].$ $\Delta^\prime$  is a term depending on learned base-learners  $g_{\phi_i},g_{\phi_j'}$  and the best base-learners  $g_{\phi_i^*},g_{\phi_j'^*}$
Theoretically, the generalization error on a novel task is bounded by the training task empirical error  $\hat{\epsilon}(f_{\theta}, \tau_i)$  plus four terms: the second term  $d_{\theta}(\tau_i, \tau_j')$  is the model-related distance between  $\tau_i$  and  $\tau_j'$ , which is derived based on Lemma 1; the third term  $\Delta'$  reflects the classification ability of  $g_{\phi}$ , which converges to zero if the learned base-learners are equal to the best ones for both tasks; the last term  $\lambda$  is the generalization error of  $f_{\theta}$  and  $g_{\phi^*}$ , which depends on the attribute discrimination ability of  $f_{\theta}$  and the hypothesis space of base-learner  $g_{\phi}$ . For a reasonable hypothesis space, if  $f_{\theta}$  has a good attribute discrimination ability on both tasks, the last term usually converges to zero. Next we generalize this bound to the setting of  $n$  training tasks.
Corollary 1. With the same notation and assumptions as Theorem 1, for  $n$  training tasks  $\{\tau_i\}_{i=1}^n$  and a novel task  $\tau_j'$ , define  $\hat{\epsilon}(f_\theta, \tau_{i=1}^n) = \frac{1}{n} \sum_{i=1}^n \hat{\epsilon}(f_\theta, \tau_i)$ , then with probability at least  $1 - \delta$ ,  $\forall g_\phi \circ f_\theta \in \mathcal{H}$ , we have
$$
\epsilon \left(f _ {\theta}, \tau_ {j} ^ {\prime}\right) \leq \hat {\epsilon} \left(f _ {\theta}, \tau_ {i = 1} ^ {n}\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} \sqrt {\frac {4}{m _ {i}} \left(d \log \frac {2 e m _ {i}}{d} + \log \frac {4}{\delta}\right)} + \frac {1}{n} \sum_ {i = 1} ^ {n} d _ {\theta} \left(\tau_ {i}, \tau_ {j} ^ {\prime}\right) + \Delta^ {\prime} + \lambda , \tag {6}
$$
where  $\lambda = \frac{1}{n}\sum_{i=1}^{n}\lambda_i + \lambda_j'$ , and  $\Delta'$  is a term depending on the learned base-learners  $\{g_{\phi_i}\}_{i=1}^n$ ,  $g_{\phi_j'}$  and the best base-learners  $\{g_{\phi_i^*}\}_{i=1}^n$ ,  $g_{\phi_j'^*}$ .