SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenxuan Ma
Beijing Institute of Technology wenxuanma@bit.edu.cn
Lincan Cai
Beijing Institute of Technology lincancai@bit.edu.cn
Jingxuan Kang
University of Liverpool sgjkang3@liverpool.ac.uk
Developing generalizable models that can effectively learn from limited data and with minimal reliance on human supervision is a significant objective within the machine learning community, particularly in the era of deep neural networks. Therefore, to achieve data-efficient learning, researchers typically explore approaches that can leverage more related or unlabeled data without necessitating additional manual labeling efforts, such as Semi-Supervised Learning (SSL), Transfer Learning (TL), and Data Augmentation (DA). SSL leverages unlabeled data in the training process, while TL enables the transfer of expertise from related data distributions. DA broadens the dataset by synthesizing new data from existing examples. However, the significance of additional knowledge contained within labels has been largely overlooked in research. In this paper, we propose a novel perspective on data efficiency that involves exploiting the semantic information contained in the labels of the available data. Specifically, we introduce a Language Semantic Graph (LSG) which is constructed from labels manifest as natural language descriptions. Upon this graph, an auxiliary graph neural network is trained to extract high-level semantic relations and then used to guide the training of the primary model, enabling more adequate utilization of label knowledge. Across image, video, and audio modalities, we utilize the LSG method in both TL and SSL scenarios and illustrate its versatility in significantly enhancing performance compared to other data-efficient learning approaches. Additionally, our in-depth analysis shows that the LSG method also expedites the training process.
Deep learning has achieved remarkable breakthroughs in various domains, including speech recognition, visual object recognition, object detection, drug discovery, genomics, and many others [33]. However, the increasing size of deep neural networks has led to a growing demand for large volumes of data. Unfortunately, not all application domains have access to extensive datasets due to the high costs associated with data collection and human annotation [38, 8, 16]. Consequently, substantial efforts have been dedicated to studying methods that enable data-efficient learning [1, 49], aiming to alleviate the requirement of extensive training data and, in particular, human supervision.
The current primary approaches to achieving data-efficient learning encompass semi-supervised learning (SSL) [62], transfer learning (TL) [49, 86], and other techniques such as data augmentation
$\text{忍}$  Corresponding author. Code available at: https://github.com/BIT-DA/LSG
(DA) [78, 9]. Semi-supervised learning [19, 34, 58, 4, 69] involves using both labeled and unlabeled data to reduce manual labeling effort and enhance model performance. By exploring vast amounts of unlabeled samples, artificial learners acquire a more comprehensive understanding of the underlying data structure and achieve improved predictions, even in scenarios where labeled samples are scarce [70]. Transfer learning leverages data from other related source domains typically offer abundant data and annotations [42, 14, 2], or it utilizes models pretrained on large-scale datasets to transfer general knowledge to minimize the data requirements in the target domain [76, 74]. Data augmentation technique creates novel examples from existing ones, thereby expanding and diversifying the training dataset. From this perspective, these data-efficient learning approaches share a common objective: to leverage additional training data without necessitating extra manual labeling efforts. In this paper, we approach data-efficient learning from a different angle: by investigating improved methods to harness the available data-label pairs. This novel perspective motivates the core objective of our work: to fully exploit semantic information contained in the labels to improve model performance with data efficiency.
In particular, it is worth noting that original human annotations typically manifest as natural language descriptions of the training data, such as "bicycle" and "kangaroo" in object recognition tasks or "jogging" and "playing piano" in action recognition. We refer to these descriptive labels with descriptions as the labeled concepts. The prevailing approach of converting these labeled concepts into indices simplifies the machine learning problem but sacrifices semantic information. Consequently, our objective is to enhance data utilization by reintegrating this information into the training process. Our idea involves introducing an auxiliary branch during training to facilitate high-level interactions between data representations and label semantics. This interaction guides the model in extracting appropriate features that capture accurate semantics. Notably, the auxiliary branch is discarded after the training process, thus avoiding an increase in inference cost. However, the question that remains is how to achieve such interaction effectively.
To this end, we propose employing graphs to model high-level semantics of the labeled concepts, owing to their generality and flexibility [72]. The initial node features within the graph comprise language embeddings of these concepts, which are generated by pretrained language models. The edges of the graph capture strong semantic connections between the concepts. We refer to this graph as the Language Semantic Graph (LSG) and train an auxiliary graph neural network to extract the knowledge within the graph and transfer it to the primary neural network model. During formal training, the data representations produced by the primary model are incorporated into the Language Semantic Graph as new nodes, based on their true labels. Consequently, the graph neural network can propagate information across all nodes in the augmented graph. Building upon this interaction mechanism, we propose two additional optimization objectives to implicitly align and explicitly regularize the data representations, ensuring their adherence to the correct semantics.
LSG demonstrates applicability across models in various modalities and exhibits improved data-efficient learning performance. To validate its effectiveness, we conduct experiments on two typical data-efficient scenarios: transfer learning and semi-supervised learning, and apply it to image, video, and audio datasets. The experimental results demonstrate that LSG significantly enhances performance and outperforms other data-efficient approaches. Our contribution includes:
- We propose a novel perspective towards data-efficient learning via enhancing the utilization efficiency on the available annotated data. Particularly, we introduce a Language Semantic Graph (LSG) to model the high-level label semantics and provide extra supervision.
- We design a comprehensive method with an auxiliary graph neural network that extracts knowledge from LSG and two novel objectives that guide the primary model to learn efficiently. Furthermore, we extend the method by incorporating semi-supervised learning to leverage unlabeled samples for further enhancement.
- Experiments on both transfer learning and semi-supervised learning scenarios are conducted crossing image, video and audio modalities. The superior performance achieved by our proposed method, along with an in-depth analysis, confirms its effectiveness.
Data-efficient learning [1] investigates the problem of data hungriness of modern machine learning algorithms, and proposes algorithms to reduce the reliance on extensive training data. Two
prominent approaches in the roadmap of data-efficient learning are Transfer Learning (TL) and Semi-Supervised Learning (SSL). TL focuses on transferring knowledge from source domains to minimize the number of labeled examples required in a target domain [86]. Within this research area, Domain Adaptation [14, 55, 82, 35] investigates transductive transfer learning where source and target domain possess different data distributions but share the same task. Fine-tuning have emerged with deep neural networks [12, 48, 76, 18], enabling the flexible transfer of knowledge from large pretrained models to benefit various downstream tasks [56, 74, 41, 23, 61]. In conclusion, TL generally leverages data from elsewhere to enhance data efficiency in the current task.
On the other hand, SSL takes a different approach by harnessing large-scale unlabeled data to reduce the need for manual labeling. Prevalent SSL methods [34, 58, 60, 3, 24] explore unlabeled samples to capture the intrinsic data structural information [70]. Notably, a widely adopted strategy in SSL is the assignment of pseudo-labels to samples [34], enabling models to improve in a self-training manner [53, 5, 75]. The combination of TL and SSL approaches has also been embraced to achieve more promising results [54, 67]. In contrast, our work presents a novel perspective within data-efficient learning. Instead of exploring data from various sources, we aim to exploit annotated data more comprehensively. We identify semantic relations between labels and utilize them to guide the model in learning better representations.
Graph neural network as a learning component. Graph neural networks (GNN) [28, 17, 63, 64] have gained significant popularity not only for their standard applications in graph-related tasks such as node classification and link prediction but also as a component within learning systems to process graph-structured inputs and propagate crucial information [15, 71, 50, 45]. In the domain of computer vision, Garcia and Bruna [15] leverage GNN for few-shot learning, NGM [20] generates graph with continuously updated adjacency matrix for 3D action recognition, BGRNet [71] includes GNN in their framework for panoptic segmentation, and Knowledge-CLIP [50] learns multi-modal representations. Furthermore, GNNs have found applications in natural language processing [45], recommendation systems [46] and etc. In our method, a GNN is trained on the semantic graph constructed from labels to guide the learning process of the primary model.
Language models for non-language tasks. The beneficial interaction between modalities, justified by the success of multimodal learning [51, 79, 85, 57], has sparked growing interest in borrowing knowledge from other modalities. LIFT [11] formalizes non-language inputs into sentences and adopts GPT [52] as general interface. Vision-Language model-based finetuning methods [84, 83, 43, 81, 80] leverage the jointly pretrained text encoders to construct robust classifiers for downstream vision tasks. Our method draws inspiration from these works but differs in its objective as well as the way to leverage linguistic knowledge. Concretely, LSG promotes data-efficient learning on independent non-language models that are not restricted to multi-modal model. Furthermore, instead of building a robust classification head, the linguistic knowledge in our method is leveraged to construct a graph that models label semantic relations then guide the non-language model learn more semantically meaningful representations.
Training a deep neural network to achieve satisfying performance could be challenging in many real-world applications due to the lack of sufficient label supervision. Our approach seeks to improve the utilization of labeled data that is currently available for enhancing data efficiency. Formally, we consider training a model on a task-related labeled dataset  $\{x_i, y_i\}_{i=1}^n$ , where  $x$  could be images, video clips or audio clips etc. for different tasks and  $y$  is the corresponding category label that is represented using natural language description such as a word or a phrase. The model (also referred as the primary model in the following content) can be regarded as a feature encoder  $F(\cdot|\theta)$  that maps the input data to embedding space and follows by a task-specific head  $C(\cdot|\phi)$  that turns feature embeddings into predictions, with  $\theta$  and  $\phi$  as parameters. The standard training objective utilizing the labeled dataset is the minimization of the empirical risk. By omitting the network parameters and denote  $C \circ F(\pmb{x}) = C(F(\pmb{x}))$ , the empirical risk minimization can be formulated as:
$$
\min  _ {\theta , \phi} \quad \mathcal {L} _ {e m p} = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(C \circ F \left(\boldsymbol {x} _ {i}\right), y _ {i}\right), \tag {1}
$$
where  $\ell (\cdot ,\cdot)$  typically be cross-entropy loss, and labels are converted to indices during computation.
Figure 1: Illustration of the LSG training framework. In the first stage, labeled concepts are turned into text embeddings via pretrained language model (PLM), and then the Language Semantic Graph (LSG) is constructed according to the embeddings. We train a GNN on the graph to extract semantic knowledge. In the second or the formal training stage, the data representations  $h$  are connected to the graph based on their labels (and the new graph topology is described in the augmented adjacency matrix). Representations are then transformed into  $h'$  by the GNN. The proposed alignment loss and regularization loss is applied on these features to guide the primary model.
We argue that such conversion of labels inevitably losses information and results in data inefficiency. Therefore, our method introduces two additional losses  $\mathcal{L}_{\text{align}}$  and  $\mathcal{L}_r$  that exploit the discarded label semantic information for complement.
This section first introduces the construction process of the Language Semantic Graph (LSG) and the training process of the auxiliary model, which are conducted before the primary training stage to capture the semantic information. Then it moves on to explain the objectives  $\mathcal{L}_{\text{align}}$  and  $\mathcal{L}_r$  guide the data-efficient training of the primary model through LSG.
3.1 Language Semantic Graph
LSG aims to model the semantic relationship between labeled concepts with high generality and flexibility. We propose a straightforward approach to construct the graph, leveraging the text embedding space that contains rich semantics. Specifically, given the natural language descriptions of the concepts  $\{\mathcal{W}_k\}_{k=1}^K$  where  $K$  is the total number of concepts or categories, we use a set of  $m$  predetermined prompts (e.g., 'This is a photo of a {concept}') to contextualize the concepts. We then obtain a set of text embeddings  $T$  by inputting contextualized concepts into a frozen pretrained language model such as BERT [25], where  $T = \cup_{k=1}^{K} \{t_k^{(1)}, t_k^{(2)}, \dots, t_k^{(m)}\} \subseteq \mathbb{R}^{d_t}$ ,  $|T| = mK$ , with  $d_t$  being the dimension of the text embedding space. Then we define the text embedding matrices  $\mathcal{K}$  and  $\tilde{\mathcal{K}}$ , such that  $\mathcal{K} = [t_1^{(1)}, \dots, t_1^{(m)}, t_2^{(1)}, \dots, t_K^{(m)}] \in \mathbb{R}^{d_t \times |T|}$  and  $\tilde{\mathcal{K}}$  be its column normalized matrix. We compute the cosine similarity matrix between text embeddings as  $S = \tilde{\mathcal{K}}^\top \tilde{\mathcal{K}}$ . Additionally, we define a text label vector  $\boldsymbol{y}_t = \begin{bmatrix} 1, \dots, 1, & \dots, & K, & \dots, & K \\ m & & m & & m \end{bmatrix}$  to indicate the category for each text embedding.
Next, we define the language semantic graph adjacency matrix based on text embedding similarities:
$$
\boldsymbol {A} = \left[ a _ {i j} \right] _ {1 \leq i \leq | T |, 1 \leq j \leq | T |} = \left\{ \begin{array}{c l} \max  \left(\boldsymbol {S} _ {i, j}, 0\right) & \text {i f} y _ {t, i} = y _ {t, j}, \\ \boldsymbol {S} _ {i, j} & \text {i f} y _ {t, i} \neq y _ {t, j} \text {a n d} \boldsymbol {S} _ {i, j} \geq \tau , \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
Note that  $\mathbf{A}$  is a weighted adjacency matrix. The first line means that we connect nodes representing the same concept with their similarity scores as weight. The second line allows certain edge connections between nodes that represent different concepts but share high feature similarities, which is controlled by an edge threshold hyper-parameter  $\tau$ . Finally, the language semantic graph we defined can be represented as:  $\mathcal{G} = (\mathcal{V}, \mathcal{A}, \mathcal{K})$ , where  $\mathcal{V}$  is the node set,  $|\mathcal{V}| = |T|$ , and  $\mathcal{K}$  is used as the node feature matrix.
To fully leverage the LSG to guide the data-efficient training of the primary model, we propose to train an auxiliary graph neural network to extract semantic information from LSG. We adopt the
standard GCN [28] architecture, which consists of several consecutive graph convolution layers. It is expected that GCN, with its strength in massage passing, first refines the initial node embeddings (or text embeddings) by leveraging graph topology and then serves as a bridge connecting the primary model and the LSG. Let  $D$  be the degree matrix of  $A$ ,  $\sigma(\cdot)$  be the ReLU [47] activation function,  $W^{(l)}$  be the learnable weight matrix for the  $l_{\mathrm{th}}$  layer and  $\mathcal{K}^{(0)} = \mathcal{K}$ , the update function of the node features can be formulated as:
$$
\boldsymbol {\mathcal {K}} ^ {(l)} = \sigma \left(\boldsymbol {D} ^ {- \frac {1}{2}} \boldsymbol {A} \boldsymbol {D} ^ {- \frac {1}{2}} \boldsymbol {\mathcal {K}} ^ {(l - 1)} \boldsymbol {W} ^ {(l)}\right). \tag {3}
$$
Analogues to the primary model, we regard our GCN model as the composition of node feature encoder  $\mathcal{F}_g$  and node classifier  $\mathcal{C}_g$ . Particularly, the encoder  $\mathcal{F}_g$  is designed to output node features that have the same dimension as input features for the later computation of  $\mathcal{L}_r$ . The whole network is trained using the node classification objective that aims to correctly predict category for each node:
$$
\mathcal {L} _ {\text {n o d e}} = - \sum_ {i = 1} ^ {| T |} \sum_ {k = 1} ^ {K} \mathbb {1} _ {[ k = y _ {t, i} ]} \log \delta \left(\mathcal {C} _ {g} \circ \mathcal {F} _ {g} (\mathcal {K} _ {., i})\right) _ {k}, \tag {4}
$$
where  $\delta(\cdot)$  is the softmax function. To this end, we obtain the LSG and the auxiliary model trained on it, which contain semantic information from the labeled concepts and will be leveraged to boost the data-efficient training of the primary model.
Remark:  $\mathcal{L}_{node}$  is only used for training GCN in this first stage. By doing this, each node processed by the GCN aggregates discriminative and more comprehensive semantic information from neighboring nodes. Also note that both the LSG and the GCN only depend on the categories of the current task. Hence, they can be constructed and trained on each task once and for all.
3.2 LSG Guided Data-Efficient Learning
To enable language semantic guidance, we add a new projector  $H(\cdot|\psi)$  on top of the feature encoder  $F$  of the primary model to project the data representation to the same dimension as the text embeddings. We denote  $\pmb{h}_i = H(F(\pmb{x}_i)) \in \mathbb{R}^{d_t}$  as the projected representation of  $\pmb{x}_i$  and  $\tilde{\pmb{h}}$  as its embedding after  $l_2$  normalization. We would like to enable interactions between the data representations and the node features within the semantic graph. Henceforth, we propose to connect features generated by the primary model into LSG according to the label  $y$  of the data.
During training process, for every minibatch of  $B$  images  $\{\pmb{x}_j,y_j\}_{j = 1}^B$ , we obtain their representations  $\{h_i\}_{i = 1}^B$  and construct an augmented graph of the original Language Semantic Graph, denoted as  $\mathcal{G}_{aug} = (\mathcal{V}_a,A_a,\mathcal{K}_a)$ . In this augmented graph,  $|\mathcal{V}_a| = |T| + B$ , the augmented node feature matrix  $\mathcal{K}_a = [\mathcal{K},h_1,\dots,h_B]\in \mathbb{R}^{d_t\times (|T| + B)}$  is the node feature matrix plus  $B$  data features, and the augmented adjacency matrix is defined as  $A_{a} = \left[ \begin{array}{cc}A & P\\ P^{\top} & M \end{array} \right]$ , where  $P\in \mathbb{R}^{|T|\times B}$  is a binary matrix indicating the connection between label nodes and data nodes,  $P = [p_{ij}] = \left\{ \begin{array}{ll}1 & \text{if } y_{t,i} = y_j,\\ 0 & \text{otherwise,} \end{array} \right.$  and  $M\in \mathbb{R}^{B\times B}$  is a symmetric binary matrix indicating the connection among data nodes,  $M = [m_{ij}] = \left\{ \begin{array}{ll}1 & \text{if } y_i = y_j,\\ 0 & \text{otherwise.} \end{array} \right.$  Based on the true label of training data, these edges build connection between the primary model and the LSG, which allows the graph neural network to propagate information and transfer knowledge that is embedded inside the labels.
With the augmented graph, we conduct language semantic guidance to the primary model through optimizing two objectives:  $\mathcal{L}_{\text {align }}$  that implicitly aligns the data representation to the semantic graph and  $\mathcal{L}_r$  that explicitly regularize the model to extract more semantically meaningful representations. We utilize the frozen auxiliary model to encode new data nodes in the augmented graph and predict their categories. Given that the trained auxiliary model can correctly predict the label nodes in the original graph and its weights are no longer updated, the data representations  $h_i$  are forced to become similar to their corresponding label node features during optimization, achieving the alignment between features from different nodes. The implicit alignment objective is as follows:
$$
\min  _ {\theta , \psi} \quad \mathcal {L} _ {\text {a l i g n}} = - \sum_ {i = 1} ^ {B} \sum_ {k = 1} ^ {K} \mathbb {1} _ {[ k = y _ {i} ]} \log \delta \left(\mathcal {C} _ {g} \circ \mathcal {F} _ {g} (\boldsymbol {h} _ {i})\right) _ {k}. \tag {5}
$$
Besides the implicit alignment loss that guides the data representations to have correct semantic relations, we further propose an explicit regularization loss to transfer the knowledge embedded
in the label nodes to the primary model. Notice that after being processed by the trained GCN, the output features of the data nodes have already aggregated information from their label node neighbors on the augmented graph, which are desirable representations that contain information from both the data samples and the semantic graph. Thus, we propose the graph semantic feature regularization loss  $\mathcal{L}_r$  that directly uses the output features of the auxiliary model encoder  $\mathcal{F}_g$  to promote their initial input features, which is why both features are designed to have the same dimension. Specifically, we denote  $\mathcal{F}_g(\pmb {h}_i)$  as the output graph feature for  $\pmb{h}_i$ . Then we normalize both features  $\mathcal{F}(\tilde{\pmb{h}}_i) = \mathcal{F}_g(\pmb {h}_i) / ||\mathcal{F}_g(\pmb {h}_i)||_2$ ,  $\tilde{h}_i = h_i / ||h_i||_2$  and use the normalized graph feature as the target for normalized input feature. We have the following regularization objective:
$$
\min  _ {\theta , \psi} \quad \mathcal {L} _ {r} = - \sum_ {i = 1} ^ {B} \left| \left| \operatorname {s g} \left(\mathcal {F} _ {g} \left(\tilde {\boldsymbol {h}} _ {\boldsymbol {i}}\right)\right) - \tilde {\boldsymbol {h}} _ {\boldsymbol {i}} \right| \right| _ {2} ^ {2}, \tag {6}
$$
where  $\mathrm{sg}(\cdot)$  denotes stop gradient. By explicitly regularizing the original data representation to include more information embedded in the graph as Eq. (6), the primary model learns more informative features from the LSG and thus allows this auxiliary branch to be discarded during inference.