Language Semantic Graph Guided Data-Efficient Learning
Authors: Wenxuan Ma
Shuang Li
Lincan Cai
Jingxuan Kang
Venue: NeurIPS 2023 poster (2023)
Keywords: Data-Efficient Learning
Language Semantic Graph
TL;DR: A novel data-efficient learning method aiming to exploit label semantic information.

Abstract: Developing generalizable models that can effectively learn from limited data and with minimal reliance on human supervision is a significant objective within the machine learning community, particularly in the era of deep neural networks. Therefore, to achieve data-efficient learning, researchers typically explore approaches that can leverage more related or unlabeled data without necessitating additional manual labeling efforts, such as Semi-Supervised Learning (SSL), Transfer Learning (TL), and Data Augmentation (DA).
SSL leverages unlabeled data in the training process, while TL enables the transfer of expertise from related data distributions. DA broadens the dataset by synthesizing new data from existing examples. However, the significance of additional knowledge contained within labels has been largely overlooked in research. In this paper, we propose a novel perspective on data efficiency that involves exploiting the semantic information contained in the labels of the available data. Specifically, we introduce a Language Semantic Graph (LSG) which is constructed from labels manifest as natural language descriptions. Upon this graph, an auxiliary graph neural network is trained to extract high-level semantic relations and then used to guide the training of the primary model, enabling more adequate utilization of label knowledge. Across image, video, and audio modalities, we utilize the LSG method in both TL and SSL scenarios and illustrate its versatility in significantly enhancing performance compared to other data-efficient learning approaches. Additionally, our in-depth analysis shows that the LSG method also expedites the training process.

Language Semantic Graph Guided Data-Efficient Learning
Wenxuan Ma
Shuang Li
Beijing Institute of Technology venxuanma@bit.. edu. cn
Beijing Institute of Technology shuangli@bit. edu. cn
Lincan Cai
Jingxuan Kang
Beijing Institute of Technology lincancai@bit. edu. cn
University of Liverpool egjkang3elvverpol.. ac. uk
Abstract
Developing generalizable models that can effectively learn from limited data and with minimal reliance on human supervision is significant objective within the machine learning community, particularly in the era of deep neural networks. Therefore, to achieve data-efficiee learning, rescarchers typically explore ap- proaches that can leverage more related or unlabeled data without necessitating additional manual labeling efforts, such Semi-Supervised Learning (SSL), Trans- fer Learning (TL), and Data Augmentation (DA). SSL leverages unlabeled data the training process, while TL enables the transfer of expertise from related data distributions. DA broadens the dataset by synthesizing new data from existing examples. However, the significance of additional knowledge contained within labels has been largely overlooked in research. In this paper, we propose a novel perspective on data efficiency that involves exploiting the semantic information contained in the labels of the available data. Specifically, we introduce a Language Semantic Graph (LSG) which constructed from labels manifest natural lan- guage descriptions. Upon this graph, an auxiliary graph neural network is trained extract high-level semantic relations and then used guide the training of the primary model, enabling more adequate utilization of label knowledge. Across image, video, and audio modalities, we utilize the LSG method in both TL and SSL scenarios and illustrate its versatility in significantly enhancing performance compared to other data-efficient learning approaches. Additionally, our in-depth analysis shows that the LSSG method also expedites the training process.
Introduction
Deep learning has achieved remarkable breakthroughs in various domains, including speech recog- nition, visual object recognition, object detection, drug discovery, genomics, and many others (33]. However, the increasing size of deep neural networks has led to growing demand for large volumes of data. Unfortunately, not all application domains have access to extensive datasets due to the high
costs associated with data collection and human annotation BX g 16 Consequently, substantial efforts have been dedicated to studying methods that enable data-efficient learning [[] 49), aiming to alleviate the requirement of extensive training data and, in particular, human supervision.
The current primary approaches to achieving data-efficient learning encompass semi-supervised learning (SSL) [62]. transfer learning (TL. 44 86 and other techniques such data augmentation
Corresponding author. Code available https://ggtthub.com/I11-00
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(DA) 78 19. Semi Semi-superrvsed learning [19 34 58 4 69 involves using both labeled and unlabeled data to reduce manual labeling effort and enhance model performance. By exploring vast amounts of unlabeled samples, artificial learners acquire a more comprehensive understanding of the underlying data structure and achieve improved predictions, even in scenarios where labeled samples are scarce [70]. Transfer learning leverages data from other related source domains typically offer abundant data and annotations [42] [14 2], or it utilizes models pretrained on large-scale datasets to transfer general knowledge to minimize the data requirements in the target domain (76) 74]. Data augmentation technique creates novel examples from existing ones, thereby expanding and diversifying the training dataset. From this perspective, these data-efficient learning approaches share a common objective: to leverage additional training data without necessitating extra manual labeling efforts. In this paper. we approach data-efficient learning from different angle: by investigating improved methods to harness the available data-label pairs. This novel perspective motivates the core objective of our work: to fully exploit semantic information contained in the labels improve model performance with data efficiency.
In particular, it is worth noting that original human annotations typically manifest natural language descriptions of the training data, such as "bicycle" and "kangaroo" in object recognition tasks or "jogging" and "playing piano" in action recognition. We refer to these descriptive labels with descriptions as the labeled concepts. The prevailing approach of converting these labeled concepts into indices simplifies the machine learning problem but sacrifices semantic information. Consequently, our objective is to enhance data utilization by reintegrating this information into the training process. Our idea involves introducing auxiliary branch during training to facilitate high-level interactions between data representations and label semantics. This interaction guides the model in extracting appropriate features that capture accurate semanties. Notably, the auxiliary branch is discarded after the training process, thus avoiding an increase in inference cost. However, the question that remains how to achieve such interaction effectively.
To this end, we propose employing graphs to model high-level semantics of the labeled concepts, owing to their generality and flexibility (72). The initial node features within the graph comprise language embeddings of these concepts, which are generated by pretrained language models. The edges of the graph capture strong semantic comnections between the concepts. We refer to this graph as the Language Semantic Graph (LSG) and train an auxiliary graph neural network to extract the knowledge within the graph and transfer it the primary neural network model. During formal training, the data representations produced by the primary model are incorporated into the Language Semantic Graph as new nodes, based their true labels. Consequently, the graph neural network can propagate information across all nodes in the augmented graph. Building upon this interaction mechanism, we propose two additional optimization objectives to implicitly align and explicitly regularize the data representations, ensuring their adherence to the correct semantics.
LSG demonstrates applicability across models in various modalities and exhibits improved data- efficient learning performance. To validate its effectiveness, we conduct experiments on two typical data-efficient scenarios: transfer learning and semi-supervised learning. and apply it to image, video, and audio datasets. The experimental results demonstrate that LSG significantly enhances performance and outperforms other data-efficient approaches. Our contribution includes:
We propose novel perspective towards data-efficient learning via enhancing the utilization efficiency on the available annotated data. Particularly, we introduce Language Semantic Graph (LSG) to model the high-level label semantics and provide extra supervision.
We design comprehensive method with auxiliary graph neural network that extracts knowledge from LSG and two novel objectives that guide the primary model to learn efficiently. Furthermore, extend the method by incorporating semi-supervised learning to leverage unlabeled samples for further enhancement.
Experiments on both transfer learning and sumi-supervised learning scenarios are conducted, crossing image, video and audio modalities, The superior performance achieved by our proposed method, along with an in-depth analysis, confirms its effectiveness.
2 Related Work
Data-efficient learning || investigates the problem of data hungriness of modern machine learn- ing algorithms, and proposes algorithms to reduce the reliance on extensive training data. Two

---

prominent approaches in the roadmap of data-efficient learning are Transfer Learning (TL) and Semi-Supervised Learning (SSL). TL focuses on transferring knowledge from source domains to minimize the number of labeled examples required in target domain [86]. Within this research 82
investigates transductive transfer learning where source and target domain possess different data distributions but share the same task. Fine-tuning have emerged with deep neural networks [12
area, Domain Adaptation [4 55
[5
48 76 [8], enabling the flexible transfer of knowledge from large pretrained models to benefit various downstream tasks [56] 74, 41 23 61]. In conclusion, TL generally leverages data from elsewhere to enhance data efficiency in the current task.
On the other hand, SSL takes a different approach by harnessing scage unlabeled data to reduce the need for manual labeling. Prevalent SSL methods (34 558 60 24] explore unlabeled samples to capture the intrinsic data structural information (70]. Notably. a widely adopted strategy in SSL the assignment of pseudo-labels to samples [4], enabling models to improve in a self-training The combination of TL. and SSL approaches has also been embraced to achieve more promising results 54 67
manner 53 5 75
contrast, our work presents a novel perspective within data- efficient learning. Instead of exploring data from various sources, we aim to exploit annotated data more comprehensively. We identify semantic relations between labels and utilize them to guide the model in learning better representations.
Graph neural network as a learning component. Graph neural networks (GNN) 28
64
have gained significant popularity not only for their standard applications in graph-relate related tasks such as node classification and link prediction but also as a component within learning systems to process graph-structured inputs and propagate crucial information [15 7 50 45). In the domain of
computer vision, Gareia and Bruna [i5 leverage GNN for few-shot learning, NGM 20 generates graph with continuously updated adjacency matrix for 3D action recognition, BGRNet 71 includes
GNN in their framework for panoptic segmentation, and Knowledge-CLIP (50) learns multi-modal representations. Furthermore, GNNs have found applications in natural language processing (45), recommendation systems (46) and etc. In our method, a GNN trained on the semantic graph constructed from labels to guide the learning process of the primary model.
Language models for non-language tasks. The beneficial interaction between modalities, justified by the success of multimodal learning [5 79 85 57], has sparked growing interest in borrowing knowledge from other modalities. LIFT [1 formalizes non-language inputs into sentences and adopts 80)
GPT [52] as general interface. Vision-Language
model-based finetuning methods [4 83
leverage the jointly pretrained text encoders to construct robust classifiers for downstream vision tasks. Our method draws inspiration from these works but differs in its objective well as the way to leverage linguistic knowledge. Concretely, LSG promotes data-efficient learning on independent non-language models that are not restricted to multi-modal model. Furthermore, instead of building robust classification head, the linguistic knowledge in our method leveraged to construct graph that models label semantic relations then guide the non-language model learn more semantically meaningful representations.
3 Method
Training deep neural network to achieve satisfying performance could be challenging in many real-world applications due the lack of sufficient label supervision. Our approach seek to improve the utilization of labeled data that is currently available for enhancing data efficiency. Formally, we consider training a model on a task-related labeled dataset (x; yi > where a could be images, video clips or audio clips etc. for different tasks and is the corresponding category label that is represented using natural language description such as word or a phrase. The model (also referred as the primary model in the following content) can be regard as a feature encoder F (0) that maps the input data to embedding space and follows by a task-specific head C(-)Ø) that turns feature embeddings into predictions, with @ and á as parameters. The standard training objective utilizing the labeled dataset is the minimization of the empirical risk. By omitting the network parameters and denote C' F(x:) ((F())). the empirical risk minimization can be formulate as:
min 6,0
Lemp
Σ (lC o F().).).,
n
where E(.-) typically be cross-entropy loss, and labels are converted to indices during computation.
3

---

Stage!
K K,..., K) to indicate the category for Stage II
1. Label embeddings.
Main Loss
Encoder
PLM
Augmented djj Matrix
Reg. Loss
Classiffer
A - p m)
2. LSG.
Projector
A LSG Adji Matrix label-dala edges
Align Lass
Graph
data-data edges
3. Train graph neural network.
Embeddings
Node Loss
h, Gung Ag.
G(V,A,
Suug Ov. Ag.
Learnable
Figure 1: Illustration of the LSG training framework. In the first stage, labeled concepts are turned into text embeddings via pretrained language model (PLM), and then the Language Semantic Graph (LSG) is constructed according to the embeddings. We train GNN on the graph to extract semantic knowledge. In the second or the formal training stage, the data representations h are connected the graph based on their labels (and the new graph topology is described in the augmented adjacency matrix). Representations are then transformed into h' by the GNN. The proposed aligmment loss and regularization loss is applied on these features to guide the primary model.
We argue that such conversion of labels inevitably losses information and results in data inefficiency. Therefore, our method introduces two additional losses Lalign and Ly that exploit the discarded label semantic information for complement.
This section first introduces the construction process of the Language Semantic Graph (LSG) and the training process of the auxiliary model, which are conducted before the primary training stage to capture the semantic information. Then it moves on to explain the objectives Lulign and 2, guide the data-efficient training of the primary model through LSG.
3.1 Language Semantic Graph
LSG aims to model the semantic relationship between labeled concepts with high generality and flexibility. We propose a straightforward approach to construct the graph, leveraging the text embedding space that contains rich semantics. Specifically, given the natural language descriptions of the concepts {W 1 where K is the total number of concepts or categories, we use a set of m predetermined prompts (e.g. 'This a photo of (concept)") ') to contextualize the concepts. We then obtain a set of text embeddings T by inputting contextualized concepts into a frozen pretrained (1) (2) :...,th t f(m)) C Rd,, TT mK, language model such BERT (25 where T - (t tk
with de being the dimension of the text embedding space. Then we define the text embedding matrices K. and K, such that K. [t(]]
(m) ( ..... (m)) [ddx and K. be its column
normalized matrix. We compute the cosine similarity matrix between text embeddings S K. K. Additionally, we define a text label vector y,
m
172
each text embedding.
Next, we define the language semantic graph adjacency matrix based on text embedding similarities:
max (S 0)
if yr, Vt.j Si, if yr. + yt. and St, 2
A (aghsssgr
(2)
otherwise.
Note that A is weighted adjacency matrix. The first line means that connect nodes representing the same concept with their similarity scores as weight. The second line allows certain edge connections between nodes that represent different concepts but share high feature similarities, which is controlled be an edge threshold hyper-parameter т. Finally, the language semantic graph we defined can represented as: G (V, A),,, where V is the node set, | [T], and K. is used as the node feature matrix.
To fully leverage the LSG to guide the data-efficient training of the primary model, we propose to train auxiliary graph neural network to extract semantic information from LSG. We adopt the

---

architecture, which consists of several consecutive graph convolution layers. It is expected that GCN, with its strength in massage passing, first refines the initial node embeddings (or text embeddings) by leveraging graph topology and then serves as a bridge connecting the primary model and the LSG. Let D be the degree matrix of A, (()) be the ReLU [47] activation function, w be the learnable weight matrix for the la layer and (( = K. the update function of the node features can be formulated as:
standard GCN 28
00 oD AD ((())))((()).
Analogues to the primary model, we regard our GCN model as the composition of node feature encoder Fg and node classifier Cg. Particularly, the encoder Fg is designed to output node features that have the same dimension input features for the later computation of Lr. The whole network is trained using the node classification objective that aims to correctly predict category for each node:
1T
K
Luode
1R==, log 8 (C, FgC.,.)))
(4)
where 5()) is the softmax function. To this end, we obtain the LSG and the auxiliary model trained on it, which contain semantic information from the labeled concepts and will be leveraged to boost the data-efficient training of the primary model.
Ladde is only used for training GCN in this first stage. By doing this, each node processed by the GCN aggregates discriminative and more comprehensive semantic information from neighboring nodes. Also note that both the LSG and the GCN only depend on the categories of the current task. Hence, they can be constructed and trained on each task once and for all.
Remark:
3.2 LSG Guided Data-Efficient Learning
To enable language semantic guidance, we add a new projector (-Ib)) on top of the feature encoder F of the primary model to project the data representation the same dimension as the text embeddings. We denote h, H (F::)) Rdd the projected representation of ar; and h as its embedding after 12 normalization. We would like to enable interactions between the data representations and the node features within the semantic graph. Henceforth, we propose to connect features generated by the primary model into LSG according to the label of the data.
During training process, for every minibatch of B images {ærj Mo 1 we obtain their representations (hr B and construct an augmented graph of the original Language Semantic Graph, denoted as Gang (Va, A., K.)). In this augmented graph, Vaal |7| + B, the augmented node feature matrix K.,, iK, hi, .... ha] E (1+1+ is the node feature matrix plus B data features, and
P1
the augmented adjacency matrix defined as A.
where P R T binary M
P
if yt. = Uj otherwise, and M E BxBB B is a symmetric binary matrix indicating the connection among data nodes, M =
matrix indicating the connection between label nodes and data nodes, P = P (o
[m:g] = [1 if Vjj Based on the true label of training data, these edges build connection
0 otherwise.
between the primary model and the LSG, which allows the graph neural network to propagate information and transfer knowledge that is embedded inside the labels.
With the augmented graph, we conduct language semantic guidance to the primary model through optimizing two objectives: Lalign that implicitly aligns the data representation to the semantic graph and L, that explicitly regularize the model to extract more semantically meaningful representations. We utilize the frozen auxiliary model to encode new data nodes in the augmented graph and predict their categories. Given that the trained auxiliary model can correctly predict the label nodes in the original graph and its weights are no longer updated, the data representations h; are forced to become similar to their corresponding label node features during optimization, achieving the alignment between features from different node. The implicit alignment objective is as follows:
B K
min P.4
Latign ΣΣ s=l
A[=- log ((C, e FF Fl())))
(5)
Besides the implicit alignment loss that guides the data representations to have correct semantic relations, we further propose explicit regularization loss to transfer the knowledge embedded
5

---

in the label nodes to the primary model. Notice that after being processed by the trained GCN, the output features of the data nodes have already aggregated information from their label node neighbors on the augmented graph, which are desirable representations that contain information from both the data samples and the semantic graph. Thus, we propose the graph semantic feature regularization loss E, that directly uses the output features of the auxiliary model encoder F, to promote their initial input features, which why both features are designed to have the same dimension. Specifically, we denote (h;)) as the output graph feature for h, Then we normalize both features Fhh)) = F (ha)/1155 (h4)ll2: h = ha/l|hale and use the normalized graph feature as the target for normalized input feature. We have the following regularization objective:
B
min 0,0
C-
Σ
(sgg(F (h.)) h; 12.
(6)
where sg(-) denotes stop gradient. By explicitly regularizing the original data representation to include more information embedded in the graph as Eq. (6). the primary model learns more informative features from the LSG and thus allows this auxiliary branch be discarded during inference.
Two proposed objectives Latign. and C, work in a complementary manner to convey the semantic re- lations between labeled concepts to primary model, whereas the classical empirical risk minimization helps the model to discriminate. Combining the three losses, our proposed method achieves fully exploitation of the information inside human annotations, and improves the data utilization efficiency, The final optimization objective of LSG is the weighted sum of the three losses joined by trade-off parameters and μ. The full two-stage training process is shown in Fig.
min 6,0,0
L
Lemp + Muutign + P.C...
The GCN and correspondingly the projector not used during inference, and the classifi- cation result is still produced by the primary model. For this reason, no extra computational cost is needed inference time.
Remark:
3.3 Extend LSG to Unlabeled Data
We further discuss a straightforward extension of the proposed LSG on unlabeled training dataset {u) that share a same label space with the labeled data. By simply assigning a hard pseudo-label to each of them, these unlabeled data can also be connected to the semantic graph and be leveraged to guide the model. We find surprisingly that the vanilla Pseuso-Labeling strategy 34 works satisfactory with our method in semi-supervised learning. To be specific, we assign pseudo-labels to all unlabeled data according to the maximum class in model output: ú = arg max, 8(C F(u)), ), and these pseudo-labels are updated after each epoch. To this end, we extend LSG to semi-supervised learning by jointly utilize labeled and unlabeled data for constructing the augmented semantic graph, and the objective now becomes
min 0.0.0
Last= Lempla)) + >. ( u) (r, u).
(8)
With these pseudo-labeled samples, our method achieves significant performance boost despite that there exists noisy pseudo-labels. We hypothesis that it is because the pseudo-labels are leveraged only for the auxiliary branch and hence do not bias the classification head, as also suggested in (5). We
believe that further improvements can be achieved via combining our method with more advanced pseudo-labeling strategies in SSL, which will be left as future works.
4 Experiments
Datasets and Models. We conduct experiments on 7 standard datasets that are intensively studied in
Transfer Learning [77 41 27 and Semi-supervised learning 66 24 and cover input data ranging
images, videos and audios. For image datasets, we adopt FGVC Aircraft 44 (10,000 images for 100
aireraft variants), Stanford Cars [30] (16,185 images for 196 car categories) and CUB-200-2011 [66
(11,788 images for 200 bird species) for iineggriied classification analysis and Office Home [65
(four domains, each contains roughly 4,000 images for 65 categories) to evaluate out-of-distribution performance. For video datasets, we use UCF-101 [59 (13,3 (13,320 video clips in 101 categories)
and HMDB5/ 31 (6,766 clips form 51 actions) For audio dataset, we report the performance on
AudioSer-20K [16].

---

Table I: Classification accuracy (%) on three fine- finegrriiedd image benchmarks under fully-supervised and semi-supervised learning settings. ResNet-50 pretrained on ImageNet- kk is adopted as backbone and SSL indicates whether the rest training data is leveraged unlabeled samples.
FGCV Aircruft
Stanford Cars
CUB-200
Method
SSL
15%
30%
50%
100%
15%
30%
50%
100%
15%
30%
50%
100%
Fine-tuning
X
41.6
57.8
68.7
80.2
41.1
65.9
78.4
87.8
51.2
64.6
74,6
81.8
LWF [37
x
44.1
60.6
68.7
82.4
44.9
67.0
77.6
87.5
56.7
66.8
73.4
81.5
DELTA 36
43.6
59.5
69.6
81.2
43.3
67.6
79.6
88.0
53.4
66.7
76.0
82.0
BSS
x
44,4
61.9
71.4
82.7
45.0
68.4
79.6
88.4
54.8
67.3
76.3
82.3
StochNorm
29
x
44.3
60.6
70.1
81.5
44.4
68.1
79.3
87.9
54.8
66.8
75.8
82.2
Co- Co-Tuning
77
x
45.9
61.6
72.7
83.9
49.0
70.6
81.9
89.5
57.6
70.1
77.3
82.5
LSG
55.6
72.0
79.5
86.7
55.4
75.5
83.8
90.7
57.7
70.6
77.5
82.2
Pscudo-Lahcling
84
46.8
62.8
73.2
40.9
67.0
78.7
45.3
62.0
72.3
FixMatch
55.5
78.3
77.5
84.8
44.1
63.5
SimCLRv2 0
40.8
59.0
68.5
45.7
61.7
77.5
45.7
62.7
71.1
Self-Tuning (67
64.1
76.0
81.2
72.5
83.6
88.1
64.2
75.1
80.2
DehhissMathh 66
59.5
71.2
77.1
75.3
86.1
90.0
64.7
75.1
77.7
LSG (extended)
71.4
85.3
87.1
88.6
79.3
87.7
90.9
91.9
66.1
76.1
80.6
82.6
Table 2: Classification accuracy (%) with self self- self--uueevised pretraining model (CLIP ViT-B).
FGCV Airoraft
Stanford Cars
CUB-200
Method
15%
30%
50%
100%
15%
30%
50%
100%
15%
30%
50%
100%
CoOp
84
34.8
38.2
42,9
49.8
75.0
81.3
83.7
84.0
64.4
72.3
76.3
78.7
ProDA 4
33.9
36.6
43.5
50.8
75.4
81.5
84.0
84.2
65.5
73.1
76.4
79.2
Tip-Adapter LSG
[1
36.9
41.7
46.7
53.8
75.6
81.4
84.1
85.0
69.0
74.9
77.9
81.2
48.9
58.6
65.0
74.5
79.4
83.2
86.1
90.1
70.2
78.4
81.9
85.4
Several deep neural networks with different architecture and pretraining dataset included in the experiments. For image tasks, we use ResNet-50 [21] and ConvNext-S [40] pretrained on ImageNet-Ik 1k [TO), ViT-B pretrained on CLIP WIT [5]], and Swin-B (39] pretrained on ImageNet- 21k. For video and audio tasks, we adopt pretrained ViT-L [3 from VideoMAE [6 and ViT-B from Audio-MAE [22] respectively.
Implementation Details. We utilize pretrained BERT-L 25 as the language model to transform labels to text embeddings. For each concept, contextualize it into complete sentences using 20 handcrafted prompts. When constructing the Language Semantic Graph, the similarity threshold is determined adaptively to include the top p 0.3% edges that connects between nodes of different labels of the fully connected graph. The GCN is trained on full graph for 5,000 iterations.
For LSG guided training process, the projector H is implemented by fully-connected layer with an output dimension of 1024 and randomly initialized weights. We find that setting À and μ to 1.0 and 8.0 generally achieves satisfying results within all the experiments. In image classification tasks, we adopt SGD with a momentum of 0.9 as the optimizer. The learning rate is set as 1e-3 for the visual backbone in most experiments and 10x larger value is applied for the classifier and projector in SSL and SDG. In video and audio tasks, we adopt the same configurations as finetuning in the official released VideoMAE and Audio-MAE codebase. Please refer to the appendix for more details.
4.1 Results of Image Experiments
To examine the effectiveness of our method, first conduct experiments three fine-grained image classification benchmarks FGVC-Airerafi, Stanford Cars and CUB-200-2011 following [77 67).
Similar to previous works, we analyze the performance of LSG under labeled data partition ratio of 15%, 30%, 50% as well as the full training set. We adopt ImageNet- 1k pretrained ResNet-50 as backbone and compare LSG with previous transfer learning baselines: vanilla fine-tuning, LWF (37. DELTA [36.. BSS 77.. StochNorm (29 and Co-Tuning [77]. In addition, to show that LSG is applicable to supersupe pretraining models, we adopt CLIP pretrained ViT-B and compare LSG to vision-language fine-tuning methods CoOP [84). ProDA (43, Tip-Adapier
[8I] Finally, we extend L.GG to superviee learning setting by including the rest training data as unlabeled data. As discussed in 3.3 we adopt Pseudo-Labellng K4 strategy for unlabeled data (which can be regarded as the baseline for extended I.SG) and compare the extended LSS with SSL methods including FixMatch (58), DebiasMatch (68), SimCLRv2 (6) and Self- Self-Tuning (67)
7

---

Table 3:
accuracy (%) for single domain generalization on Office- Office-Home. Backbone ConvNext-S and Swin-B B are pre- pre-trained on ImageNet- 1k and ImageNet-222 22k respectively.
Source:Ar
SoureerC1
Source.Pr
Source:Rw
Avg.
Backbone
Method
Ar
C1
Pr
Rw
CI
Ar
Rw
Pr
Ar
Rw
Rw
C
Pr
(ID)
(OOD)
BRM
27
85.0
53.4
72.7
78.6
85.0
67.5
72.9
75.4
94.7
61.8
49.0
80.0
91.4
12.2
52.7
80.9
89.0
68.1
Con-Next-S
LP-FT
L7
65.1
55.1
73.3
79.5
85.6
70.4
73.4
77.5
95.1
63.9
$2.0
82.0
91.4
72.0
54.1
82.2
89.3
69.6
LSG
85.8
57.7
74,0
79.9
86.3
71.7
75.4
77,8
95.1
65.8
54.7
82.1
91.2
73.8
58.0
$2.3
89.6
71.1
FRM LP
23
91.8
70.7
86.1
88.5
89.1
80.6
84.3
86.7
96.5
77.9
66.
88.1
96.2
82.6
69.1
90.4
93.2
81.0
Swin
91.5
70.4
85.9
88.6
89.4
813
87.1
96.7
78.8
68.0
53.9
98.1
82.9
70.1
90.5
93.2
72.4
89.4
70.8
The results are shown in SableTand Table 2 We observe that our method outperforms its counter- parts on each task and in both fully-supervised and semi-supervised scenarios. In fully supervised experiments, LSG significant outperforms other methods on FGVC-Aircraft and Stanford Cars es- pecially when the available labeled data is more scarce. It also achieves comparable performance on CUB200. At 15% sampling ratio, LSG improves from the vanilla fine-tuning by 14.0%. 14.3% and 6.5% three datasets, and surpasses Co-tuning by an average accuracy of 5.3%. Similarly, LSG shows promising potential in semi-supervised setting, achieving the best performance across all of the labeling rate and all three datasets. Notably, since the extended LSG incorporates data augmentation techniques adopted in semi-supervised learning. its results improvement over LSG under 100% training data shows that the proposed method is mutually beneficial to other data-efficient strategies. When applied on self-supervised pretrained model, LSG also demonstrates consistent gains comparing to fine-tuning methods that are specifically designed for vision-languan models.
We also conduct experiments on cross-domain benchmark Office-Home to evaluate the out-of- distribution performance of the LSG guided training. Following (27). we consider the challenging single domain generalization setting where models are trained on one single source domain and tested on the rest target domains without seeing any data from them. We compare LSG against the vanilla empirical risk minimization (ERM) baseline and the more advanced approach LP-FT [32].
We report the prediction accuracies of both in-distribution (ID) and out-of dittr--disttibution (OOD) in Table 3 Specifically, samples the source domain are randomly partitioned into 80% training data and 20% test data, and the results refers accuracy on test data. OOD results are obtained evaluating all data in each target domain. The results show that LSG improves model performance on both ID and OOD samples, demonstrating that the label semantic relations help the model learn features that are more robust to distribution shift.
4.2 Results of Video and Audio Experiments
Category labels in video and audio tasks also have their natural language de- scriptions, such as "push up". "kiss" in HMDB51 dataset and "singing". "groan" in AudioSer. This enables LSG construct semantic graph and improve the model training process. We adopt VideoMAE pretrained ViT-L as backbone for small-scaled video datasets UCF-101 and HMDB51, and use Audio- MAE pretrained ViT-B for audio bench- mark AudioSet.
Table 4: Accuracies (%) of video action recognition on UCF-101 and HMBD5/ under label rate of 1% and 10%. Backbone: ViT-L pretrained by VideoMAE.
UCF-191
HMDB51
Method
SSL
1%
10%
1%
10%
VideoMAE ft [6] LSG
26.80
73.26
11.82
46.85
29.07
74.49
13.28
50.50
Pseudo-1 Laheling
76.08
11.97
47.72
FixMatch 5
30.51
76.45
13.12
48.30
CMPL
32.24
77.82
13.58
50.81
LSG (exlended)
35.32
80.36
15.40
52.65
Following previous data efficient studies 24 the two video benchmarks, we train the model using 1 % and 10% labeled data respectively. We also con- duct SSL experiments similar to image experiments and compare LSG against SSL approaches in video. From the results in Table we observe that LSG consistently improves the fine-tuning accuracy over all tasks with only limited amount of labeled samples available. With unlabeled data being included, two classical SSL methods Pseudo Labeling and FixMatch both achieve certain improvements, whereas CMPL. [33] a method designed specifically for video SSL tasks, performs better. On top of that, LSG obtains superior results than these method all of the tasks, boosting the accuracy from
73 on
Table 5: Classification accuracies (%) on
AudioSer-200 with label proportion of 10%, 25% and 50%. (ft: Fine-tuning)
Method
10%
25%
50%
Audio- ft 22
2.59
15.60
25.96
LSG
11.20
21.80
27.82
8

---

Table 6: Ablation study of losses and alternative design choices on FGVC Aircraft. 15% labeled data.
Method
Lemp
Lutiin
Cr
Lr
w/
classifier
language head
Acc.
LSG w/o Ly, Latton
n/a v
41.6 (-14.0)
LSG w/o C.
n/a
44.2 (-11.4)
LSG w/o Calign
V
V
50.7 (-4.9)
LSG
V
55.6 (+0.0)
Clasiffer--Langunge Head
47.9 (-7.7)
LSG w/o stop gradient
48.3 (-7.3)
Table 7: Ablation study using different alignment strategy to transfer language semantic knowledge to the primary model on four image datasets. OH denotes for Office-Homee Home dataset.
FGCV Aircraft
Stanford Cars
CUB-200
OH Avg.
OH Avg.
Method
15%
30%
50%
15%
30%
50%
15%
30%
50%
(ID)
(OOD)
Vanilla (No Alignment)
65.9
78.4
51.2
64.6
74.6
89.0
Text Prototype Alignment
51.1
68.8
76.5
52.4
73.0
82.6
56.0
70.1
76.8
89.0
68.8
Contrastive Alignment
54.2
70.7
78.3
54.3
73.8
83.2
56.7
69.9
77.1
89.4
69.3
LSG
55.6
72.0
79.5
55.4
75.5
83.8
57.7
70.6
77.2
89.6
71.1
its baseline Pseudo-Labeling by 4.28% and 4.93%, and achieves 80.36% and 52.65% accuracies on
two datasets with only 10% labeled data.
For audio experiment, we utilize the Audio-MAE VTT-B model pretrained the full AudioSet-2M and report its finetuning results a class-balanced AudioSet-20K subset using 10%. 25% and 50% labeled audio clips. Our results is illustrated Table where we compare LSG with vanilla fine-tuning method. As consequence, LSG achieves an average of 5.56% accuracy enhancement from the baseline, which demonstrates that guiding the audio representation by their label semantic information could also be beneficial. These results on video and audio experiments demonstrate that LSS is widely application to various modalities.
4.3 Analytical Experiments
Ablation Study. We report the accuracy of different LSG variants on FGVC-Aircrafi with 15% labeled data (Table o We begin with respectively removing the regularization loss C, and the alignment loss Lutign from the total objective. The performance drops of 14.4% and 4.9% prove that both objectives contribute the semantic guidance. Moving on, we investigate two variants: replacing the trainable classifier by fixed head from language embeddings, and removing the stop gradient in C, Both variants performs much worse than the current design.
Effectiveness of the GCN model. Here we conduct two experiments to verify the effectiveness of the GCN model on refining the label embeddings and promoting seman- knowledge transfer from LSG to the primary model. First, we com- pares the between the original and refined la- embeddings, where higher index indicates better clustering performance. We use the node embeddings of the penultimate layer of the GCN as the refined label embeddings. As shown in Table 8, the index of the refined embeddings significantly higher than the original ones, which demonstrates that the GCN model learns the discriminative information and improves the label embedding quality.
Table 8: beddings from pretrained language model and the GCN refined label embeddings. OH denotes for Office- Home datasct.
Indexes of the original label em-
Method
Aircraft
StanfordCars
CUB200
OH
Original 86.5
137.2
1.5.88
85.5
GCN refined
1364.1
1481.7
916.3
1395.2
Index
Second, we compare the performance of LSG with two alternative alignment strategies: Text Prototype Alignment and Contrastive Alignment. For the former, we use the average of the label embeddings as the class prototypes and minimize the cosine similarity between features and their corresponding prototypes. For the latter, we adopt supervised contrastive loss (66 between features and label embeddings. The results are shown in Table We observe that both alignment strategies improve the performance of the vanilla finetuning, which demonstrates that learning semantic relation via the alignment loss is beneficial to the model training. However, the performance of LSG is still

---

UCF-101, rabolrado 18
Audi abel rato 28%
LSG GON LSG GAT
50
N100 Way 1h5
12
175
95
61% Ci 0n
215
ain
210
12
Audoo MAE freouning
Frimary model
10 20
90 50
A AAcourar (9)
Crossiabel eage rato
Figure 2: Analytical experiments: (a) and (b) Model accuracy on validation sets during training. (c)
Comparison between GCN and GAT. (d) Effect of the threshold syyper-arameen T.
superior to the two alternatives, which indicates that the GCN model is more effective in transferring the label semantic knowledge to the primary model.
LSG speeds up model training. It is discovered that applying LSG to guide the representation learning encourages the model to achieve a satisfactory performance faster. The accuracy curves on validation set of LSG and its baseline in both video (UCF-101) and audio (AudioSet) training are depicted in Fig. 2 (b).). The aceuracy curve of LSG rises faster and higher than vanilla fine-tuning.
Substitute GCN by GAT 63 To further investigate the effect of the graph neural network in our system, change the implementation from GCN Graph Attention Network (GAT), which is another wildly adopted GNN. The biggest different is that GAT automatically learns the weight between connected nodes instead of relying manual definition as GCN does. We compare the two variants on FGVC Aircraft SSL with 15% labeled data and discover an interesting trend. As shown in c), GAT outperforms GCN in low label regime labeled data < 15%) whereas GCN is more beneficial with more labeled samples.
Fig. (c.,
Analysis of the LSG edge connection threshold The ppper-paramtter T in Eq. controls the topology (i.e., the number of edges that connect different labels) of the Language Semantic Graph. We study the effect of this threshold on both the GCN node classification accuracy and more importantly the primary model accuracy. We observe (in Fig. ()) that as the ratio of cross-label edges decrease, the graph topology becomes more simple and GCN accuracy increases. However, due to lack of beneficial cross-label interaction, the primary model performs worse. In contrast, if the ratio is overwhelmingly high, the graph topology becomes too complicated and results degradation in both GCN and primary model performance.
5 Limitation and Future Work
The proposed semantic graph is a general concept that aims to capture the high-level semantic relations between labeled concepts. In this paper, we only use the natural language descriptions of the labeled concepts and leverage the pretrained language model to construct the graph. Although using language embedding is effective in many data-efficient scenarios across various modalities, it is still limited when the labeled concepts in the task are not well described by natural language. However, Howeeer the semantic relations between concepts can also be captured by other ways, such as manually determining the relationship between categories based on expert knowledge or leveraging a knowledge graph. In the future, we will explore how to combine the semantic relations from different modalities to construct a more comprehensive graph.
Meanwhile, since the proposed LSG a simple and lightweight modification the standard training process and does not alter the inference procedure, another promising direction would be to extend the proposed method other data-efficient scenarios, such as semantic segmentation and object detection.
6 Conclusion
In this paper, we study how to exploit the semantic information in labels for improving data efficiency. We present LSG, novel and effective data-efficient learning method that leverages Language Semantic Graph to guide the model learn semantic relations between labeled concepts. LSG consists of two parts: an auxiliary graph neural network that extracts knowledge from the semantic graph and two novel optimization objectives that transfer the knowledge to primary model. We demonstrate that L.SG is applicable on image, video and audio models and brings significant performance gains the model under Transfer Learning and Semi-Supervised Learning scenarios.
10

---

Acknowledgement
This paper was supported by National Key R&D Program of China (No. 2021YFB3301503 the National Natural Science Foundation of China (No. 62376026), and also sponsored by Beijing Nova Program (No. 20230484296), -enceete Rhino-Bire Open Research Fund.
References
[1] A. Adadi. A survey data-efficient algorithms in big data era. Journal of Big Data, 8(1):24, 2021.
[2] Y. Balaji, S. Sankaranarayanan, and R. Chellappa. Metareg: Towards domain generalization using meta-regularrzztion. In NeurIPS, 2018.
[3] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernt,, Oliver, and C. A. Raffel. Mixmatch: holistic approach to semi- supervviee learning. In NeurlPS, 2019.
[4] D. Berthelot, R. Roelofs, K. Sohn, N. Carlini, and A. Kurakin. Adamatch: A unified approach semi-supervised learning and domain adaptation. ICLR, 2022.
[5] B. Chen, J. Jiang, X. Wang, P. Wan, Wang, and M. Long. Debiased self-training for semi- supervised learning. In NeurIPS, 2022.
[6] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurlPS, pages 22243-22253 2020.
[7] X. Chen, Wang, B. Fu, M. Long, and J. Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In NeurlPS, 2019.
[8] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213-3223, 2016.
[9] E. D. Cubuk, B. Zoph, Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, pages 702-703, 2020.
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255. 2009.
[11] Dinh, Y. Zeng, R. Zhang, Z. Lin, M. Gira, S. Rajput, J.-y. Sohn, D. Papailiopoulos, and
K. Lee. Lift: Laneuage-inteffaeed fine-tuning for non-language machine learning tasks. In NeurlPS, pages 11763-11784, 2022.
[12] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In ICML, pages 647-655, 2014.
[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
[14] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICMI., pages 1180-1188 1180,,119110000 2015.
[15] V. Garcia and J. Bruna. Few-shot learning with graph neural networks. In ICLR, 2018.
[16] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP. pages 776-780, 2017.
[17] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In ICML, pages 1272-127, 2017.
[18] R. Girshick, J. Donahue, Darrell, and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.

---

[19] Y. Grandvalet and Y. Bengio. Semi- Semi-super learning by entropy minimization. In NeurlPS, pages 529-536, 2005.
[20] M. Guo, E. Chou, D.-A, Huang, S. Song, S. Yeung, and L. Fei-Fei. Neural graph matching networks for fewshot 3d action recognition. In ECCV, pages 633-669, 2018.
[21] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[22] P.-Y. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer. Masked autoencoders that listen. In NewrIPS, pages 277002 2022.
[23] H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In ACL, 2020.
[24] L. Jing, T. Parag, Wu, Y. Tian, and H. Wang. Videossl: Semi-supervised learning for video classification. In WACV, pages 1110-1119, 2021.
[25] D. M.-WW C. Kenton and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186, 2019.
[26] P. Khosla, P. Teterwak, C. Wang. A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
[27] D. Kim, K. Wang, S. Sclaroff, and K. Saenko. A broad study of pre-training for domain generalization and adaptation. ECCV, 2022.
[28] T. N. Kipf and M. Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[29] Z. Kou, K. You, M. Long, and Wang. Stochastic normalization. In NeurIPS, pages 16304- 16314, 2020.
[30] Krause, M. Stark, Deng, and Fei-Fei. 3d object representations for fin-graiied categorization. In CVPR Workshops, pages 554-561, 2013.
[31] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre Hmdb: large video database for human motion recognition. In ICCV, pages 2556-2563, 2011.
[32] A. Kumar, A. Raghunathan, R. Jones, T. Ma, and P. Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR, 2022.
[33] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning, nature, 521(75533:436-444 2015.
[34] D.-H. Lee et al. Pseudo-label: The simple and efficient semi semi-supervised learning method for deep neural networks. In JCML Workshop, 2013.
[35] S. Li, M. Xie, K. Gong, C. H. Liu, Y. Wang, and W. Li. Transferable semantic augmentation for domain adaptation. CVPR, pages 11516-11525, 2021.
[36] X. Li, H. Xiong, H. Wang, Y. Rao, L. Liu, Z. Chen, and J. Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In ICLR, 2019.
[37] Z. Li and D. Hoiem. Learning without forgetting. TPAMI, 40(12):2935-2244,, 2017.
[38] T. Y. Lin, M. Maire, S. Belongie, J. Hays, and C. L. Zitnick. Microsoft coco: Common objects in context. ECCV, pages 740-755, 2014.
[39] Z. Liu, Y. Lin, Y. Cao, H. Hu, Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 11012-11022, 2021.
[40] Z. Liu, Mao, C.-Y. Wu, C. Feichtenhofer, Darrell, and Xie. convnet for the 2020s. In CVPR, pages 1976-11986. 2022.
12

---

[41] Z. Liu, Y. Xu, Y. Xu, Q. Qian, H. Li, X. Ji, A. Chan, and R. Jin. Improved fine-tuning by better leveraging pre-training data. In NeurIPS, pages 32568-32581, 2022.
[42] M. Long, Y. Cao, Z. Cao, J. Wang, and M. Jordan. Transferable representation learning with deep adaptation networks. TPAMI, 41(12):371130083 2018.
[43] Y. Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian. Prompt distribution learning. In CVPR, pages 2206-5215, 2022.
[44] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. CoRR, 0bss/130.5555, 2013.
[45] D. Marcheggiani and I. Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In EMNLP, 2017.
[46] F. Monti, M. Bronstein, and X. Bresson. Geometric matrix completion with recurrent multi- graph neural networks. NeurIPS, 2017.
[47] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pages 807-814, 2010.
[48] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In CVPR, pages 1717-1724, 2014.
[49] S. J. Pan and Q. Yang. A survey on transfer learning. TKDE, 22(10):135--1399 2010.
[50] X. Pan, T. Ye, D. Han, S. Song, and G. Huang. Contrastive language-image pre-training with knowledge graphs. In NeurIPS, 2022.
[51] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 9768- 2021.
[52] A. Radford, K. Narasimhan, T. Salimans, Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[53] M. N. Rizve, K. Duarte, Y. S. Rawat, and M. Shah. In defense of uncertainty-awarc pseudo-labe selection framework for cemi-supervised learning. In ICLR, 2021.
An
[54] K. Saito, D. Kim, S. Sclaroff, T. Darrell, and K. Saenko. Semi-supervised domain adaptation via minimax entropy. In ICCV, pages 8050-8058, 2019.
[55] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsuper- vised domain adaptation. In CVPR, pages 3723-3732, 2018.
[56] Shen, I.. Li, I.. M. Dery, C. Staten, M. Khodak, G. Neubig, and A. Talwalkar. Cross-modal fine-tuning: Align then refine. In ICMI, 2023.
[57] S. Shen, C. Li, X. Hu, Y. Xie, J. Yang. P. Zhang, Z. Gan, L. Wang, L. Yuan, C. Liu, et al. K-lite: Learning transferable visual models with external knowledge. In NeurIPS, pages 15558-15573, 2022.
[58] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. CoRR, abs/2001. 076655 2020.
[59] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1211.0002, 2012.
[60] A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurlPS, 2017.
[61] Z. Tong, Y. Song, J. Wang, and I.. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised vidco pre-training. In NeurIPS, 2022.
13

---

[62] J. E. Van Engelen and H. H. Hoos. A survey on semi-supervised learning, Machine learning. 109(2)3333-40)) 2020.
[63] Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, Bengio, et al. Graph attention networks. stat, 1050200:10-48555 2017.
[64] Velickovic, W. Fedus, W. L. Hamilton, Liò, Y. Bengio, and Hjelm. Deep graph infomax. In ICLR, 2019.
[65] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pages 5018-5027, 2017.
[66] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[67] X. Wang, J. Gao, M. Long, and J. Wang. Self-tuning for data-efficient deep learning. In ICML, pages 10738-10748 2021.
[68] X. Wang, Z. Wu, L. Lian, and S. X. Yu. Debiased learning from naturally imbalanced pseudo- labels for zero-shot and semi-supervisee learning. In CVPR, 2022.
[69] Y. Wang, H. Chen, Fan, W. Sun, R. Tao, W. Hou, Wang, Yang, Z. Zhou, L.-Z. Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. In NeurIPS, pages 3938-3961, 2022.
[70] C. Wei, K. Shen, Y. Chen, and T. Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In ICLR, 2022.
[71] Y. Wu, G. Zhang, Y. Gao, X. Deng, K. Gong. X. Liang, and Lin. Bidirectional graph reasoning network for panoptic segmentation. In CVPR, pages 0080-9089, 2020.
[72] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip. A comprehensive survey on graph neural networks. TNNLS, 32(1):4 24, 2020.
[73] Y. Xu, F. Wei, X. Sun, C. Yang, Y. Shen, B. Dai, B. Zhou, and S. Lin. Cross-model pseudo- labeling for semi-supervised action recognition. CVPR, pages 2959-2968, 2022.
[74] L. Xuhong, Y. Grandvalet, and F. Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML, pages 2825-2834, 2018.
[75] L. Yang, W. Zhuo, L. Qi, Y. Shi, and Gao. St++: Make self-training work better for semi-supervised semantic segmentation. In CVPR, pages 4268-4277, 2022.
[76] Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NeurlPS, pages 3320-3328, 2014.
[77] K. You, Z. Kou, M. Long, and J. Wang. Co-tuning for transfer learning. In NeurlPS, pages 17246, 2020.
[78] H. Zhang, M. Cisse, N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In ICLR, 2017.
[79] H. Zhang, P. Zhang, X. Hu, Y.-.. Chen, L. Li, X. Dai, L. Wang, L. Yuan, J.-N. Hwang, and J. Gao. Glipv2: Unifying localization and vision-language understanding. In NewrIPS, pages 36067-36080, 2022.
[80] R. Zhang, X. Hu, B. Li, S. Huang, H. Deng, H. Li, Y. Qiao, and P. Gao. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In CVPR, 2023.
[81] R. Zhang, W. Zhang, R. Fang, P. Gao, K. J. J. Dai, Y. Qiao, and H. Li. Tip-adapter: Training- free adaption of clip for few-shot classification. In ECCV, pages 493-510, 2022.
[82] Y. Zhang, Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In ICML, pages 7404-7413, 2019.
14

---

[83] K. Zhou, J. Yang, C. C. Loy. and Z. Liu. Conditional prompt learning for vision- laigual models. In CVPR, pages 16816-16825, 2022.
[84] K. Zhou, Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. IJCV, 13009):2337-2348, 2022.
[85] X. Zhu, J. Zhu, H. Li, X. Wu, H. Li, X. Wang, and J. Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR, pages 2022.
[86] Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-7,, 2020.
15


Reviews:
[{"review_id": "9dX2CHEA1R", "reviewer": "Reviewer_ZDzt", "summary": "The typical supervised training ignores the semantic information in the labels. This paper proposes to use the label semantic information during fine-tuning. 1) A label semantic graph is constructed by calculating the sentence embedding similarity of the label descriptions; 2) a GNN is trained on the semantic graph with a node classification objective; 3) the GNN is frozen and guides the training of a visual classifier with two additional loss terms. Specifically, at training time, for each data batch, we can create a graph with the images and labels as nodes; the GNN can be used to encode the graph and the data representations are treated as the initial node features; one loss terms classify the final node features while one loss term minimises the distance between the data representations and the final node features after GNN.\n\nThe experiments are extensive, covering image, audio, and video classification. The approach can be used in a semi-supervised setting as well, where the pseudo-label data are only used in the additional loss terms and do not bias the classification head. The performance improvement is significant in many cases.\n", "questions": "For training the GNN, isn’t the node classification objective trivial? The objective is basically recovering the concept mentioned in the text prompt, which seems like an easy to solve task; the node feature at the first layer (K^0) should be enough to perfectly solve the task. This is different from the attribute classification task in typical GNN training, where there is not enough information to determine the node label without the adjacency matrix.", "limitations": "No evident negative societal impact.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The high-level motivation makes sense. It is desirable to utilise the semantic information within the labels during training.\n\n2. The performance improvement is significant in many settings and the evaluation is comprehensive.\n ", "weaknesses": "1. While the idea of using label semantic information is good, the presented solution seems ad-hoc and not well motivated. \nConceptually, the new information we introduce is by constructing the semantic graph A with a pre-trained text encoder; after this, it seems rather unclear what motivates the two-step approach with a GNN.\nIn addition, I find the training objective of the GNN unclear (see Question).\n\nThe role of multiple language prompts is also unclear. Is it because using multiple prompts builds a more informative graph?\n\n2. One straightforward way to utilise the label semantic information is to directly use a language encoder to encode the labels and then fine-tune the language encoder along with the data encoder (this is different from the Language Head baseline where the language embeddings are fixed). According to ELEVATAR, this greatly outperforms vanilla fine-tuning. \nI wonder if the authors have considered this baseline, especially in the setting of Table 2: can we use CLIP image encoder + CLIP language encoder? Basically, if our pre-trained model already considers the semantic information of the labels (e.g., CLIP), does the approach still bring much improvements?\n\nELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your endorsement of our idea and the insightful feedback. Please find our responses for your questions below.\n\n**Q1. Motivation of the two-step approach with GNN.**  \n\nA1.  We take vision model as an exemplar of the primary encoder $F$  to explain the idea. Our main goal is to align the image features encoded by $F$ towards the semantic space of label embeddings obtained from pre-trained language model. The GNN-based two-step approach promotes the alignment with the following two-fold reason:\n\n- In the first stage, we adopt multiple language prompts to increase the information and diversity of label embeddings for each category. Yet a side effect of is that some poorly-crafted prompt template may generate \"prompt cluster\" in embedding space due to its dominance influence on the label embedding over the concept that we want to distinguish (see Figure 3(a)). To minimize their negative impact, **GNN with its strength in massage passing, refines the initial node embeddings by leveraging graph topology**. We train GNN to obtain better label embeddings that are both semantic discriminative and structure-preserving. We observe from the T-SNE in Figure 3(a) that the node embeddings after refinement are more discriminative and preserves semantics. \n- More importantly for the second stage, GNN acts as the key component of the proposed alignment method. **By joining the image features as new nodes to the graph, we directly control the knowledge transfer process on the augmented graph via GNN**. To show the superiority of GNN-based alignment method, we conduct an ablation study by considering two classical alignment strategy as alternatives: *Prototype alignment* refers to minimizing the distance between image features and their corresponding category prototype computed from the average of label embeddings; *Contrastive alignment* refers to using supervise contrastive loss that treats label embeddings from the same category as positive samples. The results below validate the superiority of GNN-based alignment.\n\n| Variants| Aircraft-15% | Aircraft-30% | Aircraft-50% | Aircraft-100% |\n| -- | :--: | :--: | :--: | :--: |\n| Prototype alignment | 51.1 | 68.8 | 76.5 | 84.6 |\n| Contrastive alignment | 52.2 | 69.7 |77.3|85.1|\n| LSG | **55.6** | **72.0** | **79.5** | **86.7** |\n\n**Q2. Explanation of the  GNN training objective.** \n\nA2. Thanks for your question. It is a good question, and the answer to it reflects the key technical innovation of LSG. If we only consider the node classification on the original graph $\\mathcal G$, leveraging the initial label embeddings is pretty enough. However, the purpose of the GNN is not only about classifying these nodes, but more importantly to guide image features learn label semantics. Therefore, we design a pair of losses $\\mathcal L_{node}$ and $\\mathcal L_{align}$ that works cooperatively to align image features towards corresponding label embeddings in the graph, which differs from the classical purposes of GNN. \n\nSpecifically, due to the large difference between image features and label embeddings at the beginning of the second stage, even a well-trained GNN cannot classify the new nodes correctly. Since the GNN is set fixed, the only way to minimize $\\mathcal L_{align}$ is to update $F$ to generate image features more similar to label embeddings, allowing the GNN to recognize them. Therefore, the purpose of loss $\\mathcal L_{node}$ is to prepare a GNN ready for $\\mathcal L_{align}$ in the second stage. As shown in the following table, if we use a randomly initialized GCN in the second training stage (i.e., w/o $\\mathcal L_{node}$), the performance will drop and be similar to the variant where $\\mathcal L_{align}$ is excluded. \n\n|Variants | Cars-15% | Cars-30% | Cars-50% | Cars-100% |\n| -- | :--: | :--: | :--: | :--: |\n| LSG w/o $\\mathcal L_{node}$ (1st stage) | 48.6 | 68.2 | 80.4 | 89.1 |\n| LSG w/o $\\mathcal L_{align}$ (2nd stage) | 48.9 | 68.5 |80.5|89.3|\n| LSG w/ trained GCN| **55.4** | **75.5** | **83.8** | **90.7** |\n\n**Q3. Improved baseline and CLIP$_{text}$ as language model**. \n\nA3. Good comment. As you suggested, we conduct the following experiments under the same setting as table 2 with two new variants.\n\n- *Tunable language head* refers to the stronger baseline by making the language embedding initialized classifier tunable. We draw the same conclusion as in ELEVATER that the tunable head achieves better performance than random initialization. Yet, our method still outperforms this language-augmented baseline on all the tasks with a great margin. We will add this new baseline to the paper. \n- *CLIP$_{text}$* refers to substituting language model from BERT to CLIP language encoder. We find it achieves similar performance to BERT encoder. Reason behind this effectiveness is that, although CLIP image encoder already considers the label knowledge contained in CLIP$_{text}$ in pre-training stage, the fine-tuning process will distort the image feature space [31] and LSG serves as a regularization to prevent the model from forgetting such label semantic knowledge. \n\nPlease refer to Table A in the pdf for results.\n\n**Q4. The effect of multiple language prompts.** \n\nA4. We agree with your opinion. Different prompts will put the label concept into different contexts, and consequently produce divergent label embeddings. Such diversity leads to increased information and improved performance. As shown in table below, increasing the number of prompt $m$ leads to better performance. However, consider that the number of nodes in the semantic graph is proportional to $m^2$, and notices that the performance gain is no longer significant when $m>20$, we find a balance between accuracy and training cost by setting $m=20$.\n\n| Task| $m=$5 | 10 | 20 | 40 | 80 |\n| -- | :--: | :--: | :--: | :--: | :--:|\n| Air-15% | 54.8 | 55.0  | 55.6  |55.7| 55.7 |\n\nWe hope these responses address your concerns, and welcome any further questions that you find worth discussing!"}, {"review_id": "WDxO9auv3G", "reviewer": "Reviewer_biPo", "summary": "The paper introduces the Language Semantic Graph (LSG), a novel approach to data-efficient learning that leverages semantic information from labels. The LSG is used to train an auxiliary graph neural network, which then guides the primary model's training, enhancing the utilization of label knowledge. This method is applicable across various modalities, including image, video, and audio, and has shown significant performance enhancement in both Transfer Learning and Semi-Supervised Learning scenarios.\n\nExperiments were conducted on seven standard datasets covering images, videos, and audios, using several deep neural networks with different architectures and pretraining datasets. The results show that LSG significantly outperforms other methods, especially when labeled data is scarce. It also demonstrates promising potential in semi-supervised settings, achieving the best performance across all labeling rates and datasets. When applied to self-supervised pretrained models, LSG shows consistent gains. It also improves model performance on both in-distribution and out-of-distribution samples, indicating that label semantic relations help the model learn more robust features.\n\nIn video and audio experiments, LSG consistently improves the fine-tuning accuracy across all tasks with limited labeled samples. It outperforms other methods, boosting accuracy significantly. For audio experiments, LSG achieves an average of 5.56% accuracy enhancement from the baseline, demonstrating its wide applicability across various modalities.\n\n\"LSG consists of two parts: an auxiliary graph neural network that extracts knowledge from the semantic graph and two novel optimization objectives that transfer the knowledge to primary models.\" The authors demonstrate that LSG is applicable on image, video and audio models and brings significant performance gains to the model under Transfer Learning and Semi-Supervised Learning scenarios.", "questions": "*The LSG method relies heavily on the quality and semantic richness of the labels. How does the quality of the labels impact the performance of the LSG method? Could the LSG method be adapted to work effectively with less informative or ambiguous labels, and if so, how?\n\n* The paper primarily focuses on classification tasks. Could the LSG method be adapted or extended to other tasks, and if so, what modifications would be necessary?", "limitations": "* The LSG method relies heavily on the quality and semantic richness of the labels. If the labels are not well-defined, sparse, or ambiguous, the performance of the LSG method could be compromised. This dependence on high-quality labels could limit the applicability of the method in certain scenarios.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Innovative Approach: The paper introduces a novel method, the Language Semantic Graph (LSG), which leverages semantic information from labels to guide the training of machine learning models. This approach offers a new perspective on data-efficient learning that has not been extensively explored in previous research.\n* Versatility Across Modalities: The LSG method is applicable across various modalities, including image, video, and audio. This wide applicability demonstrates the robustness and flexibility of the proposed method.\n* Superior Performance: The LSG method shows significant enhancement in performance compared to other data-efficient learning approaches in both Transfer Learning and Semi-Supervised Learning scenarios. This is a strong indication of the effectiveness of the proposed method.\n* Robustness to Data Scarcity: The LSG method performs particularly well when labeled data is scarce, which is a common challenge in machine learning. This makes it a valuable tool for scenarios where data collection and labeling are costly or impractical.\n* Improved Out-of-Distribution Performance: The LSG method improves model performance on both in-distribution and out-of-distribution samples. This suggests that the label semantic relations help the model learn features that are more robust to distribution shift, enhancing the model's generalizability.", "weaknesses": "The effectiveness of the LSG method relies heavily on the quality and semantic richness of the labels. In scenarios where labels are sparse, ambiguous, or not well-defined, the performance of the LSG method could be compromised.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback and comments on our submission. Please find our responses for your questions below.\n\n**Q1. Performance evaluation on low quality labels.** \n\nA1. Thanks for the good comment. To investigate the effectiveness of LSG on low quality label scenarios, we stimulate the label corruption scenarios by manually adding label noise to the three original datasets in Table 1, following standard protocol in learning from noisy label studies. The results is shown in the following table. (20% noise means that 20% of the training data are randomly assigned to false labels.)\n\n| Variants              | Air-15%  | Air-30%  | Air-50%  | Air-100% | Car-15%  | Car-30%  | Car-50%  | Car-100% | CUB-15%  | CUB-30%  | CUB-50%  | CUB-100% |\n| --------------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Fine-tuning  clean    | 41.6     | 57.8     | 68.7     | 80.2     | 41.1     | 65.9     | 78.4     | 87.8     | 51.2     | 64.6     | 74.6     | 81.8     |\n| **LSG clean**         | **55.6** | **72.0** | **79.5** | **86.7** | **55.4** | **75.5** | **83.8** | **90.7** | **57.7** | **70.6** | **77.5** | **82.2** |\n| Fine-tuning 20% noise | 27.2     | 38.8     | 50.7     | 62.6     | 21.7     | 43.6     | 58.1     | 71.8     | 36.2     | 50.1     | 57.4     | 69.7     |\n| **LSG 20% noise**     | **41.5** | **54.8** | **61.9** | **72.6** | **36.3** | **54.3** | **67.9** | **80.8** | **42.1** | **57.3** | **64.2** | **73.8** |\n| Fine-tuning 40% noise | 15.7     | 20.7     | 33.7     | 44.3     | 13.9     | 27.8     | 37.7     | 51.3     | 23.0     | 35.2     | 44.0     | 52.6     |\n| **LSG 40% noise**     | **27.5** | **35.6** | **44.7** | **52.8** | **23.4** | **38.0** | **46.4** | **58.6** | **27.9** | **39.5** | **47.2** | **56.9** |\n\nThe results show that both methods are influenced by the label noise and their performance degradation is observed. Still, LSG outperforms fine-tuning baseline significantly on each noise level, and achieves relatively less accuracy degradation from clean label scenarios. The possible reason behind such robustness is that **LSG regularizes the image feature space to maintain the category relationships, which prevents the feature encoder learns heavily distorted feature space by overfitting to the noisy data**. Moreover, methods that identifiy and remove noisy labels can be integrated to LSG on these scenarios to improve its performance.\n\nThis experiment shows the potential of using LSG on scenarios with low quality labels. Still, we acknowledge that there are other scenarios where label quality is not satisfying, and we will leave it for future study. \n\n**Q2. Extension to semantic segmentation task.** \n\nA2. Good question! We are actively investigating the extension of LSG to semantic segmentation. Table presented below demonstrates the results of LSG applying on the classical Pascal VOC dataset under semi-supervised learning. The labeling ratio is 1/16, 1/8 and 1/4.  Our current extension involves:\n\n- Both losses $\\mathcal L_{align}$ and $\\mathcal L_{r}$ are applied on the feature map produced by the ResNet-50 backbone in a similar manner as in classification tasks. \n- Our key modification is to **group the neighboring pixel features together to reduce the number of features added to the language semantic graph each time**, since there will be too many new nodes if every pixel feature is calculated independently for dense prediction tasks. In fact, if all the pixel features are counted individually, the total number of new nodes within the mini-batch will exceed the number of original node of label embeddings, which affects the alignment.\n\nWe can observe from the following table that LSG brings improvement to the baseline in data-efficient scenarios for segmentation. As it is just a preliminary study, we will continue the investigation of making extension of the proposed LSG. \n\n|Network| Method                 | 1/16 (662 labeled) | 1/8 (1323 labeled) | 1/4 (2645 labeled) |\n|--| -- | :--: | :--: | :--: |\n|DeepLabV3+| Suponly                |        64.8        |        68.3        |        70.5        |\n|+| PseudoLabel (baseline) |        69.4        |        72.1        |        73.9        |\n|ResNet-50| LSG (extended)         |      **71.8**      |      **74.6**      |      **75.3**      |\n\nWe welcome further questions that you find worth discussing!"}, {"review_id": "U9kjM2VXMa", "reviewer": "Reviewer_pDMD", "summary": "The paper addresses the importance of labels’ semantic meanings when training models. First, the framework constructs an LSG graph. Node features are text embeddings generated by language models, and the similarity matrix constructs edges. After that, a GCN is trained to aggregate node features of the LSG with the node classification loss. The GCN will be used for data-efficient learning afterward (regularization loss and alignment loss).", "questions": "* How to make the inference? Is GCN used in the inference or only the primary model is used?\n* I am curious about the stability of the training. The augmented graph contains a batch of inputs, possibly introducing interference between samples after GCN. Will this affect the training quality/stability? For example, will the training be affected much if we change the batch size or random seed?\n* Is the primary model feature encoder the same as the model used to generate node embeddings? I.e., are “LM” and “F” in the figure 1 the same?\n", "limitations": "Authors do not include limitations. Some limitations I can think of:\n1. The current structure may have much more training burden than vanilla fine-tuning.\n2. The method is only applicable for classification methods.\n3. Authors could also try experiments on text tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation and proposed method is sound.\n* The details are well illustrated and are easy to understand.\n* Experiments cover a variety of tasks and well support the claims. Ablation studies also show the effectiveness of each component.\n", "weaknesses": "* Since the LSG is the critical component of the proposed framework, the quality analysis of LSG is preferred. For example, a visualization of LSG showing how well it grasps the semantics of labels.\n* The notion should be clearer. The F, C and $\\mathcal{C}$, $\\mathcal{F}$ are confusing when first reading. I would suggest using subscripts to distinguish them.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback and comments on our submission. Please find our responses your questions below.\n\n**Q1. Quality analysis of LSG.** \n\nA1. We provide two analysis to thoroughly evaluate LSG. \n\n- We show the T-SNE visualization of the initial node embeddings and the GCN refined node embeddings based on LSG (see Figure 3(a)). We can see that in the initial embedding space there exists a few \"prompt clusters\" where label embeddings from different categories are clustered because of some poorly-crafted prompt templates. On the other hand, **the GCN refined embedding space consists of cleaner and more compact clusters, demonstrating that LSG improves the quality of the label embeddings** (which will in turn benefit the primary network via loss $\\mathcal L_{align}$ and $\\mathcal{L}_r$). \n- Secondly, please recall that loss $\\mathcal{L}_r$ utilizes image feature refined by LSG (i.e., $\\mathcal{F}(\\tilde{h})$) as target to supervise the original one. Therefore, the improvement brought by $\\mathcal{L}_r$ gives credit to the rich semantics captured in LSG. \n|Variant |Air-15%|Car-15%|CUB-15%|\n|--|:--:|:--:|:--:|\n|LSG w/o $\\mathcal L_r$ | 44.2 |43.3|53.9|\n|LSG | **55.6** | **55.4** | **57.7**|\n\n**Q2. Change of notations.** \n\nA2. Thanks for your suggestion. We change the notation $\\mathcal{C}$ and $\\mathcal{F}$ to $\\mathcal{C}_g$ and $\\mathcal{F}_g$ to better represent the classifier and the encoder of the GCN, and we keep the notation of $C$ and $F$ for the primary network for simplicity.\n\n**Q3. About model inference.** \n\nA3. The GCN (and correspondingly the projector) is not used during inference and the classification result is still produced by the classifier. For this reason, **no extra computational cost is needed in inference time**.\n\n**Q4. Training  stability.** \n\nA4. To test the stability of our method, we conduct three experiments. \n\n- We plot the three loss curves for the primary training stage in Figure 3(b) and show that all the losses decrease steadily and converges. \n- We run LSG five times with different random seed. The accuracy curve in Figure 3(c) shows that LSG maintains a small accuracy variance throughout the training process. \n- As demonstrated in the following table, batch size adjustment within a certain range has minimal effect on the performance. In fact, the sample-sample interaction within a mini-batch, which is also mentioned by reviewer A9AK, is found to be beneficial to the performance. The reason is that such interaction allows each image feature to aggregate more information.\n\nThe table below shows the performance of LSG with different batch size.\n| Batch Size| StanfordCars-15% | StanfordCars-30% | StanfordCars-50% | StanfordCars-100% |\n| -- | :--: | :--: | :--: | :--: |\n| 24 | 55.3 | 75.4 | **84.1** | 90.7 |\n| 48 (in paper) | 55.4 | **75.5** |83.8|90.7|\n| 64 | **55.5** | 75.2| 83.8 | **90.9** |\n\nThe table below shows comparison between enabling sample-sample interaction (w/ $M$) or not (w/ identity matrix $I$).\n| Variants| Aircraft-15% | Aircraft-30% | Aircraft-50% | Aircraft-100% |\n| -- | :--: | :--: | :--: | :--: |\n| LSG w/ $I$ | 55.0 | 71.4 | 79.1 | 86.1 |\n| LSG w/ $M$ | **55.6** | **72.0** | **79.5** | **86.7** |\n\n**Q5. LM is different from $F$.** \n\nA5. \"LM\" is a pre-trained language model such as BERT that encodes sentences with concepts in to text embeddings (node embeddings). The primary model feature encoder $F$ is a model from a non-language modality such as ResNet. We will revise our paper to clarify the meaning of \"LM\" in figure 1.\n\n**Q6. About the extra training burden compared to vanilla fine-tuning.** \n\nA6. The proposed LSG does not bring too much training burden due to the following reasons. \n\n- Firstly, both GCN and the projector added to the network are lightweight modules with minimal parameters compared to the primary model. \n- Secondly, training GCN in stage one only requires less than 3 minutes, and the trained GCN can always be reused to guide different primary models on the very same task. Moreover, computing $\\mathcal L_{align}$ and $\\mathcal L_r$ is fast and will not increase much training time.\n\nThe following table shows the comparison between LSG and vanilla fine-tuning (ResNet-50 is adopted as backbone). It is concluded that the extra training cost brought by LSG is not overwhelming.\n\n|Method | Parameters (training) | Parameters (inference) | Training time |\n| -- | :--:|:--:|:--:|\n| Fine-tuning| 25.6M | 25.6M|31.9 min|\n|LSG|28.8M |25.6M|32.2 min|\n\n**Q7. Extension to boarder application range.** \n\nA7. Good comment! In fact, we are actively investigating to extend LSG to image semantic segmentation task. As shown in table below, we conduct preliminary experiments on standard Pascal VOC dataset under SSL setting with different labeling ratio, and find that LSG is also beneficial to data-efficient segmentation. We will leave further studies to future work.\n\n| Network| Method| 1/16 (662 labeled) | 1/8 (1323 labeled) | 1/4 (2645 labeled) |\n| -- | -- | :--: | :--: | :--: |\n| DeepLabV3+ | Suponly                |        64.8        |        68.3        |        70.5        |\n| +          | PseudoLabel (baseline) |        69.4        |        72.1        |        73.9        |\n| ResNet-50  | **LSG (extended)**        |      **71.8**      |      **74.6**      |      **75.3**      |\n\nAs for the text task, it may require certain modification of the current method in order to be applied. It is because the current semantic graph is constructed using pre-trained language model and thus may not be much helpful to other language models as the primary model. Still, we point out that alternative ways to build the semantic graph can be proposed and improve model performance on language models using our proposed alignment strategy in section 3.2. "}, {"review_id": "MRYP3F24uj", "reviewer": "Reviewer_3r7Y", "summary": "This paper employ a language semantic graph to cature the relationship among different class, with the hope to alleviate the requirement of extensive training data and, in particular, human supervision. Generally, this paper first build the language semantic graph  with the pre-trained language models. After that, the paper introduces two additional losses $L_align$ and $L_r$ that exploit the discarded label\n semantic information for complement. ", "questions": "1. The final loss function in eq.(7) don't include the loss L_node, and what is the influce of L_node?\n\n2. The language semantic graph is built with the embedding of label, so why you not using the embedding to capture the smantic relathionship among labels rather than using a discrete semantic graph?\n\n3. I'm confused on m the size of the prompt set, what is the prompt set? \n\n4.The section 3.2 are very confused for me, why the two proposed loss can help  data effective learning.  The author doesn't give a detail description for eq(5) and eq(6). And I'm very confused on why need the proposed two align loss, and the expective result of the proposed align loss. \n\n\n", "limitations": "The developed method are depend on pre-trained language model, and sometimes the labels in a task maybe not have an effective embedding with the pretrained language models.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "I have carefully read this papar some times, and I'm interested in injecting knowledge to the model for effective learning. The strengths can be describued as follow:\n\n1. The idea is meaningful and interesting that employing prior knowledge  for effective learning, and the proposed method is novel, at least I've not read some related paper. \n\n2. The experiment results  are effective compred with baseline models. ", "weaknesses": "\n1. The author not provide the code, and I'm very confued on the key section.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your questions. We'll do our best to explain our method here and will thoroughly revise the paper to improve its clearity. \n\n**Q1. Code for better understanding.** \n\nA1. Our code is now provided to AC, please refer to it for better understanding.\n\n**Q2. About the effect of loss $\\mathcal L _{node}$.** \n\nA2. The proposed LSG consists of two stages. In the first stage, we train a GCN on the language semantic graph $\\mathcal G$. In the second stage, we train the primary model $F$  (i.e., ResNet) with the aid of the trained GCN. The loss $\\mathcal L_{node}$ is only used for training GCN in the first stage, thus it is not appeared in eq.(7) for the second stage. \n\n$\\mathcal L_{node}$ optimizes the GCN to correctly classify each node on the semantic graph $\\mathcal G$. By doing this, each node processed by the GCN aggregates discriminative and more comprehensive semantic information from neighboring nodes (see T-SNE visualization of the label embeddings in figure 3(a)).\n\nThe trained GCN is then fixed as a graph processor and deployed on the augmented semantic graph $\\mathcal G_{aug}$ to pass semantic information to new nodes in the second stage. More specifically, eq. (5) and eq. (6)  leverage the ($\\mathcal L_{node}$) trained GCN to guide the feature learning of the primary model, which will be explained later.\n\n**Q3. About the prompt set.** \n\nA3. The prompt set is a collection of hand-crafted prompts where each one can complete a given concept into sentence. For example, one prompt can be 'This is a photo of {}' or 'This is a sculpture of {}' where we can replace '{}' by categories. The size of the prompt set $m$ determines the number of different sentences we can create for each concept, which consequently determines the number of text embeddings. (That is why we have total $|T|=mK$ embeddings for $K$ classes.) Via prompt set, we can obtain multiple different embeddings corresponding to the same category. The advantage of such diversity is fully utilized in the constructed semantic graph that will be explained next.\n\nTable below shows how $m$ affects the accuracy. We choose $m=20$ to achieve a balance between performance and cost.\n\n||$m=$5 |10|20|40|80|\n|--|:--:|:--:|:--:|:--:|:--:|\n|Air-15%|54.8|55.0|55.6|55.7|55.7|\n\n**Q4. Using a semantic graph instead of the original label embeddings** \n\nA4. The proposed semantic graph provides a better supervision than using the original embeddings directly does. The reason is three fold. \n\n- First, the semantic graph is built upon label embeddings (i.e., $\\mathcal K^{(0)}=\\mathcal K$), and thus includes all the information that label embeddings have. \n- Second, information can be passed on among neighboring nodes to produce more discriminative and comprehensive node embeddings (see T-SNE), and further be transferred to new node of image features through new links. Therefore, our semantic graph, with the aid of the GCN trained by $\\mathcal L_{node}$, allows the image features capture semantics from label embeddings. \n- Third, we empirically verify that using semantic graph is more effective than simply supervise primary model features by label embeddings using different classical alignment strategies. As shown in table below, both the prototype alignment and contrastive alignment method that only leverage initial label embeddings perform worse than LSG. \n\n||Air-15%|Air-30%|Air-50%|Air-100%|\n|--|:--:|:--:|:--:|:--:|\n|Prototype align.|51.1|68.8|76.5|84.6|\n|Contrastive align.|54.2|70.7|78.3|85.1|\n|LSG|**55.6**|**72.0**|**79.5**|**86.7**|\n\n**Q5. About Section 3.2**. \n\nA5. This section describes the process of injecting label semantic knowledge into the primary model. The motivation behind it is that common practice of cross-entropy loss turns labels into one-hot vectors, during which the semantic relationship between categories are erased. (All these one-hot vectors are perfectly perpendicular to each other.) Thus, to introduce the lost semantic information into the network makes the better use of the training data and promote data-efficient learning. Take vision model as an example, our goal is to align the image features towards the semantic space of label embeddings provided by the PLM, in which the relationship between categories are better reflected than in the one-hot space. \n\nThe alignment is conducted both implicitly (via eq. (5)) and explicitly (via eq. (6)), and their effect are explained as follows:\n\n- Eq. (5) is the node classification loss on new nodes from image features $h$. Since that the GCN trained by $\\mathcal L_{node}$ in the first stage can already classify label embeddings (original nodes) correctly and it is now set fixed, lowering the cost of eq. (5) forces the image features become similar to their corresponding label embeddings, achieving implicit feature alignment. \n- Eq. (6) computes the $l_2$ distance between original image features and the new features encoded by the GCN, therefore explicitly pushes the former one towards the latter one (which is detached). Note that the new features have aggregated label embeddings from their neighbors in the augmented semantic graph, thus are desirable representations for the vision model. Please refer to the algorithm provided in the pdf for the complete training process.\n\nThe ablation study conducted in the paper shows the effectiveness of both losses, and we present it here.\n\n||LSG w/o $\\mathcal L_{align},\\mathcal L_r$|LSG w/o $\\mathcal L_{align}$|LSG w/o $\\mathcal L_r$|LSG|\n|--|:--:|:--:|:--:|:--:|\n|Air-15%|41.6|44.2|50.7|**55.6**|\n\n**Q6. On tasks without effective language embeddings.**\n\nA6. This is a solid concern. Although LSG cannot be directly applied to these tasks, we argue that our idea of semantic graph guided knowledge transfer is applicable if we can construct a graph reflecting the relationship between classes without PLM. In this way, our proposed method can once again guide the model to learn the category relations.\n\nWe hope these responses address your concerns. "}, {"review_id": "ezwRZPPJaa", "reviewer": "Reviewer_A9AK", "summary": "This paper proposed a general framework for exploiting semantic information within labels in classification tasks to improve the performance of deep nueral networks. The framework is both task-agnostic and model-agnostic, making it applicable to a wide range of classification tasks and modalities. The framework first concatenates label concepts/descriptions with prompting templates, and feed these templates into a frozen language models to generate embeddings. These embeddings are subsequently used to construct a semantic graph, which is trained using a graph convolutional network (GCN). When applying the semantic graph for guiding the training process, two regularization objective is added based on the semantic grpah in order to enhance the model with label semantics. The goal of the whole framework is similar to the prompt learning in NLP, while its capability of applying to tasks in other modality improve its novelty and contributions. In experiments, the authors show that their proposed framework successfully outperforms many other baselines based on transfer learning, semi-supervised learning and data augmentation.", "questions": "My questions are mainly about the generalization ability of the proposed framework:\n\n\n1. Can you provide more examples of prompt templates used for generating embeddings?\n\n2. What is the performace of data on some large-scale datasets (i.e. when the data source is rich)? In Co-tuning's paper, they also conducted experiments COCO-70, which is considered a large-scale dataset with 1k samples per class.\n\n3. What is the ituition behind adding sample-sample interaction in the augmented graph (i.e. the matrix M). Does it bring any benefit to LSG?\n\n\n4. I noticed that all the tasks used in the experiments have at least 50 categories. I am curious if the LSG framework can be applied to binary classification tasks or tasks with a small number of categories (<10). In my opinion, it may be difficult to apply LSG to binary classification because the descriptions of the labels are generally two sides of one statement. For tasks with fewer than 10 categories, it may be challenging to train a GCN. It would be great if the authors could have some disscussion about these tasks.\n\n", "limitations": "The authors did not discuss limitations of the work. I do not see any potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper itself is clearly written and presented, enabling readers to understand the proposed framework easily.\n2. The proposed framework for incorporating semantic information of labels for tasks in various modalities is novel and widely applicable.\n3. From the experiment section, the performance of the proposed framework significantly outperforms several baselines based on transfer learning, semi-supervised learning, and data augmentation.", "weaknesses": "The proposed framework requires that the target task be a classification task and that each class has explicit and rich semantics. If a class does not have an explicit semantic meaning, the LSG may not work. This requirement limits the applicability of the LSG framework to many classification tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback and comments on our submission. Please find our responses your questions below.\n\n**Q1. Scenarios where label semantics are weak.** \n\nA1. Thanks for your insightful comment. It is a solid concern, as there exists certain scenarios like defect detection where different classes are merely represented as \"defect 1\", \"defect 2\" and etc. In this case, the label embedding obtained from pre-trained language model is less meaningful. \n\nHowever, we argue that **the idea of SG (semantic graph) guided knowledge transfer may still be applicable if we can construct a graph reflecting the relationship between these classes without the help of the language model**. For example, if we know that some types of the defects are caused by similar machine fault, we can use these relations to build graphs like stochastic block model, and still apply the proposed method to guide the primary model with the graph topology. \n\nTo demonstrate the wide applicability of our method, we investigate an alternative scenario where labels in the training data contain noise. It shares certain similarity with the situation you mentioned, as part of the training labels can no longer reflect the true nature of the samples. We randomly add label noise to the clean datasets and compare LSG with fine-tuning baseline. \n\nThe table below shows that LSG still outperforms fine-tuning baseline significantly, and results in less accuracy degradation from clean label scenarios. (20% noise means that 20% of the training data are randomly assigned to false labels.)\n\n| Variants| Air-15%  | Air-30%  | Air-50%  | Air-100% | Car-15%  | Car-30%  | Car-50%  | Car-100% | CUB-15%  | CUB-30%  | CUB-50%  | CUB-100% |\n| -- | -- |-- |-- |--|--|--| --| -- |--|--|--|--|\n| Fine-tuning  clean    | 41.6     | 57.8     | 68.7     | 80.2     | 41.1     | 65.9     | 78.4     | 87.8     | 51.2     | 64.6     | 74.6     | 81.8     |\n| **LSG clean**         | **55.6** | **72.0** | **79.5** | **86.7** | **55.4** | **75.5** | **83.8** | **90.7** | **57.7** | **70.6** | **77.5** | **82.2** |\n| Fine-tuning 20% noise | 27.2     | 38.8     | 50.7     | 62.6     | 21.7     | 43.6     | 58.1     | 71.8     | 36.2     | 50.1     | 57.4     | 69.7     |\n| **LSG 20% noise**     | **41.5** | **54.8** | **61.9** | **72.6** | **36.3** | **54.3** | **67.9** | **80.8** | **42.1** | **57.3** | **64.2** | **73.8** |\n| Fine-tuning 40% noise | 15.7     | 20.7     | 33.7     | 44.3     | 13.9     | 27.8     | 37.7     | 51.3     | 23.0     | 35.2     | 44.0     | 52.6     |\n| **LSG 40% noise**     | **27.5** | **35.6** | **44.7** | **52.8** | **23.4** | **38.0** | **46.4** | **58.6** | **27.9** | **39.5** | **47.2** | **56.9** |\n\nIn summary, we acknowledge that this is an important future direction for LSG  and argue that our proposed method has the potential to adapt to these more challenging scenarios. We will add discussion on this issue in our paper to motivates future works.\n\n**Q2. Examples for prompt templates.** \n\nA2. We adopt the first twenty original hand-crafted prompt templates provided by CLIP, which includes:\n- \"This is a bright photo of a {}\", \n- \"This is a bad photo of a {}\", \n- \"This is a photo of many {}\", \n- \"This is a sculpture of a {}\", \n- \"This is a tattoo of a {}\" etc. \n\nWe discover that this set of prompt templates works well even for the video and audio task, and thus do not manually design new templates. \n\n**Q3. Performance on large-scale COCO-70 dataset.** \n\nA3. Following Co-tuning, we evaluate LSG on COCO-70 (DenseNet-121) and report the results in table below. **Our method is superior than Co-tuning under every labeling ratio on this large-scale benchmark**, implying a boarder application range of LSG.\n|Method| Air-15% |Air-30%|Air-50%|Air-100%|\n| -- | :--: | :--: | :--: | :--: |\n|Fine-tune| 76.60|80.15|82.50|84.41|\n|Co-tuning|77.64|81.19|83.43|85.65|\n| LSG| **79.50** | **82.33** | **84.14** | **86.11** |\n\n**Q4. About sample-sample interaction.** \n\nA4. The interaction described in adjacency matrix $M$ creates additional path for knowledge transfer for each image feature, which brings impact to the optimization of both losses $\\mathcal L_{align}$ and $\\mathcal L_{r}$. \n\n- For $\\mathcal L_{align}$, it means that each image features are affected not only by label embeddings but also image features from the same class. To guarantee correct node classification under such influence, the consistency between image features are promoted. \n- On the other hand, the existence of these sample-sample interaction means that the image feature after GCN refinement (i.e., $\\mathcal F(\\tilde{h})$) naturally incorporates features from other images and consequently have richer information. Therefore, $\\mathcal L_{r}$ obtains better targets for the original feature $h$ to learn. The following ablation study, in which we replace $M$ by an identity matrix $I$ , supports our argument.\n\n| Variants| *Aircraft-15%* | *Aircraft-30%* | *Aircraft-50%* | *Aircraft-100%* |\n| -- | :--: | :--: | :--: | :--: |\n| LSG w/ $I$ | 55.0 | 71.4 | 79.1 | 86.1 |\n| LSG w/ $M$ | **55.6** | **72.0** | **79.5** | **86.7** |\n\n**Q5. LSG on tasks with fewer categories.** \n\nA5. Good question. We must say that it is merely a coincidence that all the tasks we evaluate in the paper involve many categories. Here we show that **LSG is still semantic meaningful and works well when the category number is small**. We evaluate single domain generalization on another classical benchmark PACS that has only 7 categories (see Table C in pdf). We find surprisingly that LSG brings more significant performance improvement than on OfficeHome. \n\nAs for the case of binary classification, it is correct that the category relationship captured by the semantic graph is no longer useful, since the only thing that matters then is to discriminate between the two classes. However, the label embeddings provided by the pre-trained language model may still be useful to prevent feature distortion."}]

Author rebuttal:
Dear Reviewers and Area Chair,

We extend our utmost gratitude for your dedicated commitment in meticulously assessing our manuscript and for providing us with your profound insights. Your considered evaluations serve as a testament to the rigor and importance of our research.

We are heartened by the reviewers' recognition of the novelty (biPo, 3r7Y, A9AK), strong motivation (ZDzt, pDMD), clear presentation (biPo, pDMD, A9AK), and significant contribution in the results (ZDzt, biPo, pDMD, 3r7Y, A9AK) of our work.

In our comprehensive individual responses to the reviewers, we have diligently addressed all raised questions and concerns. We eagerly welcome any further inquiries that may arise.

With highest regards,

Authors

Meta-review: The authors propose an interesting idea of using label information via a semantic graph as a regularizer and conduct many experiments across 3 domains (image, video, audio) and 3 out of 5 reviewers support acceptance.

Both reviewers 3r7y and ZDzt expressed the same concern, that it is intuitively unclear why the approach of using a discrete semantic graph based on label embeddings is better than just using the label embeddings directly. 

Given the substantial number of experiments provided by the authors in the rebuttal, I favor accepting the paper.
Decision: Accept (poster)