Zero-Shot Learning with Common Sense Knowledge Graphs
Authors: Nihal V. Nayak
Stephen Bach
Venue: Accepted by TMLR ()

Abstract: Zero-shot learning relies on semantic class representations such as hand-engineered attributes or learned embeddings to predict classes without any labeled examples. We propose to learn class representations by embedding nodes from common sense knowledge graphs in a vector space. Common sense knowledge graphs are an untapped source of explicit high-level knowledge that requires little human effort to apply to a range of tasks. To capture the knowledge in the graph, we introduce ZSL-KG, a general-purpose framework with a novel transformer graph convolutional network (TrGCN) for generating class representations. Our proposed TrGCN architecture computes non-linear combinations of node neighbourhoods. Our results show that ZSL-KG improves over existing WordNet-based methods on five out of six zero-shot benchmark datasets in language and vision.

Published in Transactions on Machine Learning Research (08/2022)
Zero-Shot Learning with Common Sense Knowledge Graphs
Nihal V. Nayak
waayak20cs. www.://:wwbbbbb..ooooooommmmmmm. edu
Deportment of Computer Science Brown University
Stephen H. Bach
sbach@cs. brown. edu
Department of Computer Science Brown University
Reviewed on OpenReview: https: // openreview. net/ met/teeruuueeee forum? id- 212u66cXpH
Abstract
Zero-shot learning relies on semantic class representations such as hand-engineered attributes or learned embeddings to predict classes without any labeled examples. We propose to learn class representations by embedding nodes from common sense knowledge graphs in vector space. Common sense knowledge graphs are an untapped source of explicit high-level knowledge that requires little human effort to apply to range of tasks. To capture the knowledge in the graph, we introduce ZSL-KG, general-purpose framework with novel transformer graph convolutional network (TrGCN) for generating class representations. Our proposed TrGCN architecture computes non-linear combinations of node neighbourhoods. Our results show that ZSL-KG improves over existing WordNet-bass based methods on five out of six zero-shot benchmark datasets language and vision. The code is available at httlps:/ggtthub.com BaeatBBseeaaeh//aaaeeaa /zsl-g
1 Introduction
Zero-shot learning training strategy which allows machine learning model to predict novel classes without the need for any labeled examples for the new classes RRoeea--Pareees & Torr 2015 Socher et al.
2013 Wang et al. 2019). These models are trained on set of labeled examples from seen classes, along with their class representations. During inference, new class representations are provided for the unseen classes.
Previous zcro-shot learning systems have used hand-cngincered attributes (Farhadi et al. 2009 Lampert
et al. 2013), pretrained embeddings (Frome et al. 2013 and learned embeddings (e.g., sentence embeddings) RRed et al. 2016) as class representations.
Previous approaches for class reprosentations have various limitations. Attributo-based methods provide rich features and have achieved state-of-the-ae results several zero-shot object classification datasets, but the attributes have to be fixed ahead of time for the unseen classes and cannot adapt to new classes beyond the dataset. Furthermore, creating attribute datasets can take to thousands of hours of labor (Zhao et al.) 2019). Finally, attribute-basee methods may not be readily applicable to tasks in language, as they might require greater nuance and flexibility (Gupta et al. (020).. Alternatively, pretrained embeddings such as GloVe (Pennington et al.] 2014) and Word2Vee (Mikolov al., 2013) offer the ficxibility of casily adapting to new classes but rely on unsupervised training on large corpora which may not provide distinguishing characteristics necessary for zero-shot learning. Many methods lie within this spectrum and learn class representations for zero-shot tasks from descriptions such as text and image prototypes.
Recently, there is growing interest in methods using graph neural networks on the ImageNet graph, a noun subset of the WordNet graph, to learn to map nodes to class representations (Wang et al. 2018 These graph-based methods have achieved strong performance on zero-shot object classification. They offer the benefits of high-level knowledge from the graph, with the flexibility of pre-trained embeddings. They are general-purpose and applicable to broader range of tasks beyond object classification, since show
1

---

Published in Transactions on Machine Learning Research (08/2022)
that they can be adapted to language datasets well. However, the ImageNet graph is specialized to object-type hierarchy, and may not provide rich features as suitable for a wide range of downstream tasks in multiple domains.
In our work, we propose to learn class representations better suited for a wider range of tasks from common sense knowledge graphs. Common sense knowledge graphs (Liu & Singh 2004 Speer et al. 2017 Zhang et al. 2020; Iliieski et al., 2020) are an untapped source of explicit high-level knowledge that requires little human effort to apply to range of tasks. These graphs have explicit edges between related concept nodes and provide valuable information to distinguish between different concepts. However, adapting existing zero-shot learning frameworks to Iearn class representations from common sense knowledge graphs is challenging in several ways. GCNZ (Wang et al., 2018) learns graph neural networks with a symmetrically normalized graph Laplacian and requires the entire graph structure during training, i.e., GCNZ is not inductive. Common sense knowledge graphs can be large (2 million to 21 million edges) and training a graph neural network on the entire graph can be prohibitively expensive. DGP (Kampffneyer et al., 2019) is an inductive method and aims to generale expressive class represenlations, but assumes directed acyclic graph such as WordNet. Common sense knowledge graphs do not have directed acyclic graph structure.
To address these limitations, we propose ZSL-KG, a framework with novel transformer graph convolutional network (TrGCN) to learn class representations. Graph neural networks learn to represent the structure of graphs by aggregating information from each node's neighbourhood. Aggregation techniques used in GCNZ, DGP, and most other graph neural network approaches are linear, in the sense that they take a (possibly weighted) mean or maximum of the neighbourhood features. To capture the complex information in the common sense knowledge graph, TrGCN learns transformer-basee aggregator to compute non-linear combination of the node neighbours and increases the expressiveness of the class representation (Hamilton et al. 2017a). Our framework is also inductive, i.e., the graph neural network can be executed on graphs that are different from the training graph, which is necessary for generalized zero-shot learning under which the test classes are unknown during training.
Our main contributions are the following:
1. We propose to learn to map nodes in common sense knowledge graphs to class representations for zero-shot learning.
2. We present ZSL-KG, a framework based on graph neural networks with a novel transformer graph convolutional network (TrGCN). Our proposed architecture learns permutation invariant non-linear combinations of the nodes' neighbourhoods and generates expressive class representations.
3. We demonstrate that graph-based zero-shot learning frameworks originally designed for object classification are also applicable to language Lasks.
4. ZSL-KG achieves new state-of-the-ee
accuracies for zero-shot learning on the OntoNotes Giilick et al.
2014 BBN (Weischedel & Brunstein 2005), attribute Pascal Yahoo (aPY) (Farhadi et al. 2009]
and SNIPS-NLU (Coucke et al.) 2018) datasets. Finally, ZSL-KG also outperforms existing WordNet-
based zero-shot learning frameworks ks on the ImageNet 22K dataset (all classes used for testing) Deng
et al. 2009). while maintaining competitive performance on the Animals with Attributes 2 (AWA2) dataset (Xian et al. 2018a).
2 Related Work
We broadly describe related works on zero-shot learning, graph neural networks, and common sense knowlodge graphs.
Zero-Shot Learning. Zero-shot learning has received increased interest over the years in both language and vision (Farhadi et al. 2009] Brown et al., 2020). In our work, we are focused on zero-shot fino-grained entity typing, intent classification, and object classification. Zero-shot fine-grained entity typing has been previously studied with a variety of class representations (Yogatama et al.) 2015 Yuan & Downey 2018 Obeidat et al.) 2019). ZOE (Zhou et al. (2018) is a spocialized method for zero-shot fino-grained entity typing and shows accuracy competitive with supervised methods. It uses hand-crafted type definitions for each dataset and tuned thresholds, developed with knowledge of the unseen classes, which makes them transductive zero-shot.
2

---

Published in Transactions on Machine Learning Research (08/2022)
method. Our work focuses on sraph-based methods in zero-shot learning where unseen types are not revealed during training or tuning. There has been much work on zero-shot object classification (Frome et al. 2013 Lampert et al. 2013 Wang et al.) 2018 Xian et al. 2018b). Recent works in zero-shot learning have used graph neural networks for object classification (Wang ct al. 2018 Kampffmeyer et al.) 2019). In our work, we extend their approach to common sense knowledge graphs to generate class representations with a novel transformer graph convolutional network. Liu et al.] (2020a) proposed a graph propagation mechanism for zero-shot object classification. However, they construct the graph by leveraging the predefined attributes for the classes, which are not easily adapted to language tasks. HVE (Liu et al.) 2020b) learns similarity between the image representation and the class representations in the hyperbolic space. However, the class representations are not learned with the task instead they are pretrained hyperbolic embeddings for Ve and WordNet whereas, in our work, we focus on methods that learn class representations explicitly from the knowledge graphs in the task. More recent work on zero-shot object classification (Chen et al. 2021 2022alb b) use transformer-basee networks with attributes whereas we learn a transformer graph convolutional networks with a common sense knowledge graph and outperform them on both AWA2 and aPY datasets.
Other notable works in zero-shot learning include text classification (Chang et al.
2008
Yin et al. 2019).
video action recognition (Gan et al. 2015), machine translation (Johnson et al.
2017). and more (Wang]
ct al. 2019
Recent work shows that large-scale models like CLIP (Radford et al. 2021 and GPT-3 (Brown et al. 2020 exhibit zero-shot generalization to new tasks and datasets. However, a key challenge is the potential overlap of the seen and unseen datasets. For example, analysis of the training data used in CLIP showed that 24 out of the 35 held-out datasets detected overlap (Radford et al., (021]. ). In contrast, we systematically study the zero-shot generalization of classos without any overlap in the seen and unseen classes. Furthermore, these models use lext-based prompts to represent classes that may be not provide sulficient control to represent complex fine-graine greinerrind classes. In contrast, ZSL-KG offers a flexible way of representing unseen fine- ane-ggrined and complex classes in a knowledge graph. One limitation to note is that ZSL.KG requires the classes to be mapped to the knowledge graph. Fine-grained object classification datasets may contain classes that might be missing from the off-the-shelf Cen graph. For example, Concept does not have a node for the class Forster's Tern in the CUB dataset (Wah et al., (2011). To extend ZSL-KG to rare and domain-speeific classos, we would need specialized knowledge graph.
Graph Neural Networks Graph neural networks learn node embeddings that reflect the structure of the graph (Hamilton et al.) 2017b). Recent work on graph neural networks has demonstrated significant improvements for several downstream tasks such as node classification and graph classification (Hamilton
et al.
2017a Kipl & Welling 2017 Marcheggiani & Titov] 2017 Schlichtkrul et al. 2018 Veličković et a
2018
Wu et al.) 2019 Shang et al. 2019: Vashishth et al., 2020 0. In this work, we introduce transformer graph convolutional networks for zero-shot learning. Since our preprint appeared on ArXiv, several variants of graph transformers have been proposed in the literature (Dwivedi & Bresson 2021 Kreuzer et al. 2021
Ying et al. 2021 Mialon et al. 2021 Dwivedi et al., 2022). The primary motivation of their work is to model long-range interactions in the graph. They assume all the nodes are connected to each other and learn a transformer with positional and structural representations over the entire graph. However, this increases the computational complexity of the model. Suppose we have a graph with n nodes, traditional graph neural networks have a complexity of O(n) whereas graph transformers have complexity of O(?)) to compute embeddings for all the nodes in the graph. In contrast, our proposed TrGCN increases the expressivily of the node representations by operating on the local neighbourhood with a complexity of O(m²... n) where << is the maximum number of node neighbours during training and testing. Prior work (Hu et al.) 2020 Yun et al. 2020) has also considered transformers as a method to learn meta-paths in heterogencous graphs rather than as a neighbourhood aggregation technique. Finally, several diverse applications using graph neural networks have been explored: fine-grained entity typing (Xiong et al., 2019), text classification
Yao et al. 2019). reinforcement learning (Adhikari et al. 2020) and neural machine translation (Bastings
et al. 2017) For more in-depth review, we point readers to Wu et al. (2021).
Common Sense Knowledge Graphs. Common sense knowledge graphs have been applied to range of
tasks [Lin et al.
2019 Zhang el al. 2019h Bhagavatula et al.] 2019 Bosselut et al. 2019 Shwartz et al.) [020]
Yasunaga et al.
2021 ConceptNet has been used for transductive zero-shot text classification as shallow
3

---

Published in Transactions on Machine Learning Research (08/2022)
features for class representation (Zhang et al., 2019b; along with other knowledge sources such as pretrained emnebeddings and textual description, which differs from our work we generate dense vector represent an from Concept.Net with TrGCN. Finally, TGG (Zhang et al., 2019a) uses common sense knowledge graph and graph neural networks for transductive zero-shot object classification. TGG learns to model scen-unseen relations with a graph neural network. Their framework is transductive as they require knowledge of unseen classes during training and use hand-crafted attributes. In contrast, ZSL-KG is an inductive framework which does not require explicit knowledge of the unseen classes during training and learn class representations from the common sense knowledge graph.
3 Background
In this section, we briefly summarize zero-shot learning and graph neural networks.
Zero-Shot Learning. Zero-shot learning has several variations [Wang et al.) 2019). Formally, we have the training set {(1) y1), ..., (Ins yn)} where ye belongs to the set of seen classes Ys. In the conventional zero-shot learning setting (ZSL), we assign examples to the correct class(es) from the unseen classes Yu and Ys n Yv Ø. In generalized zero-shot learning (GZSL) setting, we assign examples to the correct class(es) from the set of seen and unseen classes YU+s = Ys UY. Based on prior work in each task, we choose to evaluate in conventional zero-shot learning, generalized zero-shot learning, or both.
Zero-shot classifiers are trained on the seen classes, but unlike traditional supervised learning, they are trained along with class representations such as attributes, pretrained embeddings, etc. Recent approaches learn a class encoder d(y) E I to produce vector-valuce class representations from an initial input, such as a string or other identifier of the class. (In our case, y is node in a graph and its k-hop neighborhood.)) During inference, the class representations are used to label examples with the unseen classes by passing the examples through an example encoder ((x) E Rdd and predicting the class whose representation has the highest inner product with the example represenlation.
Recent work in zero-shot learning commonly uses one of two approaches to learn the class encoder oly). One approach uses a bilinear similarity function defined by a compatibility matrix W E Rdxd Frome al. 2013 Xian et al. 2018b):
(1) The bilinear similarity function gives score for each example-class pair. The parameters of 6 W, and e are learned by taking a softmax over f for all possible seen classes E Ys and minimizing either the cross entropy loss or a ranking loss with respect to the true labels. In other words, f should give a higher score for the correct class(es) and lower scores for the incorrect classes. W is often constrained to be low rank, to reduce the number of learnable parameters (Yogatama al. 2015). Lastly, other variants of the similarity function add minor variations such as non-linearities between factors of W (Xian et al. 2016
f (0(z), W, p(v)) = A(x) T TW Woy).
The other common approach is to first train a neural network classifier in supervised fashion. The final fully connected layer of this network has a vector reprosentation for cach seen class, and the remaining layers are used as the example encoder ((x). Then, the class encoder ②(y) is trained by minimizing the L2 loss between the representations from supervised learning and Φ(y) (Socher et al., 2013 Wang et al. 2018). The class encoder that we propose in Scction can be plugged into either approach.
Graph Neural Networks. The basic idea behind graph neural networks is to learn node embeddings that reflect the structure of the graph (Hamilton et all 2017b). Consider the graph G EV,ER R), where V is the set of vertices with node features X, and (o;, r. 12) E are the labeled edges and r E R are the relation types. Graph neural networks learn node embeddings by iterative aggregation of the k-hop neighbourhood. Each layer of a graph neural network has two main components AGGREGATE, and COMBINE. (Xu et al. 2019):
i AGGREGATE ((n!-1) Yu N(v)))
(2)
where 4 E c is the aggregated node feature of the neighbourhood, h is the node feature in neighbourhood (()) of node including self loop. The aggregated node is passed to the COMBINE to
4

---

Published in Transactions on Machine Learning Research (08/2022)
legal
Transformer
legal
judge
Seen class
biL
Judge
lawyer
says Mr. Howorth partner in the Dallas office of Andrews & Kurth,..
lawyer
Dat product
person
advocate
advocate
person
Backprpaggtion Forvard Pass
Mean pooling
preach
Example
Class rep.
counsel
rep.
counsel
Score
Figure Left:A sample from the 2-hop neighbourhood for the concept lawyer from ConceptNet. Right: The ZSL-KG architecture. The text with the named entity is passed through the example encoder (biLSTM take the k--hop
with attention) to generate the example reprosentation. For the class reprosentation, neighbourhood and pass the nodes through their hop-specific transformer graph convolutional network (TrGCN). Next, we take the dot products between the example representations and the class representations to get compatibility scores. Finally, we train the example encoder and the class encoder by minimizing the task-specific loss over the true labels.
generate the node representation h E Rd for the 1-th layer:
h = COMBINE.") (nS-1) aa
(3)
h where D. is the initial feature vector for the node. Previous works graph neural networks for zero-shol learning have used GloVe (Pennington et al. 2014) to represent the initial features (Wang et al.) 2018). Finally, Hamilton et al. (2017a) proposed using LSTMs as non-linear aggregators. However, their outputs can be sensitive to the order the neighbours the input, i.e., they are not permutation invariant. For example, on the Animals with Attributes 2 dataset, we find that when given the same test image 10 times with different neighbourhood orderings, predictions 16% of the time (Appendix D). One recent work considers trying to make LSTMs less sensitive by averaging the outputs over permutations, but this significantly increases the compulational cost and provides only small boost to prediction accuracy Murphy et al. (2019). In contrast, our proposed TrGCN in ZSL-KG is non-lincar and naturally permutation invariant.
LSTM-based graph neural network outputs inconsist.ent
4 The ZSL-KG Framework
Here introduce ZSL-KG: framework with novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.
Common sense knowledge graphs organize high-level knowledge implicit to humans in a graph. The nodes in the graph are concepts associated with each other via edges. These associations in the graph offer a rich and large-scale source of high-level information, which makes them applicable to a wide range of tasks. To learn class representations, we look to existing zero-shot learning frameworks with graph neural networks. Existing zero-shot learning frameworks such GCNZ (Wang al. 2018) and DGP (Kampffmeyer et al.) 2019 that learn class representations from structured knowledge are applicable only to small graphs like ImageNet WordNet as they make restrictive assumptions. First, GCNZ requires the full graph Laplacian be known during training and performs several expensive computations that do nol scale well. GCNZ computes the graph Fourier transform that requires multiplication the node features with the eigenvector matrix of the graph Laplacian and computing the cigenvector matrix itself computationally expensive (Kipf & Welling 2017). Since publicly available common sense knowledge graphs range roughly from 100,000 8 million nodes and 2 million to 21 million edges (Speer et al. 2017 Zhang et al. 2020), computing the graph Fourier

---

Published in Transactions on Machine Learning Research (08/2022)
transform for such graphs is impractical. Further, in zero-shot learning tasks, the graph may change at test time new classes are added. Second, DGP requires a directed acyelic graph or parent-child relationship in the graph. In common sense knowledge graph, we are not restricted to parent-child relationships.
To overcome these limitations, we propose to learn class representations with a novel graph neural network: transformer graph convolutional networks (TrGCN). Transformers (Vaswani et al., 2017 are non-linear modules typically used for machine translation and language modeling tasks. They achieve a non-linear combination of the input sets using multilayer perceptrons and self attention. We exploit this property to learn permutation invariant non-linear aggregator that captures the complex structure of common sense knowledge graph. Finally, unlike GCNZ, TrGCN is inductive and learns node representations by aggregating the local neighbourhood features. This means the learned model can be used to predict with new graph structures without retraining, which makes them well-suited for zero-shot learning.
Here we describe TrGCN (see Figure L. We pass the neighbourhood node features hu (1-1)) through a two-layer perceptroo with a ReLU activation between the layers. The previous features are added to its output features with a skip connection, followed by layer normalization (LN) (Ba et al. 2016):
h 1 0ー =LN (LN(W1) (W/ [o (W! of hu - + -
where Wi E 1ydun and w xdu-1> learnable weight matrices for the feedforward neural network and ol) is ReLU activation. The non-linear neighbourhood features are then passed through the self attention layer to compute the weighted combination of the features for each query node:
fh
QKT Vd(e
1 Vu Vu N(w) = softmax z
V
(5)
where Q (W hi he is the set of all neighbourhood query vectors, K = w( -n -h--1) is the set of all key vectors, V = w:!). h is the set of values vectors, and Wg E Rde-n)dpp We E xd W, E R4a 1)x are learnable weight matrices with the projection dimension d(p)- The output features from the attention layer is projected with another linear layer and added to its previous features with a skip connection, followed by layer normalization:
-IN Vu Vu ENN) = LN (W.)..
((1-1) h nn"")
(6)
where !! E (Rdos is a learnable weight matrix.
To get the aggregated vector ! for node v, we pass the output vectors (=( Vu E N(v)) from the transformer encoder through permutation invariant pooling function u(.) such mean-pooling. The aggregated vector is passed through a lincar layer followed by non-lincarity (.) such ReLU or LeakyReLU:
u (1= VWEN(W)))
1 = (ww..a) a!!)
where w(0 E 11d1 is learnable weight matrix.
Differences between TrGCN and GAT. Joshi (0200) has drawn parallels between transformers and graph attention networks (GAT), suggesting that they are equivalent. Their work shows GAT is equivalent to transformers if the graph is treated as fully connected. While there are similarities between GAT and TrGCN, there are also key differences. GAT applies self attention compute scalar weights for [C(') and takes the linear combination of the scalar weights and [Wr') neighbours with complexity of ((IN(e))) to get the node representation. In contrast, TrGCN applies self attention to all node pairs in the neighbourhood to compute new non-lincar representations and then aggregates with pooling function to get a node representation with a computational complexity of (+)) This design makes TrGCN more computalionally expensive than other inductive graph convolutional networks such as GAT, but TIGCN can often improve accuracy across tasks (sce Table D. We also include the differences the resource requirements for TrGCN and GAT in Appendix C Finally, in Section 6. we show that TrGCN performs better than GAT with more heads, suggesting that our self attention and non-linear aggregation contributes to the difference in performance.

---

Published in Transactions on Machine Learning Research (08/2022)
Neighbourhood Sampling. In our experiments, we use ConceptNet (Speer et al., 2017 as our common sense knowledge graph, but ZST-KG is agnostic to the choice of the knowledge graph. ConceptNet has high node degree, which poses a challenge to train the graph neural network. To solve this problem, we explored numcrous neighbourhood sampling strategies. Existing work on sampling neighbourhood inchudes random sampling (Hamilton et al. 701 importance sampling (Chen et al. (018a), random walks (Ying el al. 2018), etc. Similar to PinSage (Ying al., 2018), we simulate random walks for the nodes the graph and assign hitting probabilities the neighbourhood nodes. During training and testing the graph neural network, we select the top N nodes from the neighbourhood based on their hitting probability.
Stacked Calibration. In generalized zero-shot learning setting, we predict both seen and unseen classes. However, generalized ZSL models tend to overpredict the seen classes (Chao et al. 2016). Following prior work [u et al. 2020 we solve this issue by simply lowering the compatibility scores for the seen classes:
((((()), W, 6(u)) = f (0(x), W, Myi) 7
where the calibration coefficient y R is a hyperparameter tuned on the held-out seen classes.
5 Tasks and Results
We evaluate our framework on three zero-shot learning tasks: fine- tne-ggrined entity typing, intent classification, and object classification.
In all our experiments, we compare ZSL-KG with other graph-based methods: GCNZ, SGCN, and DGP. GCNZ (Wang et al., 2018) uses symmetrically normalized graph Laplacian to generate the class representations. SGCN, introduced as a baseline in Kampffneeer el al. (2019), uses an asymmetrical normalized graph Laplaciann to learn the class representations. Finally, DGP (Kampfimeyer et al. 2019) uses a dense graph connectivity scheme with a two-stage propagation from ancestors and descendants to learn the class ropresentations.
In each lask, we also evaluate ZSL-KG against the specialized stale-of-the-al methods. These methods are considered specialized because they either cannot easily be adapted to multiple domains (e.g., attribute-based methods cannot easily be adapted to language dataset) require greater human effort to make them applicable to other domains (e.g., mapping ImageNet to Wikipedia articles). The code for our experiments has been released
Setup. We use two-layer graph neural networks for the graph-based methods and ZSL-KG. We mapped the classes to the nodes in the WordNet graph for each dataset and use the code obtained from Kampffneyer et al. (209)) to adapt the methods to language datasets. We provide details related to the mapping of classes to NonceptNet, post-processing ConceptNet, sampling, and random walk details in Appendix and the pseudocode for ZSL-KG in Appendix D Additional task-specific details are below.
5.1 Fine-Grained Entity Typing
Fine-grained entity typing is the task of categorizing the entities of sentence into more narrowly scoped semantic types. The ability to identify novel fine-grained types without additional human effort would benefit several downstream tasks such as relation extraction (Yavuz et al. 2016 and coreference resolution DDrrettt & Klein 2014).
Datasets. Fine-grained entity typing a zero-shot multi-label classification task because each entity can be associated with more than one type. We evaluate on popular fino-grained entity typing datasots: OntoNotes Gillick et al. 2014) and BBN (Weischedel & Brunstein 2005). We split. the dataset into two: coarse-grained labels (e.g., /location) and fine-gra aue-grained labels (e.g., /location/city)) See Appendix G for more details on the datasots. Following the prior work (Obcidat et al. 2019), we train on the coarso-grained labels and predict on both coarse-grained and fine-grained labels in the test set. J.1
Experiment. We use a bilinear similarity function (Eq. 1 for ZSL-KG and the other graph-based methods
The example encoder is AttentiveNER bILSTM (Shimaoka ct al. 2017) (soc Appendix H and the class
7

---

Published in Transactions on Machine Learning Research (08/2022)
OntoNoles
BBN
Overall
Striet
Loose Mic.
Loose Mac.
Striet
Loose Mic.
Loose Mac.
Avg. Striet
GCNZ
41.46 + 0.81
64.61 + U.52
61.65 = 0.52
21.47 + 1.22
47.17 + 1.04
56.81 = 0.68
31.47
SGCN
42.64 + 0.69
56.25 t 0.29
62.93 t 0.39
24.91 t 0.24
50.02 t 0.73
59.68
0.54
33.77
DGP
41.11
0.78
54.08
0.62
61.38 = 0.76
23.99 + 0.14
47.19 + 0.51
57.62
0.89
32.55
ZSL-KG
45.21 1 0.36
55.81 1 0.41
63.64 1 0.51
26.69 1 2.41
50.30 1 1.21
57.66 1 1.15
35.95
Table 1: The results for generalized zero-shot fine- fine-grained entity typing Ontonotes and BBN. We report the average strict accuracy, loose micro F1, and loose macro Fl of the models on 5 random seeds and the standard error.
encoder is the graph neural network. We also reconstruct the specialized state-of-the-art methods for task: OTyper (Yuan & Downey, 2018 and DZET (Obeidat et al.) 2019 OTyper uses average word embeddings and DZET uses Wikipedia Descriptors as their class representations. The methods are trained by minimizing the cross-entropy loss. For more details on the experiment, training, and inference see Appendix
As is common for this task (Obeidat et al., 2019 evaluate the performance of our model on strict accuracy, loose micro Fl, and loose macro Fl (Appendix KJ. Strict accuracy penalizes the model for incorrect label predictions and the number of the label predictions have to match the ground truth, whereas loose micro F1 and loose macro F1 measures if the correct label is predicted among other false positive predictions.
OntoNotes
BBN
Striet
Strict
OTyper DZET
41.72 + 0.44
25.76 + 0.25
42.88 + 0.47
26.20 0.13
ZSL-KG
45.21 + 0.36
26.69 + 2.41
Table 2: The results for generalized zero- shot learning on OntoNotes and BBN with specialized state-of-the-art methods.
Results. Table D shows that, on average, ZSL-KG outperforms the best performing graph-based method (SGCN) by 2.18 strict accuracy points. SGCN has higher loose miero Fl on OntoNotes and loose macro F1 on BBN datasets because it overpredicts labels and has greater false positives compared to the ZSL-KG. Our method has higher precision for label predictions and therefore, higher strict accuracy compared to other methods. Table 2 shows that ZSL-KG outperforms the specialized basclines and achieves the new state-of-the-art on the task.
5.2 Intent Classification
We next experiment on zero-shot intent classification. Intent classification is a text classification task of identifying users' intent expressed in chatbots and personal voice assistants.
Dataset, We evaluate on the main open-source benchmark for intent classification: SNIPS-NL.U (Coucke et al. 2018). The dataset was collected using crowdsourcing to benchmark the performance of voice assistants. The training set has 5 seen classes which we split into 3 train classes and 2 development classes.
Experiment. Zero-shot intent classification is a multi-class classification task. The example encoder used in our experiments is a biLSTM with attention as seen in the previous section (Appendix D We train the model for 10 epochs by minimizing the cross entropy loss and pick the model with the least. loss on the development. set. We measure accuracy on the test classes.
SNIPS-NLU
Accuracy
Zero-shot DNN
71.16
IntentCapaNet ReCapsNet-ZZS
77.52
79.96
GCNZ
82.47 + 03.009
We compare ZSL-KG against existing spocialized state-of-the-art methods in the literature for zero-shot intent classification: Zero-shol DNN (Kumar IntentCapsNett (Xia et al. 2018), and ResCaps.Net-22 (Liu 2019a). Intent IntentCapsNet and ResCapsNet--Ss are CapsulcNet-baseee ct al.
SGCN
50.27 : 14.13
DGP
64.41 + 12.87
let al.
2017).
ZSL-KG
88.98 + 01.22
Table 3: The results for intent classification on the SNIPS-NLU datasct.
(Sabour et al. 2017) approaches and have reported the best performance
the task.
8

---

Published in Transactions on Machine Learning Research (08/2022)
Hit
2-Hops
3-Hops
2
5
10
20
1
2
5
10
20
2
5
10
20
GCNZ
19.8
33.3
53.2
65.4
74.6
4.1
7.5
14.2
20.2
27.7
1.8
3.3
6.3
12.7
SGCN
40.4
60.2
71.9
81.0
6.0
10.4
18.9
36.9
2.8
4.9
19.3
ZSL
DGP
26.6
40.7
60.3
72.3
81.3
6.3
10.7
19.3
27.7
37.7
3.0
5.0
9.3
13.9
19.8
ZSL-KG
26.3
40.6
60.3
71.9
81.2
6.3
11.1
20.1
28.8
38.8
3.0
5.3
9.9
14.8
21.0
GCNZ
20.4
42.6
57.0
68.2
2.2
11.9
25.6
1.0
2.3
5.3
11.7
SGCN
11.9
27.0
50.8
65.1
75.9
3.2
7.1
16.1
24.6
34.6
1.5
3.4
7.8
12.3
18.2
GZSL,
DGP
10.3
26.4
50.3
65.2
76.0
2.9
7.1
16.1
24.9
35.1
1.4
3.4
7.9
12.6
18.7
ZSL-KG
11.1
26.2
50.0
64.3
75.3
3.4
7.5
16.9
26.1
36.5
1.7
3.8
8.5
13.5
19.9
Table 5: The results for object classification on ImageNet dataset. We report the class-balanced top-k accuracy zero-shot learning (ZSL) and generalized zero-shot learning (GZSL) for ImageNet classes k-hops away from the ILSVRC 2012 classes. The results for GCNZ, SGCN, and DGP are obtained from Kampflneyer
ct al. 2019
AW12
T1
T1
Avg. H
GCNZ
77.00 1: 2.05
86.62 1: 1.28
81.60
0.18
73:30 4: 0.75
51.68 A: 0.74
49.11 d: 0.39
71.26
0.20
58.14 :: 0.29
65.74
SGON
77.10 1.49
67.51
81.18
0.11
73.67 + 0.74
52.32
47.29
0.63
71.05 0.18
56.78
65.23
DGP
77.10
+ 1,33
71.26 +: 1.02
79.39
0.09
75.09 + 0.61
49.73 +: 0.30
46.16 =: 0.06
70.36 +: 0.25
56 08 = 0.52
65.38
ZSL-KG
78.08 0.84
06.80
0.70
84.42 0.33
74.08 +
60.54 0.58
55.16
0.50
0.52
61.57 + 0.45
68.07
Table The results for generalized zero-shot object classification on the AWA2 and aPY dataset. We report the average Hass-balances accuracy of the models for ZSL (TI) and GZSL (U, S, H) on 5 random seeds and the standard error.
Results. Table shows the results. ZSI.KKG significantly outperforms the existing approaches and improves the state-of-the-art accuracy to 88.98%. The graph-based methods have mixed performance on intent classification and suggest. Ihat. ZSL-KKG works well on a broader range of tasks.
5.3 Object Classification
Object classification is the computer vision task of categorizing objects.
Datascts. Zero-shot object classification is a multiclass classification task. We evaluate our method on the large-scale ImageNet (Deng et al. 2009,, Attributes (AWA2) (Xian et al. 2018b), and attribute Pascal Yahoo (aPY) (Farhadi et al. 2009) datasets. See Appendix G for more details on the datasets.
Experiment. Following prior work [Kapffmeyer al. 2019 Wang et al. 2018), we learn class representations by minimizing the L2 distance between the learn class representations and the weights of the fully connected layer of ResNet classifier pretrained on the ILSVRC 2012. Next, we freeze the class representations and finetune the ResNet- backbone on the training images from the dataset. More details on the experiments are included in Appendix J.2
AWA2
aPY
T1
H
ZSML 2020
77.5
58.9
74.6
65.8
64.0
36.3
46.6
40.9
APNet 20208
$3.9
66.4
32.7
74.7
45.5
58.6
DPPN
312T
73.1
61.2
48.4
TransZero-
MSDN 2022b
70.1
62.0
74.5
67.7
ZSL-KG
60.5
69.7
61.6
Table 4: The results for generalized zero-shot object classification on the AWA2 and aPY dataset with best performing specialized methods.
Following prior work (Kampfimcyer al.) 2019), we evaluate ImageNet on two set- tings: zero-shot learning (7ST.) where the model predicts only unseen classes, and generalized zero-shot.

---

Published in Transactions on Machine Learning Research (08/2022)
learning (GZSL) where the model predicts both seen and unseen classes. We follow the train/test split from Frome et al. (2013), and evaluate 7ST on three levels of difficulty: 2-hops (1549 classes), 3-hops (7860
classes), and All (20842 classes). The hops refer to the distance of the classes from the ILSVRC train classes. We report the Mlas-balanced top-K @Hit@k) accuracy for each of the hops.
We evaluate AWA2 and aPY in the ZST. and G7ST. settings as well. Following prior work (Xian et al. 2018b); for ZSL, we use the pretrained ResNet101 as the backbone and report the class-balanced accuracy on the unseen classes (TI). Following prior work (Min et al., 2020 for GZSL, we use the finetuned ResNet101 to report the class- balance accuracy the unseen classes (U), seen classes (S), and their harmonic mean (H).
Results. Table 6 shows that ZSL-KG outperforms existing graph-based methods on aPY dataset by 3.53 points on the harmonic mean and shows an average improvement across both the datasets by an average of 2.33 points on the harmonic mean metric. We also observe that the graph- based methods show a significant drop in accuracy from AWA2 to aPY, whercas our method consistently achieves high accuracy on both datasets.
Table ssows ZSL-KG compared with specialized attribute- methods. Our results show that we signifi- cantly outperform existing attribute-basod methods on the aPY datasots and show competitive performance on the AWA2 dataset. We suspect that pretraining ZSL-KG class representations on ILSVRC 2012 classes helps the performance. Existing attribute-based methods cannot be pretrained on ILSVRC because ILSVRC does not have hand-crafted attributes. This highlights the potential benefits of using class representations from graphs. Furthermore, we note that AGZSL (Chou et al. 2021) and ZSML (Verma et al. 2020 allow the model to access the attributes and names of the unseen classes during training, which makes them transductive. Nonetheless, we either outporform them or achiove comparable performance. Finally, we include extended results with other specialized methods in Appendix A
Table 5 shows results for ImageNet dataset. ZSL-KG outperforms existing graph- based methods on 3-hops (7860 classos) and All (20842 classes) for ZSL and GZSL evaluation. ZSL.KG despite being trained on a noisier graph, achieves competitive performance on the ImageNet dataset on both ZSL and GZSL settings.
5.4 Discussion
Overall, our results show that ZSL-KG improves over existing WordNet-based methods on OntoNotes, BBN, SNIPS-NLU, aPY, and ImageNet (all test classes). DGP does slightly better on AWA2, but it performs relatively poorly aPY and ImageNet. In contrast, we see that ZSL-KG achieves the highest performance larger test sets of ImageNet and achieves new state-of-the--rt for aPY, while maintaining competitive performance on AWA2, Finally, averaging the strict accuracy on OntoNotes and BBN, harmonic mean on AWA2 and aPY, top-1 GZST, accuracy for all classes on ImageNet, and accuracy on SNIPS-NLUU we observe that ZSL-KG outperforms the best performing points and all the WordNet-based methods by an average of 2.4 accuracy points. These results demonstrate the superior flexibility and generality of ZSL-KG to achieve competitive performance on both language and vision tasks.
BoodNet-based method (GCNZ) by an average of 3.5 accuracy
6 Comparison of Graph Aggregators
We conduct an ablation study with different aggregators with our framework. Existing graph neural networks include GCN (Kipf& & Welling 2017), GAT (Velččkovi et al. 2018), RGCN (Schlichtkrul) al. 2018), and LSTM (Hamilton et al. 2017a). We provide all the architectural details in Appendix L We train these models with the same experimental setting for the tasks mentioned in their respective sections.
ZSL-KG with different aggregators. Table shows results for our ablation study. Our results show that T&GCN often outperforms existing graph neural networks with linear aggregators and TrGCN adds up to 1.23 accuracy points improvement on these tasks. We observe that GAT and GCN outperform TrGCN on BBN and ImageNet datasets. However, no architecture superior on all tasks, but TeGCN the best on a majority of them. With relational aggregators (ZST-KG-RGCN), we observe that they do not outperform
10

---

Published in Transactions on Machine Learning Research (08/2022)
OntoNotes
BBN
SNIPS-NLU
AWA2
aPY
ImageNet
Strict
Strict
Acc.
H
H
All (%)
ZSL-KG
45.21
26.69
88.98
74.58
61.57
1.74
-GCN
42.19
27.44
84.78
73.08
60.45
1.78
-GAT
43.48
32.65
87.57
73.35
60.91
1.77
RGCN
43.88
27
87.47
47.96
31.60
-LSTM
44.52
26.77
88.81
55.55
57.02
0.83
Table The results for zero-shot learning with alternate graph ncural networks as class encoders in ZSL-KG.
AWA2
aPY
H
H
ZSI-KG-GAT (1-head)
73.35
60.91
ZSL-KG-GAT (2- 2-hhads))
72.68
60.72
ZSL-KG-GAT (3-heads)
71.71
60.65
ZSL-KG (TrGCN)
74.58
61.57
Table 9: The results for generalized zero-shot learning on AWA2 and aPY datasets with multiple heads in graph attention networks.
ZSL-KG and may reduce the overall performance (as seen in AWA2 and aPY). ZSL-KG-LSTM which uses an LSTM-based aggregalor shows inconsistent performance across different lasks.
We also compare the ZSL-KG with TrGCN and other graph aggregators on conventional zero-shot learning, where only unseen classes are present during testing. Table shows that ZSL-KG with TrGCN outperforms other graph neural networks on two of the three object classificttinn datasets.
Comparison of WordNet and ConceptNet. It is also worth understanding the effect of the knowledge graphs used for vero-shot learning. We compare SGCN and GCL-KG as they use the same lincar aggregator to learn the class representation but train with different knowledge graphs, Le.. SGCN uses WordNet whereas ZSI.-G-GCN uses ConceptNet. We see that ZST-KG-GCN trained on common sense knowledge graphs adds an improvement as high 6.7 accuracy points across the tasks suggesting that the choice of knowledge graphs is crucial for downstream performance.
AWA2
aPY
ImageNet
T1
All (%)
ZSL-KG
78.08
60.54
3.01
-GCN
74.81
57.76
2.97
-GAT
75.29
59.06
3.08
RGCN
66.27
29.51
LSTM
66.46
50.84
2.65
Table 8: The results for zero-shot learn- ing tasks with other graph neural net- works.
We further investigate the differences between WordNet and Con- ceptNet by training TrGCN with WordNet (see Appendix B We show that. TrGCN can benefit existing applications with WordNet, but might tend to work better with richer graph structure such NonceptNet.
Comparison of TrGON and GAT with multiple heads. To better understand the differences between GAT and TrGCN, we perform an ablation with multihead attention in GAT to increase the expressivity in the ZSL-KG framework. The multihead atlention in GAT is similar to Vaswani el. al. (2017) but instead average the output from the heads to keep the same output dimension for the node representations. Table g shows that adding more heads ZG--KG-GAT hurts performance, while ZSL-KG with TrGCN achioves the highest performance. This suggests that the self attention and non-linear aggregator in TrGON contributes to the difference in performance.
11

---

Published in Transactions on Machine Learning Research (08/2022)
7 Conclusion
ZSL-KG is flexible framework for zero-shot learning with common sense knowledge graphs and can be adapted to wide variely of lasks without requiring additional anmotation effort. Our framework introduces novel transformer graph convolutional network to learn rich representations from common sense knowledge graphs. Our work demonstrates that common sense knowledge graphs are source of high-level knowledge that can benefit many tasks.
Acknowledgements
We thank Yang Zhang for help preparing the Ontonotes dataset. We thank Roma Patel, Elaheh Raisi, Charles Lovering, and anonymous reviewers for providing helpful feedback on our work. This material is based research sponsored by Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) under agreement number FA8750-19-2-1006 The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) or the U.S. Government. We gratefully acknowledge support from Google and Cisco. Disclosure: Stephen Bach is advisor to Snorkel AT, a company that provides software and services for weakly supervised machine learning.
References
Ashutosh Adhikari, Xingdi Yuan, Mare-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondcau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. Learning dynamic knowledge graphs to generalize on text-based games. In Aduances Neural Information Processing Systems (NeurTPS), 2020.
Yashas Annadani and Soma Biswas. Preserving semantic relations for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognilion (CVPR), 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arxiv, 2016.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. Graph convolutional encoders for synlax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirioal Methods in Natural Language Processing (EMNLP), 2017.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning, In International Conference on Learning Representations (ICLR), 2019.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NewrIPS), 2020.
Ming- Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. Importance semantic representation: Dataless classification. In AAAT Conference on Artificial Intelligence (AAAT), 2008
Wei-Lum Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalizod zero-shot learning for object recognition in the wild. In European conference on computer vision (ECCV), 2016.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Regresentations (TCLR), 201 8a.
Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recognition using semantics-peeerrving adversarial embedding networks. In Proceedings of the TEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2018b.
Shiming Chen, Zi-Quan Hong, Guosen Xie, Jian Zhao, Hao Li, Xinge You, Shuicheng Yan, and Ling Shao. +ranszero++: Cross attribute-guided transformer for zero-shot learning. ArXiv, abs/2112.0880 2021.
Shiming Chen, Ziming Hong, Yang Liu, Guo-Sen Xie, Baigui Sun, Hao Li, Qinmu Peng, Ke Lu, and Xinge You. Transzero: Adtribute-guided transformer for zero-shot learning. In Proceedings of the Thirby Sirth AAAI Conference on Artificial Intelligence (AAAI), 2022a.
Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmmu Peng, Kai Wang, Jian Zhao, and Xinge You. Transzero: catribute-guidess transformer for zero-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition ( CCVRR) ), 2022b.
YunYing Chou, Hsuan-Tien Lin, and Tyng-Luh Liu. Adaptive and generative zero-shot learning. In International Conference on Learning Representations (ICLR), 2021.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gissclbrecht, Francesco Caltagirone, Thibaut Lavril, et al. Snips voice platform: an embedded spoken language understanding system for privale-by-design voice interfaces. In ICML workshop on Privacy in Machine Learning and Artificial Intelligence, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Ieee, 2009.
Greg Durrett and Dan Klein. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Associalion for Compulational Linguistics (TACL), 2:477-490 2014.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. In AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2021.
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua. Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations, 2022.
Ali Farhadi, Ian Endres, Derek Hoiem, and David A. Forsyth. Describing objects by their attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems (NewrIPS), 2013.
Chuang Gan, Ming Lin, Yi Yang, Yucting Zhuang, and Alexander G Hauptmann. Exploring semantic inter-class relationships (sir) for zero-shot action recognition. In AAAI Conference on Artificial Intelligence (AAAT), 2015.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. Allennlp: A deep semantic natural language processing platform. In Workshop for NLP Open Source Software (NLP-OSS), 2018.
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. Context-dependent fine-grained entily type lagging. arXie, 2014.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural module networks for reasoning over text. In International Conference on Learning Regresentations (ICLR), 2020.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Newral Information Processing Systems (NeurIPS), 2017a.
William L Hamilton, Rex Ying, and Jure Leskovee. Representation learning on graphs: Methods and applications. TEEE Dala Engineering Bulletin, 40(3):52 74, 2017b.
Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive embedding for generalized zero-shol learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. Tn Proceedings of The Web Conference 2020, pp. 2704-2710, 2020.
Filip llievski, Pedro Szekely, and Bin Zhang. Cskg: The commonsense knowledge graph. In Eztended Semantic
Web Conference (ESWC), 2020.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics (TACL.) 3:339-351, 2017.
Chailanya Joshi. Transformers are graph neural networks. The Gradient, 2020.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric Xing. Rothinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Rohit Keshari, R. Singh, and Mayank Vatsa. Generalized zero-shot learning via over-complete distribution. Tn Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. (CVPR), pp- 13297-13305, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létournean, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34, 2021.
Anjishnu Kumar, Pavankumar Reddy Muddireddy, Markus Dreyer, and Björn Hoffmeister. Zero-shot learning across heterogencous overlapping domains. In Proceedings of Interspeech, 2017.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pallern Analysis and Machine Intelligence (PAMI), (3):453-445) 465, 2013.
Jingjing Li, Mengmeng Jing, Ke Lu, Zhengming Ding, Lei Zhu, and Zi Huang. Leveraging the invariant side of generative vero-shot learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2019.
Yun Li, Zhe Liu, L. Yao, Xianzhi Wang, Julian McAuley, and Xiaojun Chang. An entropy-guided reinforced partial convolutional network for zero-shot learning. ArXiv, 2022.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
2019.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Xiao Ling and Daniel S Weld. Fine-grained entity recognition. In AAAI Conference Artificial Intelligence (AAAT), 2012.
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert YS Lam. Reconstructing capsule networks for zcro-shot intent classification. In Proceedings of the 2019 Conference on Empirical Melhods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) 2019a.
Hugo Liu and Push Singh. Conceptnct- a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211 226, 2004.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Attribute propagation network for graph zero-shot learning. In AAAT Conference on Artificial Intelligence (AAAT), 2020a.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Xuanyi Dong, and Chengqi Zhang. Isomctric propagation network for generalized zero-shot learning. In International Conference on Learning Represenlations (ICLR), 2021a.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wal Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang- Hyperbolic visual embedding learning for zero-shot recognition. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pollern Recognilion, pp. 9273 9281, 2020b.
Yang Liu, Jishun Guo, Deng Cai, and Xiaofei He. Attribute attention for semantic disambiguation in zero-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019b.
Zhe Liu, Yun Li, Lina Yao, Julian McAuley, and Sam Dixon. Rethink, revisit, revise: A spiral reinforced self-revised network for zero-shot learning. arXiv preprint 212.0000,, 2021b.
Sébastien Marcel and Yann Rodriguez. Torchvision: The machine-vision package of torch. In International
Conference on Multimedia, 2010.
Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
Grégoire Mialon, Dexiong Chen, Margot Selosse, and Julien Mairal. Graphit: Encoding graph structure in transformers. arXiv preprint arXim:2106.n0 0566677, 2021.
Tomas Mikolov, Llya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zueng-Jun Zha, and Yongdong Zhang. Domain-aware visual bias eliminating for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Ryan L Murphy, Balasubramanian Srinivasan, Vinayak Rao, and Bruno Ribciro. Janossy pooling: Learning deep coamutaationinvariinn functions for variable-size inputs. In International Conference on Learning Representations (ICLR), 2019.
Sanath Narayan, A. Gupta, F. Khan, Cocs G. M. Snock, and Ling Shao. Latent embodding foodback and discriminative features for zero-shot classification. In European Conference on Computer Vision (ECCV), 2020.
Rasha Obeidat, Xiaoli Fern, Hamed Shahbazi, and Prasad Tadepalli. Description-based zero-shot fino-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACI-HI.T), 2019.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. Modeling fine-grained entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Associalion for Compulational Linguistics (ACL), 2021.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, Glove: Global vectors for word representa- tion. In Proceedings of the 2014 conference on empirioal methods in natural language processing (EMNLP), 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nlya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schicle. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognation (CVPR), June 2016.
Xiang Ren, Wenqi He, Mong Qu, Lifu Huang, Heng Ji, and Jiawei Han. Afct: Automatic fine-grained entity typing by hierarchical partial-label embedding. In Proceedings of the 2016 Conference on Empirioal Melhods in Natural Language Processing (EMNLP), 2016.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. Tn International Conference on Machine Learning (ICML), 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211 21-222221 2015.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Michael Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference (ESWC), 2018.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. End-to-end structure-aware convolutional networks for knowledge base completion. In AAAI Conference on Artificial Intelligence (AAAT), 2019.
Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. Neural architectures for fine-grained entity type classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2017.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLP), 2020.
Ivan Skorokhodov and Mohamed Elhoseiny, Class normalization for zero-shot learning. In International Conference on Learning Representations (ICLR), 2021.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through codal transfer. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI Conference on Artificial Intelligence (AAAT), 2017.
Flood Sung, Yongxin Yang, Zhang, T. Xiang, P. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), 2018.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition- based multi-relational graph convolutional networks. In International Conference, on Learning Representations (ICLR), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018.
Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. A meta-learning framework for generalized zero-shot lcarning. In AAAI Conference on Artificial Intelligence (AAAT), 2020.
M. R. Vyas, Hemanth Venkateswara, and S. Panchanathan. Leveraging seen and unseen semantic relationships for generative zero-shot learning. In European Conference on Computer Vision (ECCV), 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Chaoqun Wang, Shaobo Min, Xuejin Chen, Xiaoyan Sun, and Houqiang Li. Dual progressive prototype network for generalized zero-shot learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NewrPPS), 2021.
Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-37, 2019.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Ralph Weischedel and Ada Brunstein. Bbn pronoun coreference and entity type corpus, 2005.
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. In International Conference on Machine Learning (ICML), 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEFE Transactions on Neural Networks and Learning Systems, 32(1): 4 24, 2021.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. Zero-shot user intent detection via capsule neural networks. In Proceedings of the 2018 Conference Empirical Melhods in Natural Longuage Processing (EMNLP), 2018.
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zcro-shot classification. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognátion (CVPR), 2016.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning a comprehensive evaluation of the good, the bad and the ugly. IEFF Transactions Pattern Analysis and Machine Intelligence (PAMI), 212:)22251 2265, 2018a.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEFE Conference on Compuler Vision and Pattern Recognilion (CVPR), 2018b.
Guo-Sen Xic, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, and Ling Shao. Attentive region embedding network for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2019.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Imposing label-relational inductive bias for extremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR), 2019.
Wenjia Xu, Yongqin Xian, Jiuniu Wang, B. Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HLLT), 2021.
Semih Yavuz, Izzeddin Gur, Yu Su, Mudhakar Srivatsa, and Xifeng Yan. Improving semantic parsing via answer type inference. In Proceedings of the 2016 Conference on Empirical Methods in Naturol Language Processing (EMNLP), 2016.
Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNIP-IICNIP)) 2019.
Chengxuan Ying, Tianle Cai, Shengjic Luo, Shuxin Zheng, Guolin Ke, Di Hc, Yanming Shen, and Tic-Yan Liu. Do transformers really perform badly for graph representation? Advances in Newral Information Processing Systems, 34, 2021.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2018.
Dani Yogatama, Danicl Gillick, and Nevena Lazic. Embodding methods for fine grained entity type classifica- tion. In Proceedings of the 53rd Arnual Meeling of the Associalion for Compulational Linguistics and the 7th International Joint Conference Natural Longuage Processing (ACL-IJCNLP), 2015.
Y. Yu, Z. Ji, J. Han, and Z. Zhang. Episode-base based prototype generating network for zero-shot learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Zheng Yuan and Doug Downey. Otyper: A neural architecture for open named entity typing. In AAAI Conference on Artificial Intelligence (AAAT), 2018.
Scongjun Yun, Minbyul Jeong, Rachyun Kim, Jacwoo Kang, and Hyunwoo Kim. Graph transformer networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Chenrui Zhang, Xiaoqing Lyu, and Zhi Tang. Tgg: Transferable graph generation for zero-shot and few-shot learning. In Proceedings of the 27th ACM International Conference on Multimedia, 2019a.
Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth. Transomes: From linguistic graphs to commonsense knowledge. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), 2020.
Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. Integrating semantic knowledge to tackle zero-shot text classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies (NAACI--HITT), 2019b.
18

---

Published in Transactions on Machine Learning Research (08/2022)
Bo Zhao, Yanwei Fu, Rui Liang, Jiahong Wu, Yonggang Wang, and Yizhou Wang. A large-scale attribute dataset for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pollern Recognition Workshops, 2019.
Ben Zhou, Daniel Khashabi, Chen-Tse Tsai, and Dan Roth. Zero-shot open entity typing as type-compatible grounding. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
19

---

Published in Transactions on Machine Learning Research (08/2022)
A 2WA2
aPy
T1
U
S
H
T1
U
S
H
SP-AEN (Chen 2018b
58.5
23.3
90.9
37.1
24.1
13.7
63.4
22.6
PSR (Annddani & Biswas
2018
63.8
20.7
73.8
32.3
38.4
13.5
51.4
21.4
Relation Net (Sung et al.
2018
64.2
30.0
93.4
45.3
(Liu et a..
2019b
68.1
27.0
93.4
41.9
AREN (Xic ct al. 2019
66.9
54.7
79.1
64.7
39.2
30.0
47.9
36.9
EVAEGAN-D2 (Xie el
2019
70.3
57.1
76.1
65.2
ZSML /eerma et 2020
77.5
58.9
74.6
65.8
64.0
36.3
46.6
40.9
GXE [Li ct al. 2019
71.1
56.4
81.4
66.7
38.0
26.5
74.0
39.0
OCD-CVAE (Keshari et al.
2020
71.3
59.5
73.4
65.7
E-PGN Yu et al. 2020
73.4
52.6
83.5
64.6
LsrGAN Vyas et al. 2020
54.6
74.6
63.0
TF-VAEGAN (Narayan al.
2020
73.4
55.5
83.6
66.7
APN (Xu et al.]
2020
71.7
62.2
69.5
65.6
APNet (Liu et al.
2020a
54.8
83.9
66.4
32.7
74.7
45.5
DBVE Min et al.
2020
62.7
77.5
69.4
37.9
55.9
45.2
CN-ZSL Skorokhodov Elhoseiny
2021
60.2
77.1
67.6
CF-GZSI. Han et al. 2021
70.4
63.1
78.6
70.0
IPN (Lin et al. 2021
67.5
79.2
72.9
37.2
66
47.6
AGZSL Chou et al.
2021
76.4
69.0
86.5
76.8
43.7
36.2
58.6
44.8
DPPN Wang et al.
2021
63.1
86.8
73.1
40.0
61.2
48.4
RSR (Tiu et al.] 202 1b
68.4
55.3
76.0
64.0
45.4
31.3
50.9
38.7
Trans CCen et al.
2022a
70.1
61.3
82.3
70.2
TransZero- (Chen al.
2021
72.6
64.6
82.7
72.5
ERPCNet Li et al.
2022
71.8
59.1
82.0
68.7
43.5
32.7
49.3
39.3
MSDN (Chen et al.
2022b
70.1
62.0
74.5
67.7
GCNZ Wang
2018
77.0
66.6
81.6
73.3
51.7
49.1
71.3
58.1
SGCN Kampflmeyer et al.
2019
77.1
67.5
81.2
73.7
52.3
47.3
71.1
56.8
DGP
Kampllmeyer et al.
2019
77.1
71.3
79.4
75.1
49.7
46.2
70.2
55.7
ZSL-KG (ours)
78.1
66.8
84.4
74.6
60.5
55.2
69.7
61.6
Table 10: ZSL-KG compared to attribute-bassd zero-shot learning methods.
A Results on AWA2 and aPY
Table
10 shows results comparing ZSL-KG with other related work from zero-shot object classification.
B TrGCN with WordNet
To further illustrate the importance of using richer knowledge graph such ConceptNet, experiment with TrGCN trained on the WordNet graph. Table TI shows that TrGCN with WordNet on OntoNotes oulperforms other WordNet-based methods but underperforms the best-performing WordNet-basee method on the rest of the datasets. However, we see that ZSL-KG, ie., TrGCN with ConceptNet always improves the performance compared TrGCN with WordNet. These inconsistent results suggest that TrGCN can benefit existing applications with WordNet, but might tend work better with richer graph structure such ConceptNet.
20

---

Published in Transactions on Machine Learning Research (08/2022)
OntoNotes
BBN
SNIPS-NLU
AWA2
aPY
Strict
Strict
Acc.
H
H
GCNZ
41.50
21.50
82.47
73.30
58.10
SGCN
42,60
24.90
50.30
73.70
56.80
DGP
41.11
23.99
64.41
75.10
55.70
TrGCN (WordNet)
44.12
23.09
41.85
72.16
55.39
7SL-KG (ConceptNet)
45.21
26.69
88.98
74.58
61.57
Table 11: The results showing zero-shot performance of existing WordNet-based methods, TrGON with WordNet, and ZSI,KG, ie., TrGCN with NonceptNet.
60
Method TrGON
Methed
0.0
seconds)
50
TrGCN
Peak GRI Memory
2.5
30
2.0
1.5
1.0
10
BBN
OntoNotes
BAN
Deiollates
Dataset
Dataset
Figure 2: Graphs showing time taken per epoch (in seconds) and peak GPU memory (in GB) for fine-grained entity lyping datasets.
C Resource Requirements for ZSL-KG with TrGCN and GAT
Here measure differences between TrGCN and GAT in terms of resource requirement. In particular, we perform experiments to compute the average training time per epoch and GPU memory requirements. We run experiments with the fino-grainod entity typing datasets, namely BBN and OntoNotes. We use the same hyperparameters as mentioned in Appendix and run our experiments on an NVIDIA RTX 3090 with 24GB of GPU memory.
Our results in Figures 2 show that, during training, TrGCN takes slightly longer time per epoch and greater GPU memory compared to GAT. We note that both GAT and TrGCN benefit from the importance sampling as they can be batched and processed efficiently. On the other hand, GCNZ results in out-of-memory (0OM) error as it requires the full graph during training and testing.
D LSTM Predictions
LSTMs have been used in graph neural networks as aggregators to generate more expressive node embeddings. However, LSTMs assume an ordering of the imputs which is not present in a graph neighbourhood. To apply LSTMs to an unordered set of nodes, Hamilton et al. (2017a) randomly permute the nodes in the neighbourhood.
We test whether randomly permuting the node neighbours makes LSTM-based aggregator permutation invariant. We replicate the I.STM-basod aggregator to work with the ZSL.KG framework. The model is trained with the setup described in Section 5.3 We run the prediction on the Animals with Attributes 2 dataset by computing 10 class representation for each of the classes using the trained LSTM--based aggregator model.
The experiments reveal that 1325 out of 7913 (16.78%) have multiple predictions for the same image in the unseen classes. For images that have multiple predictions, take the count of the mode prediction and plot.
21

---

Published in Transactions on Machine Learning Research (08/2022)
400
Number Predictions
100
1
2
3
7
Prediction Consistency
Figure 3: Graph showing distribution of inconsistencies in the LSTM-based aggreagtor predictions.
the histogram. Figure 3 shows inconsistency in predictions. The graph for given value on the x-axis is read as for every 10 prediction, times the same output is predicted.
E ConceptNet Setup
In all our experiments with ZSL-KG, we map each class to node in ConceptNet 5.7 (Spee et al. 2017 and query its 2-hop neighbourhood. Next, we remove all the non-English concepts and their edges from the graph and make all the edges bidirectional. Then, we optionally take the union the concepts' neighbourhood that share the same prefix noun prefix. For example, we take the union of the neighbourhood nodes of /c/ en/laver and /c/e///e//ee e//wwyerr/ Then, we compute the embeddings for the concept using the pretrained 300 dimensional GloVe 840B (Pennington et. al. 2014). We average the individual word in the concept get. the embedding. These embeddings serve initial features for the graph neural network.
For the random walk, the number of steps is 20, and the number of restarts is 10. We add one smoothing to the visit counts and normalize the counts for the neighboring nodes. During training, we sample K neighbours with the highest hitting probabilities at each hop. For OntoNotes, BBN, AWA2, aPY, and SNIPS-NLU, we sample 50 neighbours in the first hop and 100 neighbours in the second hop. For ImageNet we sample 100 and 200 neighbours in the first and second hop.
F Pseudocode
we describe the forward pass with the ZSL-KG framework. TrGCN computes the class representations for the nodes in the graph. The class representations used in the bilinear similarity function to compute the compatibility scores for the classes.
In Algorithm
I
G Dataset Details
Table P and Table 13 show the statistics for zero-shot datasets used in our experiments. Apart from these datasots, we also evaluate ZSL-KG on the ImageNet dataset. ImageNet dataset has total of 1000 seen classes and 20842 unseen classes
We obtain the OntoNotes and BBN dataset from Ren et al. 2016). OntoNotes has three levels of types such as /location, /Location/strucuur,, /1ocation/structtii are treated as coarse-graine grained entity types and ietcaation/sttuuuuue//ooote is treated fine-grained entity type. Similarly, BBN has two levels of types and consider the level two
where /location and
22

---

Published in Transactions on Machine Learning Research (08/2022)
Algorithm 1: Forward pass with the ZST-KG framework.
Input :example 2, example encoder ((x), linear layer W, class encoder o(y), graph g(V, E, R), class nodes {ol 22 -... bg); node initialization H [nt ...., h²), depth L, graph neural network weights {W1, ,... W transformers {T1 ..... ] neighbourhood sample sizes {81,1-.,81) 8r.)
Output: :ogits for classes y = {111 12, .... Jn
TrGCN 1= 0);
if I L then
return Hy +- H(V);
else
i 0:
for v E V do
N f N(v, 8141;;
H2 + TrGCN (N, 1 1);
Zy 4 1((+1) HN);
a (1+1) t mean Zx);
((+1) +- (WW(L++1) am);
H [++1] (((11) /1lb! (1+1)
2:
-+i+1
end
relurn H ((+1)
end
((y) - (FGCN( .........)) ")
return
Datasct
Seen classes
Unseen classes
Seen examples
Test examples
Ontosotes
40
32
220398
9604
BBN
15
24
85545
12349
SNIPS-NLLU
5
2
9888
3914
Table 12: Zero-shot fine-grained entity typing (OntoNotes and BBN) and intent classification (SNIPS-NLU) datasets used in our experiments.
types as fine-grained types. Furthermore, we process the datasets by removing all the fine-grained entity types from the train set. We also remove all the examples from the train set where coarse-grained entity types are not present in the test. set. We nole that the OntoNotes datasel has /other lype which cannot. mapped to a meaningful concept or a wikipedia article. Since /other is a cearse-grained entity type, we train weight vector for the type and treat class representation.
For object classification datasets, our work follows the proposed splits suggested in Xian et al. (018b). Since aPY has multiple objects in image, we crop objects with the bounding box information provided and use the cropped images for training and testing.
AttentiveNER
Here, we describe AttentiveNER (Shimaoka al. 2017) used the example encoder in the fine-grained entity typing task. Each mention m comprises of tokens mapped to pretrained word embedding from GloVe 840B. We average the embeddings to obtain single vector Um:
Um n
m
(9)
23

---

Published in Transactions on Machine Learning Research (08/2022)
Dataset
Seen classes
Unseen classes
Seen examples
Seen test examples
Unseen test examples
AWA2
40
10
23527
5882
7913
aPY
20
12
5932
1483
7924
Table 13: Zero-shot object classification datasets used in our experiments.
where m, E Rd is the pretrained embedding.
We learn the context of the mention using two biLSTM with attention layers. The left. context represented by (4,,4)...,4 and the right context r by {ri, r2,,,, rx] where I Rd and ry E Rd are the pretrained word embeddings for the left and the right context. We consider context window size of s. We pass and through their separate biLSTM layers to get the hidden states h2, h for the left context and h, h, for the right context. The hidden states are passed through the attention layer to compute the attention scores. The attention layer two-layer feedforward neural network and computes the normalized attention for each of the hidden states U. E R.a:
a W. (tant( We
(10)
exp (a!)
of Exp (a!) + Ej exp (aa)
(11)
The scalar values are multiplied with their respective hidden states to get the final context vector representation
Vo:
V.
h
(12)
1=0
Finally, we concatenate the context vector ve and Um to get the example representation æ.
The AttentiveNER can be replaced with a pretrained language model Onoe et al. 2021 as the example encoder and potentially improve the performance, which we leave for future work.
BiLSTM with Attention
The biLSTM with altention is used to learn class representations in DZET (Obeidat al. 2019). The input. tokens Wo. Wi,..., W,, represented by pretrained embeddings are passed to the biLSTM to get the hidden states h and h. The hidden states are concatenated to get = ho, hi.... hn. Next, they are passed to the attention module get the scalar attention values:
a; Wa (tanh(W: (W. h:))
(13)
The scalar attention values are then normalized with a softmax layer:
exp (as)
a, Σ, exp (a:)
(14)
The normalized scalar attention values multiplied with their respective hidden vectors to get the final representation t,p
ty
ag
(15)
1=0
24

---

Published in Transactions on Machine Learning Research (08/2022)
J. Experiment Details
J.1 Fine-grained Entity Typing
For all the experiments, we initialize the tokens in the example encoder with 300 dimensional GloVe 840B embeddings. We reconstruct OTyper YYuan & Downey 2018) and DZET (Obeidat et al. 2019) for fine-grained entity typing. Otyper averages the 300 dimensional Ve 840B embeddings for the tokens in the entity type or the class. For DZET, we manually mapped the classes to Wikipedia articles. We pass each article's first paragraph through a learnable biL.STM with attention to learn the class representations (Appendix !
For both OntoNotes and BBN, the methods are trained for epochs by minimizing the cross-entropy loss using Adam with a learning rate 0.001. During inference, we pass the scores from the bilinear similarity model through a sigmoid and pick the labels that have a probability of 0.5 or greater as our prediction. Since have coarso-grained types along with fine grained types, set calibration coefficient y to 0.
J.2 Object Classification
For AWA2 and aPY, we follow the same L2 training scheme and train for 1000 epochs on 950 random classes from 1000 ILSVRC 2012 classes, while the remaining 50 classes are used for validation. The model with the least loss on the validation classes is used to generate the seen and unseen class representations with the graph. Since only a subset of the seen classes for AWA2 (22 out of 40) and aPY (2 out of 20) are part of IL.VVRC 2012 classes, we freeze the class representations for the seen classes and fine-tune pretrained ResNNet101-bcckoone on the individual datasets for 25 epochs using SGD with a learning rate 0.0001 and momentum of 0.9. We calibrate "y the validation splits as suggested in Xian al. (2018b) and set y 3.0 for AWA2 and 7 = 2.0 for aPY for all the methods.
For the ImageNet experiment, we train ZSL-KG by minimizing the L2 distance between the learned class representations and the weights fully connected layer of a ResNet50 classifier for 3000 epochs on 1000 classes from the ITSVVC 2012. Similar to DGP and SGCN, we freeze the class representations for the class representations and finc-tune the Ressootbackkone on the ILSVRC 2012 dataset for epochs using SGD with a learning rate 0.0001 and momentun of 0.9. During inference, we switch the knowledge graph to the ImageNet graph and generate class representations from them. For fair comparison with other graph-based methods on ImageNet, we use ResNet50 model (He et al. 2016) in Torchvision (Marcel & Rodrigucz 2010 pretrained on ILSVRC 2012 (Russakovsky et. al. 2015) and do not calibrate the outputs from ZSI-KG.
K Fine-grained entity typing evaluation
We follow the standard evaluation metric introducod in Ling & Weld 2012 and Loose Macro F1.
Strict Accuracy, Loose Micro F1
We denote the set of ground truth types as T and the set of predicted types as P. We use the Fl computed from the precision p and recall for the evaluation metrics mentioned below:
Strict. Accuracy. The prediction is considered correct if and only if le = fe:
= ECEP-T 1lte = Ec)
(16)
PP]
Eee Prr
1lle Ec)
(17)
[T]
Loose Micro. The precision and recall scores are computed as:
p Eeep Ite n Eel Ecep Vill
(18)
25

---

Published in Transactions on Machine Learning Research (08/2022)
Method
Aggregate
Combine
7SL-KG-GCN
a.¹ Mean (fnl-") a E (((1)))
h (wi)l!")
ZSL-KG-GAT
(1) = Attn (100
(0) o Ninht 100 hu 111-1))
ZSL-KG-RGON
a Eter EsENCOY our Eser 1.
ho = o(a. ((!
ZSL-KG-LSTM
a LSTM (nl Vu e N(v))
h W [1t t1 (1 (1(a
Table 14: Graph Aggregators
p Eeer le n bel Eeer lfel
(19)
Loose Macro. The precision and recall scores computed
te n fel ICel
p
(20)
P
PEP
Me n fel
(21)
fel
L
Graph Neural Networks Architecture Details
Task
Inp. dim.
Hidden dim.
Attn. dim.
Low- Low-rank dim. h
Intent classification
300
32
20
16
Fine-grained entity typing
300
100
100
20
Table 15: Hyperparameters for the biLSTM with attention example encoder in the language related tasks
Table describes the aggregator and combine function for the ablation experiments. ZSL-KG-GCN uses a aggregator to learn the neighbourhood structure. ZSL-KG-GAT projects the neighbourhood nodes to a new features h ((-1) = Wh(-"), The neighbourhood node features are concatenated with self feature and passed through self-attention module for get the attention coefficients. The attention coefficients are multiplied with the neighbourhood features to the get the node embedding for the I-th layer in the combine function. ZSL-KG-RGCN uses relational aggregator to learn the structure of the neighbourhood. To avoid overparameteerizatioo from the relational weights, perform basis decomposition of the weight vector into B bases. We learn B relational coefficients and [B] weight vectors in the aggregate function and add with the solf feature in combine function. ZSI-KG-LSTM uses LSTM as an aggregator to combine the neighbourhood features. The nodes in the graph are passed through an LSTM and the last hidden state is taken as the aggregated vector. The aggregated vector is concatenated with the node's previous layer feature and passed to the combine function to get the node representation.
M Hyperparameters
In this section, detail the hyperparameters used in our experiments.
26

---

Published in Transactions on Machine Learning Research (08/2022)
M.1 Training
Our framework is built using Py and AllenNLP (Gardner et al. 2018): In all our experiments, use Adam (Kingma & Ba 2015) to train our parameters with a learning rate of 0.001, unless provided in the experiments. We set the weight decay to 5e 04 for OntoNotes, ImageNet, AWA2, and aPY and 0.0 for BBN. For intent classification, we experiment with weight decay le 05 and 5e 05. We found that weight decay of 5e-05 gives the best performance overall for all the baseline graph aggregators and 1e-05 for ZSL-KG. We set the weight decay 0.0 when fine-tuning the ResNet backbone with SGD for ImageNet, AWA2, and aPY.
For fine-grained entity typing, we assume low-rank for the compatibility matrix W. The matrix W E Rdexd. is factorized into A E gixd, and B E IRd,xh where de is the size of example representation, de is the size of the the class representation, and h is the size the joint semantic space. Table 15 summarizes the hyperparameters used the example encoders which a biLSTM with attention or a task-specific variant of
it.
M.2 Graph Neural Networks
Task
layer-1 1
layer-2
Object classification
2048
2049
Intent classification
64
64
Fine-grained entity typing
128
128
Table 16: Output dimensions for the layers in the graph neural networks.
Table 16 details the output dimensions of the graph neural network layers. ZSL-KG-GAT uses LeakyReLU activation in the attention. LeakyReLU has a negative slope of 0.2. ZSL-KG-RGCN learns B bases weight vectors in the baseline. We found that B performs the best for fine-grained entity typing and object classification. In fine-grained entity typing, the activation function after the graph neural network layer is ReLU and following prior work in object classification, the activation function is LeakyReLU with negative slope of 0.2.
layer
layer 2
d(n)
d(p)
d(r)
d(p)
OntoNotes
150
150
32
64
BBN
250
150
32
64
SNIPS
150
150
32
32
AWA2/aPY
100
150
1024
1024
ImageNet
150
150
1024
1024
Table 17: The hyperparameters used in our transformer graph convolutional network.
In our transformer module, there are four hyperparameters- input dimension d(ı-1), output dimension dq-1)): feedforward layer hidden dimension d(f), and projection dimension dip)- The input dimension and output dimensions are the same in the aggregator. Table 17 details the hyperparameters used in our transformer graph convolutional networks. For fine- -re-ind entity typing, we manually tuned the hyperparameters. We tuned the hyperparamoters on the held-out validation classes for object classification.
27


Reviews:
[{"review_id": "LW8tFXST0S", "reviewer": "Reviewer_C1Vc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the idea of deploying a graph neural network on top of ConceptNet for ZSL very interesting. However, some claims of the article are not well supported by experimental evidence and/or algorithmic choices, thus I lean toward a negative rating for the current version of the manuscript. Below I detail what I deem as strengths and weaknesses of the work. \n\n**Strengths:**\n\n1. ZSL models generally assume that unseen classes are described in the same way as seen ones. Collecting precise attribute-level information for all categories can be expensive, especially with a large number of unseen classes. Using knowledge bases for ZSL is a good direction to reduce the burden of category-level attribute annotation, potentially leading to better results than word embeddings.\n\n2. The proposed TrGCN is sound, using a transformer to merge node information in a permutation invariant manner.\n\n3.  The approach achieves good results on a variety of benchmarks (i.e. almost 10% top-5 accuracy on unseen categories for ImageNet, Table 4), being either on par or superior to existing graph-based approaches. \n\n**Weaknesses:**\n1. The main weakness of the manuscript is that some claims are not well supported. Specifically:\n- Section 4 (second paragraph) underlines how encoding available knowledge graphs with existing models is computationally expensive. In the third paragraph, the text states that the proposed TrGCN addresses this problem. However, it is unclear how TrGCN reduces the computational cost given that i) it has a computational cost that is the power of two of the GAT models, and ii) neighborhood sampling is needed to reduce the computational cost during training (following previous works, e.g. Ying et al. 2018).  These claims should be either reduced or the current limitations of the model explicitly discussed. Note also that, given the focus of the manuscript on using knowledge bases (as for the title), it would be interesting to revise TrGCN in such a way to be computationally more efficient than existing models. It would also be helpful to add an analysis of the computational cost of various models, to assess which one is most viable for working with such large knowledge graphs.\n\n- Table 7 shows that, in some settings, existing GCN and GAT, outperform the proposed TrGCN even on the very challenging ImageNet benchmark. Despite these findings, all the discussion in Section 6 focuses on the advantage of the approach, and further ablation studies (Table 8 and Table 9) are conducted in the two settings (AWA2 and aPY) where TrGCN outperforms the competitors. Note also that Table 9 focuses on the comparison with GAT but does not show how GAT would perform on ImageNet, where it already outperforms TrGCN (according to Table 7). Not discussing the limitations of the approach or its possible drawbacks makes the content of the manuscript not complete and less informative for future works. I would like to stress that if the approach is weaker than the competitors in some settings it is fine, as far as the article properly acknowledges that and discusses the possible reasons, making them valuable starting points for future works. \n\n- In Section 2, the text highlights how existing large-scale models (e.g. CLIP) struggle in fine-grained recognition, while the proposed approach offers a flexible way to encode fine-grained categories. However, the experiments on object recognition are conducted in coarse-grained settings (i.e. AWA2, ImageNet, aPY). Showing results on common fine-grained datasets (e.g. CUB, Flowers, see for instance Xian et al. 2018b) would verify this claim. On the other hand, if case the model struggles in such a setting, the results would nevertheless be interesting for a discussion on the limitations of the approach.\n\n2. The article proposes to couple graph neural networks with transformers, with the TrGCN module. However, this is not the first attempt in the literature, as explored already in e.g. [a,b,c]. In Section 2, there is no discussion on how TrGCN relates to existing works in the same direction. This discussion should be added to stress the technical contribution, clarifying how TrGCN differs from existing works.\n\n3. The title highlights the impact of common sense knowledge graphs. However, the manuscript does not contain any analyses on how the performance of the model varies w.r.t. the used knowledge base (e.g. what if ConceptNet is replaced by WordNet? How would the results of ZSL-KG vary?). This type of analysis would clarify the impact of the knowledge graph w.r.t. one of the proposed technical components, and how sensible the latter is to the underline graph.\n\n4. Following on the previous, it would be helpful to see how the performance of the model compares with existing ZSL models when paired with commonsense knowledge. ConceptNet provides related word embeddings (i.e. ConceptNet Numberbatch). How does ZSL-KG compare with any of the approaches in Table 6 if coupled with such embeddings? And what happens if DGP and SGCN use ConceptNet as a graph rather than WordNet (Section 5.4)? This would allow assessing whether the proposed approach is the best in exploiting commonsense knowledge or whether existing ones can already suffice.\n\n**Minors:**\n- $y_n$ denotes a sample in the training set in Section 3, but it is also used to denote an unseen sample and a test sample. I would re-consider writing a more generic $y_i$ when referring to it. \n- The set $Y_{U+S}$ is not defined.\n\n**References:**\n\n[a] Yun, Seongjun, et al. \"Graph transformer networks.\" NeurIPS 2019.\n\n[b] Ying, Chengxuan, et al. \"Do Transformers Really Perform Badly for Graph Representation?.\"  NeurIPS 2021.\n\n[c] Hu, Ziniu, et al. \"Heterogeneous graph transformer.\" The Web Conference 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The tackles zero-shot learning (ZSL) by exploiting common-sense knowledge graphs. Specifically, the article proposes a new graph neural network, TrGCN, that uses a transformer to aggregate the information of multiple nodes in the graph. The final aggregated information is compared with the features extracted from the input through a dot product. Experiments show that ZSL-KG achieves state-of-the-art results in multiple benchmarks, generally outperforming existing graph-based methods for ZSL.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Following on the above weaknesses, I deem it critical the acceptance that the article:\n1. Revises existing claims on the efficiency of TrGCN.\n2. Discusses the weaknesses of the proposed approach, especially in the light of Table 7.\n3. Expands the related works with existing methods merging GCN and Transformers.\n4. Provides deeper evidence on the impact of the knowledge base on the final performance and whether the proposed method is the best in exploiting such a source of knowledge.\n\nTo strengthen the work, it would be helpful to show and discuss the results of the model on fine-grained object classification.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sYYZ2qnDDs", "reviewer": "Reviewer_cjHE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The related work and background sections of the paper is extensive and I believe it would benefit readers with less expertise on this topic. The paper is well organized and the quality of presentation is above the bar for TMLR. \n\nIt is also good to see that the experiments cover a diverse range of zero-shot classification applications including Fine-Grained Entity Typing, Intent Classification, and Object Classification. \n\nI appreciate that the authors being upfront about the commonalities and differences between the proposed work and related work including GCGZ, DGP, and GAT. I agree that ZSL with KG as opposed to WordNet introduces new challenges such as the much larger number of nodes and the lack of directions in the graph. However, the difference between GAT and the propose TrGCN is somehow more subtle. It is good to see that empirically TrGCN can outperform GAT. \n\nThe proposed method is somewhat straightforward, simple, but seems effective. Given that the guidelines for TMLR focus more on correctness than on impact/significance, I would say that this paper is above the bar for TMLR. \n\nThe exploration of using KG as a means to learn class representations for zero-shot classification is useful and the conclusions seem solid and convincing. \n\nGiven KG and WordNet are somewhat different sources of information, it would be interesting to see if they are complementary, that is whether combining them (e.g., DGP and ZSL-KG) would lead to further performance improvement. \n\nWhile the experiments show some promise, the baselines used seem a bit outdated. The latest baseline for all three tasks is from 2019. It is therefore unclear that the claim that the proposed method achieves new state-of-the-art accuracies is true. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed transformer GCN (TrGCN) to address the problem of zero shot learning, or specifically zero-shot classification. The idea is to rely on the knowledge graph (KG) to generate class attributes that are later used for zero-shot classification; this is in contrast to prior works that use word embeddings or a curated graph of words, e.g., WordNet (though to me, KG is already quite close to WordNet as they are both graphs). TrGCN maps nodes in the common sense KG to class attributes (representations) for zero-shot classification. Experiments show that the proposed TrGCN achieve performance that either surpasses or comparable to some existing methods. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Given KG and WordNet are somewhat different sources of information, it would be interesting to see if they are complementary, that is whether combining them (e.g., DGP and ZSL-KG) would lead to further performance improvement. \n\nWhile the experiments show some promise, the baselines used seem a bit outdated. The latest baseline for all three tasks is from 2019. It is therefore unclear that the claim that the proposed method achieves new state-of-the-art accuracies is true. \n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mUx8vOWtSv", "reviewer": "Reviewer_MySv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------------\n\n1) TrGCN is a transductive and attribute-free approach, which makes it **easy to apply it to any problem** without making too many prior assumptions.  \n2) The modeling is general enough to be applied to a wider variety of problems in vision or NLP.\n3) The paper is **well written and easy to follow**.\n4) SOTA performance on few tasks on zero shot learning.\n5) The overall method is **simple and has lot of potential** to unlock future developments in this space.\n\n\nClarifications requested\n----------------------------\n\n1) It is not clearly explained how or why the class embedding learned by different nodes in the graph would be different from GLoVE/Word2Vec embedding which also model the relationships between words from natural text corpora. A comparison by just using glove (that is not training the graph neural network and just using the initial embedding as is) will make this more clear.\n\n2) The performance is almost on par with using GAT on most datasets. Additionally, GAT has time complexity O(N.$m$) for N nodes and m neighbors, while current method is O(N.$m^2$), so a more detailed analysis on why current method is better than a more simpler GAT is missing. While Table 9 indicates multiple heads might hurt accuracy for GAT, a more accurate comparison should be made using multiple heads for current methods as well. \n\n3) If two nodes (words) have same neighbors, then according to your formulation, they have exactly the same representation. Firstly, is this possible? If so, is it a good or bad thing for the current problem?\n\n4) Sec 5.3 - \"_Table 5 shows that ZSL-KG, on average, achieves a significantly higher harmonic mean than the graph-based methods on AWA2 and aPY_.\" -  this statement is not true for AWA2 I suppose?\n\n5) \"_We suspect that pretraining ZSL-KG class representations on ILSVRC 2012 classes help the performance_.\" How is this pretraining carried out? No mention of this is made in the text or experiment setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a new method for attribute-free zero-shot learning for NLP and vision problems by leveraging knowledge from concept graphs. They introduce a Graph Attention Transformer to model relations between nodes in a graph, where the node embedding can then be used as a class embedding (class representations). Additionally, TrGCN works with dynamic graphs and hence can be used to predict over arbitrary label spaces at test-time with no knowledge of zero-shot (unseen) classes apriori.\n\nCompared to established notion of using attribute-based learning and reasoning over seen and unseen classes, this paper takes a new approach to directly infer the class representations from concept graphs which already have rich hierarchical relationships modeled between words.\n\n\n\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, I think this paper is okay as is and doesn't require significant changes. Addition of experiments as requested above can be helpful to provide further insights into the method. Few spelling mistakes, which need to be corrected in the final version.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles zero-shot learning, where unseen categories must be classified given training only on a dataset with non-overlapping seen categories and, in this case, additional knowledge through as knowledge graphs. The method proposes to use rich knowledge graphs, such as ConceptNet, to learn class representations and the authors propose a transformer-based permutation-invariant non-linear combination function across the neighborhoods. Results are shown across a set of vision and language datasets, including some that are coarse and some that are fine-grained. Overall, the method improves results over the current state-of-art, though not uniformly so. 

The reviewers appreciated several aspects of this work, including great writing and structure, the move away from pre-defined attributes and limitations on the knowledge graph (e.g. DAG structures), and results on diverse datasets. However, they pointed out a number of concerns as well, including 1) Increased computational complexity, 2) Fair description of performance claims w.r.t. to similar methods such as GAT (which does better in some cases), and 3) An understanding of how much of the benefits come from richer knowledge graphs (ConceptNet vs. WordNet) or the transformer module. After a thorough rebuttal (including some new results) and discussion with the authors, the reviewers recommended acceptance. Of course the authors should add all revisions coming out of the discussion. Further, it would be valuable to more directly address the computational complexity question in the revision: Specifically, quantitative comparisons of computation and memory would be valuable to the community to clearly show how much is being added to resource requirements; while big O notation is great, there are often large constants or other factors in how the method actually changes these requirements and it should be easy to do such a quantitative characterization. 

Overall, assuming such revisions are made, the paper is well-written, provides an interesting update to the use of knowledge graphs, supporting richer structures and proposing an updated transformer-based combination, and does a good job of making reasoned claims and situating the work fairly w.r.t. to prior work. Given the rise of multi-modal models such as CLIP for zero-shot learning, the introduction of richer structures is of interest to the community and this paper does a good job of summarizing this line of work while also proposing a new successful method. Therefore, I recommend acceptance conditioned on revisions.
Decision: Accept (poster)