Zero-Shot Learning with Common Sense Knowledge Graphs
Authors: Nihal Nayak
Stephen Bach
Venue:  (2021)
Keywords: Zero-Shot Learning
Common Sense Knowledge Graphs
Graph Neural Networks

Abstract: Zero-shot learning relies on semantic class representations such as hand-engineered attributes or learned embeddings to predict classes without any labeled examples. We propose to learn class representations from common sense knowledge graphs. Common sense knowledge graphs are an untapped source of explicit high-level knowledge that requires little human effort to apply to a range of tasks. To capture the knowledge in the graph, we introduce ZSL-KG, a general-purpose framework with a novel transformer graph convolutional network (TrGCN) to generate class representations. Our proposed TrGCN architecture computes non-linear combinations of the node neighbourhood and leads to significant improvements on zero-shot learning tasks. We report new state-of-the-art accuracies on six zero-shot benchmark datasets in object classification, intent classification, and fine-grained entity typing tasks. ZSL-KG outperforms the specialized state-of-the-art method for each task by an average 1.7 accuracy points and outperforms the general-purpose method with the best average accuracy by 5.3 points. Our ablation study on ZSL-KG with alternate graph neural networks shows that our transformer-based aggregator adds up to 2.8 accuracy points improvement on these tasks.

Under review as a conference paper at ICLR 2021
ZERO-SHOT LEARNING WITH COMMON SENSE KNOWLEDGE GRAPHS
Anonymous authors Paper under double-blind review
ABSTRACT
Zero-shot learning relies on semantic class representations such as hand-engineered attributes or learned embeddings to predict classes without any labeled examples. We propose to learn class representations from common sense knowledge graphs. Common sense knowledge graphs are untapped source of explicit high-level knowledge that requires little human effort to apply to range of tasks. To capture the knowledge in the graph, we introduce 7SL,KG, a general-purposs framework with a novel transformer graph convolutional network (TrGCN) to generate class representations. Our proposed TrGCN architecture computes non linear combina- tions of the node neighbourhood and leads to significant improvements on zero-shot learning tasks. We report new state-of-the-are accuracies six zero-shot bench- mark datasets in object classification, intent classification, and fine- fine-grained entity typing tasks. ZSL-KG outperforms the specialized state-of-the-art method for each task by an average 1.7 accuracy points and outperforms the general- purpose method with the best average accuracy by 5.3 points. Our ablation study on ZSL-KG with alternate graph neural networks shows that our transformer-basee aggregator adds up to 2.8 accuracy points improvement on these tasks.
INTRODUCTION
Deep neural networks require large amounts of labeled training data to achieve optimal performance. This is a severe bottleneck, as obtaining large amounts of hand-labeled data is an expensive process. Zero-shot learning is a training strategy which allows a machine learning model to predict novel classes without the need for any labeled examples for the new classes (Romera-Pareee
& Torr 2015)
Socher al., 2013 Wang al. 2019). Zero-shot models learn parameters for seen classes along with their class representations. During inference, new class representations are provided for the unseen classes. Previous zero-shot learning systems have used sand-engineered attributes (Akata
et al. 2015 Farhadi et al.) 2009; Lampert et al. 2014), pretrained embeddings (Frome et al. 2013) and learned embeddings (e.g. sentence embedding) (Xian et al. 2016) as class representations.
Class representations in a zero-shot learning framework should satisfy the following properties: (I) they should adapt to unseen classes without requiring additional human effort, (2) they should provide rich features such that the unseen classes have sufficient distinguishing characteristics among themselves, (3) they should be applicable to a range of downstream tasks. Previous approaches for class representations have various limitations. On end of the spectrum, attribute- based methods provide rich features but the attributes have to be fixed ahead of time for the unseen classes. On the other end of the spectrum, pretrained embeddings such as GloVe (Pennington et al. 2014) and Word2Vec (Mikolov et al., 2013) offer the flexibility of easily adapting to new classes but rely on unsupervised training on large corpora-whit characteristics necessary for zero-shot learning. Many methods lie within the spectrum and learn class representations for zero-shot tasks from descriptions such as attributes, text, and image prototypes. Existing approaches that have achieved state-of-the-an performance make task task-specific adjustments and cannot exactly be adapted to tasks in different domains (Liu et al. 2019a Verma et al., 2020). Methods using graph neural networks on the ImageNet graph to learn class representations have achieved strong performance on zero-shot object classification (Kampffmeyer et al. 2019 Wang et al. 2018). These methods general- purposs,, since show that they can be adapted to other tasks as well. However, the ImageNet graph may not provide rich features suitable for a wide range of downstream tasks.
may not provide distinguishing

---

Under review as a conference paper at ICLR 2021
In work, we propose learn class representations from common sense knowledge graphs.
Common sense knowledge graphs (Liu & Singh] 2004; Speer et al. 2017 Tandon et al. 2017; Zhang
et al. 2020) are untapped source of explicit high-level knowledge that requires little human effort to apply to a range of tasks. These graphs have explicit edges between related concept nodes and provide valuable information to distinguish between different concepts. However, adapting existing zero-shot learning frameworks to learn class representations from common sense knowledge graphs challenging in several ways. GCNZ (Wang et al. 2018) learns graph neural networks with symmetrically normalized graph Laplacian, which not only requires the entire graph structure during training but also needs retraining the graph structure changes, i.e., GCNZ is not inductive. Common sense knowledge graphs can be large (2 million to 21 million edges) and training a graph neural network on the entire graph can be prohibitively expensive. DGP (Kampffmeyer et al., 2019) is an inductive method and aims to generate expressive class representations but assumes a directed acyclic graph such as WordNet. Common sense knowledge graphs do not have a directed acyclic graph structure.
To address these limitations, we propose ZSL-KG, a general-purpose framework with a novel transformer graph convolutional network (TrGCN) to learn class representations. Graph neural networks learn to represent the structure of graphs by aggregating information from each node's neighbourhood. Aggregation techniques used in GCNZ, DGP, and most other graph neural network approaches are linear, in the sense that they take a (possibly weighted) mean or maximum of the neighbourhood features. To capture the complex information in the common sense knowledge graph, TrGCN learns transformer-base based aggregator to compute the non-linear combination of the node neighbours. A few prior works have considered LSTM-based aggregators (Hamilton al. 2017a as way to increase the expressivity of graph neural networks, but their outputs can be sensitive to the ordering imposed on the nodes in each neighborhood. For example, on the Animals with Attributes dataset, we find that when given the same test image 10 times with different neighborhood orderings, an LSTM-based graph neural network outputs inconsistent predictions 16% of the time (Appendix
A.. One recent work considers trying to make LSTMs less sensitive by averaging the outputs over permutations, but this significantly increases the computational cost and provides only a small boost to prediction accuracy (Murphy al.) 2019). In contrast, TrGCN learns transformer-based aggregator, which is non-linear and naturally permutation invariant. Additionally, our framework is inductive, ie., the graph neural network can be executed on graphs that are different from the training graph, which is necessary for inductive zero-shot learning under which the test classes are unknown during training.
We demonstrate the effectiveness of our framework on three zero-shot learning tasks in vision and language: object classification, intent classification, and fine-grained entity typing. We report new statc-of-the-art accuracies on six zero-shot benchmark datasets (Xian et al., 201 8aj Farhadi et al. 2009 Deng et al., 2009] [ouckee et al., 2018: Gillick al. 2014] Weischedel & Brunstein, 2005). ZSL-KG outperforms the state-of-the th--te--rttttttt-- specialized method for each task by an average 1.7 accuracy points. ZSL-KG also outperforms GCNZ, the best general-purpos method on average by 5.3 accuracy points. Our ablation study on ZSL-KG with alternate graph neural networks shows that our transformer-basee aggregator adds up to 2.8 accuracy points improvement on these tasks.
In summary, our main contributions are the following:
1. We propose to learn class representations from common sense knowledge graphs for zero- shot learning.
2. We present ZSL-KG, a general-purpose purpose framework based on graph neural networks with a architecture. Our proposed architecture learns non-lineat combina- novel transformer-basee
tion of the nodes neighbourhood and generates expressive class representations.
3. ZSL-KG achieves new state-of-the-art accuracies Animals with Attributes 2 (Xian et al. 2018a), aPY (Farhadi et al.] (2009), ImageNet (Deng et al. 2009). SNIPS-NLU (Coucke
et 2018), Ontonotes (Gillick et al., 2014), and BBN (Weischedel & Brunstein, 2005).
2 BACKGROUND
In this section, we summarize zero-shot learning and graph neural networks.
2

---

Under review as a conference paper at ICLR 2021
2.1 ZERO- TROO-SHOT LEARNING
Zero-shot learning has several variations (Wang et al. 2019 Xian et al. 2018a). Our work focuses on inductive zero-shot learning, under which we do not have access the unseen classes during training. We train a zero-shot classifier by optimizing over the seen classes. But, unlike traditional methods, zero-shot classifiers are trained along with class representations such as attributes, pretrained embeddings, etc.
Recent approaches learn a class encoder ((y) E Rd produce vector-valued class representations from an initial input, such as string or other identifier of the class. (In our case, y is a node in a graph and its k-hop neighborhood.) During inference, the class representations are used to label examples with the unseen classes by passing the examples through an example encoder ((x) E Rd and predicting the class whose representation has the highest inner product with the example representation.
Recent work in zero-shot learning commonly uses one of two approaches to learn the class encoder d(v). One approach uses a bilinear similarity function defined by a compatibility matrix W E gdxd
(Frome et al., 2013 Xian et al. 2018a):
f (0(x), W, wv((v) ((z) Wo(y)
(I)
The bilinear similarity function gives a score for each example-clan pair. The parameters of 8, W, and @ are learned by taking a softmax over for all possible seen classes y E Ys and minimizing either the cross entropy loss or a ranking loss with respect to the true labels. In other words, f should give a higher score for the correct class(es) and lower scores for the incorrect classes. W is often constrained to be low rank, to reduce the number of learnable parameters (Obeidat et al. 2019 Yogatama et al. 2015). Lastly, other variants of the similarity function add minor variations such as non-linearities between factors of W (Socher et al. 2013 Xian et al. 2016).
The other common approach final fully connected layer of this network has vector representation for each seen class, and the remaining layers are used as the example encoder (xx) Then, the class encoder o(y) trained by minimizing the 1.2 loss between the representations from supervised learning and Φvy) (Kampffmeyer
first train a neural network classifier in a supervised fashion. The
ctt al. 2019; Socher et al. 2013 Wang et al. 2018).
The class encoder that we propose in Section caan be plugged into either approach.
2.2
GRAPH NEURAL NETWORKS
The basic idea behind graph neural networks is learn node embeddings that reflect the structure of the graph (Hamilton et al. 2017b). Consider the graph G = (V, E, R). where V is the set of vertices with node features X, and (U:, T, v2) E E are the labeled edges and r E are the relation types. Graph neural networks learn node embeddings by iterative aggregation of the k-hop neighbourhood. Each layer of graph neural network has two main components AGGREGATE and COMBINE (Xu et al. 2019)
L AGGREGATE" ((N5-V Vu EN(())
(2)
where a E d is the aggregated node feature of the neighbourhood, ha (1-1) is the node feature in neighbourhood (V)) of node v including self loop. The aggregated node passed to the COMBINE to generate the node representation te E IRd for the (-th layer:
h = COMBINE (n0-"),a!)
(3)
S = 2. where v, is the initial feature vector for the node. Previous works on graph neural networks for zero-shot learning have used Ve (Pennington et al. 2014) to represent the initial features Kampffmeyer et al. 2019: Wang et al. 2018).
THE ZSL-KG FRAMEWORK
Here introduce ZSL-KG: a general-purpose framework with novel Transformer Graph Convo- lutional Network to learn class representation from commmon sense knowledge graphs. Figure] shows the ZSL-KG architecture with an example elephant concept.
3

---

Under review as a conference paper at ICLR 2021
Ivory
Transformer gratfe
cage
wild
girafle
GNN
20a
wairus
produutt
elephant
2o0
lusk
Mean pooling
tusk
Example
Class
walrus
rep.
Score
wild bowr
hvrry
alephant
Figure 1: Left:A sample from the 2-hop neighbourhood for the concept e ephant from ConceptNet. Right: The figure describes the architecture for ZSL-KG. The image of an elephant is passed through the example encoder (ResNet101) to generate the example representation for the image. For the class representation, we take the sampled k-hop neighbourhood and pass the nodes through their hop-specific Transformer Graph Convolutional Network. We take the dot product of the example representation and the class representation to compute the score for the class. The same architecture followed for language tasks with a task-specific example encoder.
3.1 COMMON SENSE KNOWLEDGE GRAPHS FOR ZERO-SHOT LEARNING.
Common sense knowledge graphs organize high-level knowledge implicit to humans in graph. The nodes in the graph are concepts associated with each other via edges. These associations the graph offer rich source of information, which makes them applicable a wide range of tasks. Publicly available common sense knowledge graphs range roughly from 100,000 to million nodes and million to 21 million edges (Speer al. 2017 Zhang et al. 2020). To learn class representations from common sense knowledge graphs, we look to graph neural networks.
Existing zero-shot learning methods such GCNZ (Wang al. 2018) and DGP (Kampfimeyer et al. 2019) that learn class representations from structured knowledge are applicable only to small graphs such ImageNet or Wordnet as they make restrictive assumptions. GCNZ requires the entire graph structure during training, and learns class representations from the ImageNet graph which is significantly smaller than common sense knowledge graphs. For instance, the ImageNet graph used in GCNZ has about 32,000 nodes and 65, 55,000 edges. DGP learns more expressive class representation but requires a directed acyclic graph or parent-child relationship in the graph. We are not restricted to parent-child relationships in a common sense knowledge graph.
3.2 TRANSFORMER GRAPH CONVOLUTIONAL NETWORK.
To overcome these limitations, we propose to learn class representations with novel graph neural network: transformer graph convolutional networks (TrGCN). Transformers (Vaswani et al.) 2017) are non-linear modules typically used for machine translation and language modeling tasks. They achieve non-linear combination of the input sequences using two-layer feedforward neural networks and scaled dot product attention. We exploit this property to learn a non-linear aggregator that captures the complex structure of common sense knowledge graph. Finally, they are not tied the graph structure which makes them well suited for inductive zero-shot learning.
Here describe TrGCN. We pass the neighbhourhood node features h (1-1) through two-layer feedforward neural network with a ReLU activation between the layers. The previous features are added to its output features with skip connection, followed by layer normalization (Ba et al. 2016):
h = LayerNorm W ReLU (w ( [ + (1-1) Vu N())
(4)
where ( (1) hf e xddd and ((() xdq--) are learnable weight matrices for the feedfor- ward neural network. The non-linear neighbourhood features are then passed through the scaled dot product attention layer to compute the weighted combination of the features for each query node:
QKT
{ 20 Vu NNo) = softmax
V
(5)
ViT
4

---

Under review as a conference paper at ICLR 2021
(1) d'D 1 ht is the of all neighbourhood query vectors, K W hig 1i11 is the
where Q
set of all key vectors, V = w hn " is the set of values vectors, and Wg E Rda-1)x 1)xde) We E Rd Xd W, E Rd are learnable weight matrices with the projection dimension d(p)- The output features from the attention layer is projected with another linear layer and added to its previous features with skip connection, followed by layer normalization:
{ Fu Yu E [No) = LayerNorm (
(0) zu
(1-1) + h 10-1 Vu EuE))
(6)
where !! E R x))- a learnable weight matrix.
th) To get the aggregated vector a for node u, we pass the output vectors { Vu E NN)) from the transformer through a permutation invariant pooling function μa.)) such as mean-pooling. The aggregated vector is passed through a linear layer followed by non-linearity a such as ReLU or Lea:yReLU:
1 (f=% Vu EwEN))))))))
h (ww).. a!
where (' E 12 xda) is a learnable weight matrix.
Existing work has drawn parallels between transformers and graph attention networks (GAT), sug- gesting that they are equivalent (Joshi 2020). GAT (Veličković et al. 2018) computes the aggregated vector by taking the linear combination of the node features in the neighbourhood. In contrast, TrGCN learns transformer-based aggregator and computes a non-linear combination of the node features in the neighbourhood, followed by pooling function to get the aggregated vector which leads the difference in architecture.
Neighbourhood Sampling. In our experiments, we use ConceptNet (Speer et al., 2017) our common sense knowledge graph but our approach applies to other knowledge graphs. ConceptNet has high node degree, which poses a challenge to train the graph neural network. To solve this problem, we explored numerous neighbourhood sampling strategies. Existing work on sampling neighbourhood includes random sampling (Hamilton et al., (017a), importance sampling (Chen) et al. 2018a), random walks (Ying et al., 2018). etc. Similar to PinSage (Ying et al. (018), we simulate random walks for the nodes in the graph and assign hitting probabilities to the neighbourhood nodes. During training and testing the graph neural network, select the top N nodes from the neighbourhood based on their hitting probability.
4 TASKS AND RESULTS
We evaluate our framework on three zero-shot learning tasks: object classification, intent classification, and fine-gn fine-grained entity typing. In all our experiments, compare ZSL-KG with the state-of-the-a art specialized methods for the task as well as general-purpose methods: GCNZ, SGCN, and DGP. The code and hyperparameters are included in the supplementary material, which will be released upon acceptance.
Setup. We two-layer graph neural networks for the general-purpose baselines and ZSI-KG. We use the code from Kampffmeyer et al. (2019) and adapt GCNZ (Wang et al.) 2018), SGCN, and DGP (Kampffmeyer | (019)) for zero-shot learning tasks in language (See Appendix C.I). In all our experiments with ZSL-KG, we map each class to a node in ConceptNet 5.7 (Speer et al. 2017) and query its 2-hop neighbourhood. We simulate random walks for each node in the ConceptNet graphs and compute the hitting probability for the nodes in the neighbourhood. Then, we sample 50 and 100 node neighbours with the highest hitting probabilities for the first and the second hop, respectively. See Appendix C.2 for more details.
4.1 OBJECT CLASSIFICATION
Object classification a computer vision task of categorizing objects into different class categories.
Datasets. We evaluate the Animals with Attributes 2 (AWA2) (Xian et al. 2018a), attribute Pascal Yahoo (aPY) (Farhadi et al.) 2009). and ImageNet datasets (Deng et al. 2009). AWA2 contains images of animals with 40 classes in the train set and 10 classes in the test set. aPY contains images
5

---

Under review as a conference paper at ICLR 2021
AWA2
aPY
SNIPS-NLU
Accuracy
Accuracy
Accuracy
SP-AEN
59.7
37.0
Zero-shot DNN
71.16
LisGAN
55.8
36.8
IntentCapsNet
77.52
ZSML
77.5
64.0
ReCapsNet-ZS
79.96
GCNZ
77.00 + 2.05
56.02 + 0.63
GCNZ
82.47 + 03.09
SGCN
77.76 1.60
54.98 + 0.66
SGCN
50.27 + 14.13
DGP
76.26 + 1.69
52.22 + 0.49
DGP
64.41 + 12.87
ZSL-KG
78.08 0.84
64.36 0.59
ZSL-KG
88.98 + 01.22
Table I: Results for object classification on the AWA2 and aPY dataset. We report the average class-balee basssbbllancdd accuracy of the models on 5 ran- dom seeds and the standard error. The results for SP-AEN, LisGAN, and ZSML are obtained from
Table 2: Results for intent classification on the SNIPS-NLU dataset. We report the average accuracy of the models on 5 random seeds and the standard error. The results for Zero-shot DNN, IntentCapsNet, and ReCapsNet-ZS are obtained from Liu et (0119a).
Verma et al. 2022
Hit@Fk(S))
2- H-Hops
3. 3-Hops
All
2
5
10
20
2
20
7.5
14.2
20.2
27.7
3.3
ZSL
SGCN
26.2
40.4
60.2
71.9
81.0
6.0
10.4
18.9
27.2
36.9
2.8
4.9
9.1
13.5
19.3
DGP
26.6
40.7
60.3
72.3
81.3
10.7
19.3
27.7
37.7
3.0
5.0
9.3
13.9
19.8
ZSL-KG
26.8
41.3
61.5
73.2
82.0
6.6
11.4
20.8
30.0
39.7
3.1
5.5
15.3
21.7
GCNZ
9.7
20.4
42.6
57.0
68.2
2.2
5.1
11.9
18.0
25.6
1.0
2.3
5,3
8.1
11.7
GZSL
SGCN
11.9
27.0
50.8
65.1
75.9
3.2
7.1
16.1
24.6
34.6
1.5
3.4
7.8
12.3
18.2
DGP
10.3
26.4
50.3
65.2
76.0
7.1
16.1
24.9
35.1
1.4
3.4
12.6
18.7
ZSL- KG
11.5
26.5
51.2
65.8
76.5
3.5
7.7
17.5
27.0
37.4
1.8
3.9
8.8
13.9
20.5
Table 3: Results for object classification ImageNet dataset. We report the class-balanced top-k accuracy on zero-shot learning (ZSL) and generalized zero-shot learning (GZSL) for ImageNet classes k-hops away from the ILSVRC 2012 classes. The results for GCNZ, SGCN, and DGP are
obtained from Kampfimeyer et al. (2019).
of objects with classes in the train set and 12 classes the test set. ImageNet contains images from 1000 classes in the train set and 21K classes in the test set.
Experiment. Following prior work (Kampffmeyer et al. 2019: Wang al., 2018), here we use 1.2 loss architecture for zero-shot learning. The example encoder and seen class representations come from the ResNet 101 model (He et al. (2066) in Torchvision (Marcel & Rodriguez 2010) pretrained on ILSVRC 2012 (Russakovsky et al. 2015). We map the ILSVRC 2012 training and validation classes, and the AWA2 and aPY test classes to ConceptNet. The model trained for 1000 epochs on 950 random classes and the remaining 50 ILSVRC 2012 classes used for validation. We use the same setting for GCNZ, SGCN, and DGP using the authors' implementation. The model with the least loss on the validation classes is used make predictions ImageNet experiment, we train the model for 3000 epochs on 1000 classes from ILSVRC 2012 with the common sense knowledge graph and switch ImageNet graph during test time. Similar to DGP, we freeze the final layer with the generated class representations and fine-tune the ResNet- on the VVC images for epochs using SGD with a learning rate 0.0001 and momentum of 0.9. Following prior work, we report the class-balanced accuracy (Xian et al. 2018a) on unseen classes for AWA2 and aPY. We follow the train/test split from (Frome et al., 2213)) ) and evaluate ZSL-KG on two levels of difficulty: 2-hops, and 3-hops. The hops refer to the distance of the classes from the ILSVRC train classes. We evaluate ZSL-KG on two settings: zero-shot learning (ZSL) where only unseen classes are present and generalized zero-shot learning (GZSL) where both seen and unseen classes are present. Following previous work (Kampfiineyer et al. 2019) on ImageNet evaluation, report the balsncal top-K accuracy.
the test classes. For the

---

Under review as a conference paper at ICLR 2021
We compare ZSL-KG against state-of-tee oftte-of-hhe-ar specialized methods for AWA2 and aPY: SP-AEN (Chen
(et al., 2018b), LisGAN (Li et al. 2019). and ZSML (Verma et al. 2020). ZSML is a GGN-basee approach and has reported the highest results on AWA2 and aPY, and DGP and SGCN have reported the highest on results on ImageNet.
Results. Table ! shows the results for zero-shot object classification. ZSL-KG outperforms existing state-of-the-art methods on the AWA2 and aPY datasets. The general-purpose methods show a significant drop in accuracy from AWA2 aPY, whereas our method consistently achieves the highest accuracy on both the datasets. This suggests that the class representations generated from a richer graph help in performance. In contrast specialized methods, ZSL-KG trains only one model and does not require any specialized training datasets to achieve state-of-the-ar tht performance on both the datasets. Finally, our experiments with ImageNet show that ZSI,-KG, despite being trained on a noisier graph, reports the tate-of-the-an on zero-shot learning and generalized zero-shot learning. ESL-KKG offers accuracy improvements up to 2.3 points and relative improvements up to 20% over the state-of-the-art methods.
4.2 INTENT CLASSIFICATION
To assess ZSLs versatility, we experiment on zero-shot intent classification. Intent classification text classification task of identifying users' intent expressed in chatbots and personal voice assistants.
Dataset. We evaluate on the main open-source benchmark for intent classification: SNIPS-NLU (Coucke et al. (2018). The dataset was collected using crowdsourcing to benchmark the performance of voice assistants. The training set has seen classes which we split into 3 train classes and development classes.
Experiment. Zero-shot intent classification is a multi-class classification task. The example encoder used in our experiments is biLSTM with attention (Appendix (F). We train the model for 10 epochs by minimizing the cross entropy loss and pick the model with the least loss on the development set. We measure accuracy on the test classes.
We compare KSI against existing specialized kate-of-the-art methods the literature for zero- shot intent classification: Zero-shot DNN (Kumar et al., (017). Inten:CapsNett (Xia et al. 2018). and ResCapsNet-ZS (Liu et al.) 2019a). IntentCapsNect and ResCapsNeetZSS are CapsuleNet-bart (Saoour al. 2017) approaches and have reported the best performance on the task.
Results. Table shows the results. ZSL-KG significantly outperforms the existing approaches and improves the state-of-the-arr accuracy to 88.98%. The ceneral-purpont methods have mixed performance on intent classification and suggests ZSL-KG works well on a broader range of tasks.
4.3 FINE-GRAINED ENTITY TYPING
To test ZSL-KG's ability to classify finegrained types, we experiment on zero-shot fine- fine-grained entity typing. Fine- -eee-grrindd entity typing is the task of classifying named entities into one more narrowly scoped semantic types. This task also tests generalized zero-shot learning when seen classes appear in the test set.
Datasets. We evaluate on popular fine-gras fine-grained entity typing datasets: OntoNotes (Gillick et al. 2014) and BBN (Weischedel & Brunstein, 2005).
We split the dataset into two: coarse-grained labels and fine-grained labels. Following the prior work (Obeidat et al. 2019), we train the coarse- -raiind labels and predict on both coarse-grained and fine-grained labels in the test set. Details about the datasets can be found in Appendix
Experiment. Fine- gianngrained entity typing is zero-shot multi-label classification task because each entity can be associated with more than one type. We reconstructed OTyper (Yuan & Downey 2018) and DZET (Obeidat et al. (201). the state-of-the-arr specialized methods for this task. Both methods use the AttentiveNER biLSTM (Shimaoka et al. 2017) as the example encoder. See Appendix G for more details.
We train each model for 5 epochs by minimizing the cross-entropy loss. During inference, we pass the scores from the bilinear similarity model through sigmoid and pick the labels that have probability
7

---

Under review as a conference paper at ICLR 2021
Ontonotes
BBN
Strict Acc.
Loose Mic.
Loose Mac.
Strict Acc.
Loose Mic.
Loose Mac.
OTyper DZET
41.72 +: 0.44
55.00 0.37
61.25 0.53
25.76 + 0.25
54.75 = 0.68
62.88 0.67
42.88 + 0.47
56.16 0.41
62.26 = 0.46
26.20 + 0.13
57.31 = 0.24
63.44 0.30
GCNZ
41.46 + 0.81
54.61 + 0.52
61.65 = 0.52
21.47 + 1.22
47.17 = 1.04
56.81
0.68
SGCN
42.64 +: 0.69
56.25 + 0.29
62.93 0.39
24.91 + 0.24
50.02 = 0.73
59.68
0.54
DGP
41.11 + 0.78
54.08 + 0.62
61.38 0.76
23.99 + 0.14
47.19 = 0.51
57.62 0.89
ZSL-KG
45.21 0.36
55.81 0.41 63.64 = 0.51
26.69
2.41
50.30 + 1.21
57.66 1.15
Table 4: The results for zero-shot fine-grained entity typing on Ontonotes and BBN. We report the average strict accuracy, loose micro F1, and loose macro Fl of the models random seeds and the standard error.
AWA2
aPY
SNIP-NLU
OntoNotes
BBN
Accuracy
Accuracy
Accuracy
Strict Acc.
Strict Acc.
ZSL-KG-GCN
74.81 + 0.93
61.63 0.54
84.78 + 0.77
42.19 0.75
27.44 2.94
ZSL-KG-GAT
75.29 + 0.64
62.87 0.36
87.47 + 1.59
87.57
43.48 0.91
32.65 2.27
ZSL-KG-RGCN
66,27 + 1.01
54.20 0.63
43.88 0.85
27.89 1.47
ZSL-KG-LSTM
66.46 + 1.19
54.89 +: 0.74
88.81 + 1.17
44.52 = 0.22
26.77 = 0.63
ZSL-KG
78.08 + 0.84
64.36 + 0.59
88.98 + 1.22
45.21 0.36
26.69 2.41
Table 5: The results for zero-shot learning tasks with other graph neural networks. We report the average performance on 5 random seeds and the standard error.
of 0.5 greater as our prediction. As is common for this task (Obeidat et al. 2019),, we evaluate the performance of our model on strict accuracy, loose micro FI, and loose macro (Appendix H.. Striet accuracy penalizes the model for incorrect label predictions and the number of the label predictions have to match the ground truth, whereas loose micro Fl and loose macro Fl measures the correct label is predicted among other false positive predictions.
Results. Table shows the results. ZSL-KG outperforms the existing state-of-the-art specialized methods strict accuracy for both OntoNotes and BBN. DZET has higher loose micro on both the datasets because it overpredicts labels and has greater false positives compared to the ZSL-KG. 7SL.KG has higher precision for label predictions and therefore, higher striet accuracy compared to other methods. These results demonstrate that our method works well even in a generalized multilabel setting where the test set has multiple seen and unseen labels.
4.4 COMPARISON OF GRAPH AGGREGATORS
We conduct an ablation study with different aggregators with our framework. Existing graph neural networks include GCN (Kipf Welling 2017), GAT (Veličković al. 2018). RGCN (Schlichtkruil ct al. 2018) and LSTM (Hamilton et al. 2017a). We provide all the architectural details in Appendix We train these models with the same experimental setting for the tasks mentioned in their respective sections.
Results. Table shows results for our ablation study. Our results show that ZSL-KG almost always outperforms existing graph neural networks with linear aggregators, ZSL-KGLLSTM which uses an LSTM--based aggregator shows inconsistent performance across different tasks indicating that they are more useful on low-dimensional tasks such as node classification. With relational aggregators (ZSL-KG-RGCN), we observe that they do not outperform ZSL-KG and may reduce the overall performance (as seen in AWA2 and aPY). Finally, it is worth comparing SGCN and ZSL-KG-GCN as they use the same linear aggregator to learn the class representation but train on different graphs. We that ZSL-KG-GCN trained common sense knowledge graphs adds average improvement of 8.1 accuracy points across the tasks suggesting that the choice of graphs is crucial for downstream performance.
8

---

Under review as a conference paper at ICLR 2021
5 RELATED WORK
We broadly describe related works on zero-shot learning and graph neural networks.
Zero-Shot Learning. Zero-shot learning has been thoroughly researched in the computer vision
community for object classification (Akata et al. 2015 Farhadi et al. 20099] Frome et al., 2013
Lampert et al., 2014 Wang et al., 2019 Xian et al., 20] Recent works in zero-shot learning
have used graph neural networks for object classification (Kampfimeyer et al. 2019 Wang et al.
2018). In our work, we extend their approach to common sense knowledge graphs to generate class representations. Furthermore, we learn TrGCN, a novel graph neural network with a non-linear aggreagtor to learn the structure of common sense knowledge graphs. Recent work on zero-shot object classification has used attention over image regions to achieve strong results on the task (Zhu (et al. 2019 [iu et al. 2019b; Xie et al. 2019; Liu et al. 2020). In contrast, our work focuses on the class encoder (TrGCN), where we learn attention over the graph and could potentially complement methods that focus on image encoders. Recently work on zero-shot learning has focused on applying
attention over the image regions. (Zhu et al. 2019 Liu al. 2019b; Xie et al. [019 Liu et al. (020) but they focus on the example encoder, particularly, they apply attention over the image regions. Other notable works use generative methods for generalized zero-shot learning where both seen and unseen classes are evaluated at test time (Kumar Verma et al. 2018 Schonfeld et al. 22019). However, these methods still rely on hand-crafted attributes for classification. Zero-shot learning has been studied in text classification as well (Dauphin et al., 2013 Nam et al.) 2016 Pappas & Henderson 2019 Zhang et al., 2019b; Yin et al., 2019). Previously, ConceptNet has been used for transductive zero-shot text classification as shallow features for class representation (Zhang et al., 2019b). They use ConceptNet to generate a sparse vector which combined with pretrained embeddings and natural language description to obtain the class representation. On the other hand, we use ConceptNet to generate dense vector representations from a graph neural network and use them as our class representation. Finally, zero-shot fine-grained entity typing has been previously studied with variety of class representations (Ma et al. 2016 Obeidat et al. 2019; Yuan & Downey| 2018). We note that ZOE (Zhou et al., 2018) is specialized method for zero-shot fine-grained entity typing and achieves best results on the task. However, they use a subset of the test set that contains both seen and unseen types to tune the threshold parameters which reveals information about the unseen types and makes ZOE a transductive zero-shot learning method.
Graph Neural Networks. Recent work on graph neural networks has demonstrated significant improvements for several downstream tasks such as node classification and graph classification (Hamilton et al. 2017a b) Kipf& & Welling 2017 Veličković et al. [018 Wu et al. 2019). Extensions of graph neural networks to relational graphs have produced significant results in several graph-related tasks (Marcheggiani & Titov| 2017 Schlichtkrull et al.) 2018 [Shang et al.) 2019 Vashishth et al.,
2020). Previous work has used transformers with graph neural networks as method to generate meta- paths in the graph rather than as neighbourhood aggregation technique (Yun et al. 2019). A related work Zhang et al. (2019a) combines common sense knowledge graph and graph neural networks for zero-shot learning. ZSL-KG learns to map nodes in knowledge graph to class representations without any other human input. In contrast, Zhang al. (2019a) learns to transfer hand-engineered attributes to new nodes in the graph. On the other hand, our work learns a graph neural network over the common sense knowledge graph to generate rich class representations. Finally, several diverse applications using graph neural networks have been explored: common sense reasoning (Lin) al. 2019), fine-grained entity typing (Xiong et al. 2019). text classification (Yao et al. 2019) reinforcement learning (Adhikari et al. 2020) and neural machine translation (Bastings el al.
2017
For more in-depth review, we point readers to Wu et al. (2020).
CONCLUSION
ZSL-KG is flexible framework for zero-shot learning with common sense knowledge graphs and can be adapted to a wide range of tasks without requiring additional human effort. Our framework introduces novel transformer graph convolutional network (TrGCN) which captures the complex associations in the graph to learn class representations. We achieve state-of-the-as performance on five benchmark datasets across three zero-shot tasks. Our work demonstrates that common sense knowledge graphs are a powerful source of high-level knowledge and can benefit range of tasks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ashutosh Adhikari, Xingdi Yuan, Mare-Alexandee Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and William L Hamilton. Learning dynamic knowledge graphs to generalize on text-based games. arXiv preprint 2020.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE Transactions Pattern Analysis and Machine Intelligence (PAMI), 2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv:1607.06450, 2016.
arXiv preprint
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an an. Graph convolu- tional encoders for syntax-award neural machine translation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5327-5336, 2016.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. International Conference Learning Representations (ICLR), 2018 8a.
Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recognition using semaatics-preserrinng adversarial embedding networks. In CVPR, 2018b.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. Snips voice plat- form: an embedded spoken language understanding system for private-by-desigg voice interfaces. ICML workshop on Privacy in Machine Learning and Artificial Intelligence, 2018
Yann N Dauphin, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. Zero-shot learning for semantic utterance classification. arXiv preprint arXiv: 0509, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Ali Farhadi, Ian Endres, Derek Hoiem, and David A. Forsyth. Describing objects by their attributes. Proceedings of the IEEE Conference on Computer Vision and Patern Recognition (CVPR), 2009.
Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. Allennlp: A deep semantic natural language processing platform. 2017.
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. Context-dependent fine ime-grained entity type tagging. arXiv preprint 4rXiv:1412.1820, 2014.
Will Hamilton, Zhitao Ying, and Jure Leskovee. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2017a.
William Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data Engineering Bulletin, 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
Chaitanya Joshi. Transformers are graph neural networks. The Gradient, 2020.
10

---

Under review as a conference paper at ICLR 2021
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang. Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks In International Conference on Learning Representations (ICLR), 2017.
Anjishnu Kumar, Pavankumar Reddy Muddireddy; Markus Dreyer, and Bjöm Hoffmeister. Zero-shot learning across heterogeneous overlapping domains. Proc. Interspeech 2017, pp. 2914-2918 2017.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learning via synthesized examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-ban -babed classification for zero-shot visual object categorization. IEEE Transactions on Pullern Analysis and Machine Intelligence (PAMI), 2014.
Jingjing Li, Mengmeng Jing, Ke Lu, Zhengming Ding, Lei Zhu, and Huang. Leveraging the invariant side of generative zero-shot learning. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 7402-7411, 2019.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Xiao Ling and Daniel S Weld. Fine-grained entity recognition. In AAAI Conference on Artificial Intelligence (AAAI), 2012.
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert YS Lam. Reconstructing capsule networks for zero-shot intent classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019a.
Hugo Liu and Push Singh. Conceptnet-a practical commonsense reasoning tool-kit. BT technology journal, 2004.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengyi Zhang. Attribute propagation network for graph zero-shot learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34: 4868 4875, Apr. 2020. doi: 101.:///aaa..3444.555.. index. http::/p/////atttt////////////// 59223
URL httph: w.w.aaaai....................... .rg/
Yang Liu, Jishun Guo, Deng Cai, and Xiaofei He. Attribute attention for semantic disambiguation in zero-shot learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6698-6707, 2019b.
Yukun Ma, Erik Cambria, and Sa Gao. Label embedding for zero-shot fine-grained named entity typing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Pupers, 2016.
Sébastien Marcel and Yann Rodriguez. Torchvision: The machine-vision package of torch. In International Conference on Multimedia, 2010.
Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
11

---

Under review as a conference paper at ICLR 2021
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Ashish Mishra, Shiva Krishna Reddy, Anurag Mittal, and Hema A Murthy. generative model for zero shot learning using conditional variational autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2188-21989 2018.
Ryan I. Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs. International Conference Learning Representations (ICIRR) 2019.
Jinseok Nam, Eneldo Loza Mencía, and Johannes Fürnkranz. All-in text: Learning document, label, and word representations jointly. In AAAI Conference Artificial Intelligence (AAAI). 2016.
Rasha Obeidat, Xiaoli Fern, Hamed Shahbazi, and Prasad Tadepalli. Description-based zero-shot fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Nikolaos Pappas and James Henderson. Gile: A generalized input-label embedding for text classifi- cation. Transactions of the Association for Computational Linguistics, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. Afet: Automatic fine-graine entity typing by hicrarchical partial-label embedding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1369-1377 2016.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In International Conference Machine Learning (ICML), 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, 2018
Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou, End-to-end structure- aware convolutional networks for knowledge base completion. In AAAI Conference Artificial Intelligence (AAAI), 2019.
Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. Neural architectures for fine-grained entity type classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers (EACI.), 2017.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems (NeurIPS). 2013.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI Conference on Artificial Intelligence (AAAI), 2017.
12

---

Under review as a conference paper at ICLR 2021
Niket Tandon, Gerard De Melo, and Gerhard Weikum. Webchild 2.0:
Fine-grainet commonsense knowledge distillation. In Proceedings ACL 2017, System Demonstrations, 2017.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition- based multi- relational graph convolutional networks. In International Conference on Learning Representations (ICLR), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations (ICLR), 2018.
Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. A meta-learning framework for generalized zero-shot learning. AAAI Conference on Artificial Intelligence (AAAI), 2020.
Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions Intelligent Systems and Technology (TIST), 2019.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Ralph Weischedel and Ada Brunstein. Bbn pronoun coreference and entity type corpus. Linguistic Data Consortium, Philadelphia, 112, 2005.
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. International Conference Machine Learning (ICML), 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. Zero-shot user intent detection via capsule neural networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata, Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligncce (PAMI), 2018a.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5542-5551, 2018b.
Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, and Ling Shao. Attentive region embedding network for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9384-9393, 2019.
Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Imposing label-relaiional inductive bias for extremely fine- aoe-grained entity typing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers) (NAACL). 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations (ICLR), 2019.
13

---

Under review as a conference paper at ICLR 2021
Wenjia Xu, Yongqin Xian, Jiuniu Wang, B. Schiele, and Zeynep Akata. Attribute prototype network for zero-shot learning. ArXiv, bbbs/2008.082900, 2020.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In AAAI Conference on Artificial Intelligence (AAAT). 2019.
Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2019.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
Dani Yogatama, Daniel Gillick, and Nevena Lazic. Embedding methods for fine grained entity type classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume Short Papers), 2015.
Y. Yu, Z. Ji, J. Han, and Z. Zhang. Episode-based prototype generating network for zero-shot learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10032-14041, 2020. doi: 1110999CVR44660020000000011
Zheng Yuan and Doug Downey. Otyper: A neural architecture for open named entity typing. In AAAI Conference on Artificial Intelligence (AAAI), 2018
Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo Kim. Graph transformer networks. In Advances Neural Information Processing Systems, pp. 1 1960-11970, 2019.
Chenrui Zhang, Xiaoqing Lyu, and Zhi Tang. Tgg: Transferable graph generation for zero-shot and few-shot learning. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 1641-1649, 2019a.
Hongguang Zhang and Piotr Koniusz. Zero-shot kernel learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7670-7679, 2018
Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth. Transomes: From linguistic graphs to commonsense knowledge. Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), 2020.
Jingqing Zhang, Piyawat Lertvittayakumjon, and Yike Guo. Integrating semantic knowledge to tackle zero-shot text classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), 2019b.
Li Zhang, Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2021- 2030, 2017.
Ben Zhou, Daniel Khashabi, Chen- Tsai, and Dan Roth. Zero-shot open entity typing as type- compatible grounding. In EMNLP, 2018.
Yizhe Zhu, Jianwen Xie, Zhiqiang Tang, Xi Peng, and Ahmed Elgammal. Semantic-guided multi- attention localization for zero-shot learning. In Advances in Neural Information Processing Systems, pp. 14943-14953, 2019.
14

---

Under review as a conference paper at ICLR 2021
500
400
Number of Predictions
300
200
100
0
1
2
4
7
9
Prediction Consistency
Figure 2: Graph showing distribution of inconsistencies in the LSTM-basst aggreagtor predictions.
A LSTM PREDICTIONS
LSTMs have been used in graph neural networks aggregators to generate more expressive node embeddings. However, LSTMs assume an ordering of the inputs which not present in a graph neighbourhood. To apply LSTMs to unordered set of nodes, Hamilton et al. (2017a) randomly permute the nodes in the neighbourhood.
We test whether randomly permuting the node neighbours makes LSTM-based aggregator permutation invariant. We replicate the LSTM- baseed aggregator to work with the ZSL-KG framework. The model is trained with the setup described in Section 4.1 We run the prediction on the Animals with Attributes 2 dataset by computing 10 class representation for each the classes using the trained LSSTM-based aggregator model.
The experiments reveal that 1325 out 7913 (16.78%) have multiple predictions for the same image. For images that have multiple predictions, take the count of the mode prediction and plot the histogram. Figure shows inconsistency in predictions. The graph for a given value on the x-axis read as for every 10 prediction, p times the same output is predicted.
B DATASET DETAILS
Here, provide details about the datasets used our experiments. Table O shows the statistics about the datasets.
We obtain the OntoNotes and BBN dataset Ren 2016). OntoNotes has three levels of types such as /location, / /tt.aation//srrutteeuuu / /locat on structure//gottt /nmeettenn are treated as coaise-grained entity types and is treated as fine fine-grained entity type. Similarly. BBN has two levels of types and we consider the level two types as fine-grained types. Furthermore, we process the datasets by removing all the fine-grained labels from the train set. We also remove all the from the train set where coarse-grand grained entity types are not present in the test set. Another key point to remember is that /other type present in the OntoNotes dataset cannot be mapped to meaningful concept or a wikipedia article. To overcome this limitation, we learn an embedding for / /other type during training.
where / /location and http://stuuttuteteveeeeoooee
15

---

Under review as a conference paper at ICLR 2021
Dataset
Seen classes
Unseen classes
Train examples
Test examples
AWA2
40
10
7913
aPY
20
12
7924
SNIPS-NLU
5
9888 220398 85545
3914
OntoNotes
40
32
9604
BBN
15
24
12349
Table 6: Zero-shot datasets used in our experiments
For object classification datasets, AWA2 and aPY, do not require the training examples because we use pretrained weights from ResNet101 to learn the class representations. We crop objects from APY test dataset as multiple objects present in the same image. To crop the objects, we use bounding box provided Farhadi et al. (2009), add 15px padding on all sides and crop them.
C EXPERIMENTAL SETUP
C.I GRAPH NEURAL NETWORK SETUP
GCNZ (Wang et al. 2018) uses symmetrically normalized graph Laplacian generate the class representations. SGCN N et al. 2019) uses an asymmetrical normalized graph Laplacian learn the class representations. Finally, DGP (Kampffmeyer al. (019) exploits the hierarchical graph structure and avoids dilution of knowledge from intermediate nodes. They use dense graph connectivity scheme with a two-stage propagation from ancestors and descendants to learn the class representations.
C.2 CONCEPTNET SETUP
We further preprocess the queried graph from ConceptNet for all the datasets. We remove all the non-English concepts and their edges from the graph and make all the edges bidirectional. For fine-grained entity typing and object classification, neighbourhood that share the same prefix. For example, we take the union of the /c/en/// her//eeephen lephant and c eeceee/ellae hwtpe/eepanttt/eeeeee /n. Then, we compute the embeddings for the concept using the pretrained GloVe 840B (Pennington et al., 2014). We average the individual word in the concept to get the embedding. These embeddings serve as initial features for the graph neural network.
also take the union of the concepts'
For the random walk, the number of steps is 20, and the number of restarts is 10. We add one smoothing to the visit counts and normalize the counts for the neighboring nodes.
C.3 FINE-GRAINED ENTITY TYPING SETUP
We reconstructed OTyper (Yuan & Downey 2018) and DZET (Obeidat et al. 2019) for fine- ine-grained entity typing. Otyper averages the GloVe embeddings for the words in the name of each class to represent it. For DZET, we manually mapped the classes to Wikipedia articles. We pass each article's first paragraph through learnable biLSTM with attention obtain the class representations (Appendix
D PSEUDOCODE
Our framework can be trained in two ways joint training (2) class encoder training. Joint training means that our example encoder and class encoder is trained jointly with the task. We use joint training for intent classification and fine-grained entity typing. Class encoder training means that pretrained example encoder such as ResNet101 and train only the class encoder. In object classification, we train only the class encoder and use pretrained R01Net101. In Algorithm [ we describe the forward pass with the 7GL-KG framework.
16

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Forward pass with the ZSL.-KG framework
Input :example z, example encoder 0(x), linear layer W, class encoder oly), graph
E,, R), class nodes {vy y 2 "g node initialization H [h ..... h" depth L,
graph neural network weights (W), ..... Wh,, transformers {T) ...., Th
neighbourhood sample sizes {81 1 8r.),
Output: :logits for classes {y), 1/,
Un
TrGCN (V, 2= 0):
i L then
return Hy 4- H(V;
else
< 0: for v E V do
N (1,
Hy + TrGCN (N, 1+ 1);
Zn + [(l+1)(Hs)
dp (t+1) + mean ZNe):
ho ((+1) t -o(W((+)) a.):
H!(+1) 11( [|lnht+)]
i+i+1
end
return (++1)
end
((v) + TrGCN (1)), uy :;)); return (xx) W (ty); Woll);
E OBJECT CLASSIFICATION RESULTS ON AWA2 AND APY
Table 7 shows results comparing ZSL-KG with other related work from zero-shot object classification.
F BILSTM WITH ATTENTION
The biLSTM with attention is used the example encoder in intent classification and class representation for DZET (Obeidat et al. 2019). BiLSTM with attention has two components: (1) biLSTM model (2) attention model. The imput tokens
Wy, W1, ..., W,, represented by GloVe passed through the biLSTM to get the hidden states h and h. The hidden states are concatenated to get h hg: h1,..., ..., hn, They passed through the attention module to get the scalar attention values and normalize them:
0i W.. (Lanh( (W, h2))
exp (ax)
a;
(9)
exp (as)
The scalar values are multiplied with their respective hidden vectors to get the final representation ty:
11
bng a,
(10)
1=0
G ATTENTIVENER
We describe AttentiveNER (Shiaooka et al. 2017) that we use to represent the example in the fine-grate fine-grained entity typing task. Each mention comprises of n tokens mapped to pretrained word embedding from GloVe. We average the embeddings to obtain a single vector u..:
Um
m
(11)
n
17

---

Under review as a conference paper at ICLR 2021
AWA2
aPY
Accuracy
Accuracy
LATEM
Xian et al. 2016)
55.8
35.2
ESZSL *
Romera Paredes
Tor
2015
58.6
38.3
SYNC* Changpinyo et al.
2016
46.6
23,9
DEM
Zhang et al. 2017)
67.1
35.0
ZSKL* Zhang & Koniusz
2018
70.5
45.3
SP-AEN (Chen al., 2018 8b
59.7
37.0
CVAE-ZSL* (Mishra et al.
2018
71.4
F-CLSWGAN* x (Xian et al.
2018b
68.2
LisGAN (Li al. 2019)
55.8
36.8
ZSML* Verma cf al. 2020
77.5
64.0
AREN Xie al. 201
67.9
39.2
LGGAA (Liu et al. 201
68.0
41.3
SGMAL (Zhu al.
68.8
TGG (Zhang et al. 2019a
77.2
63.5
APNet (Liu
2020
68.0
E-PGN Yu et al. 2020
73.4
APN (Xu al.
2020)
73.8
GCNZ Wang et (2018)
77.00 = 2.05
56.02 + 0.63
SGCN KKampimmyer al. 2019
77.76 = 1.60
54.98 = 0.66
DGP
Kamptimeyer et al. 2019)
76.26 1.69
52.22 + 0.49
ZSL-KG (Ours)
78.08 = 0.84
64.36 + 0.59
Table Results for object classification on the AWA2 and aPY dataset. Resutlts for methods with * (asterisk) obtained from Verma et al. (2020). E-PGN from (Yu et al., (2020), APNet from Liu et al.
(2020), AREN from (Xie et 2019), ) LGGAA from (Liu et al. 2019b), SGMAL from (Zhu et al.) 2019). TGG from (Zhang et al. 2019a). and APN from (Xu et al. 2020). We report the average
class-balanced accuracy of the for GCNZ, SGCN, DGP, and ZSL- KG on 5 random seeds and the standard error.
where my E Fdd is the pretrained word embeddings from GloVe.
We learn the context of the mention using two biLSTM with attention modules. The left context / is represented by 12, 11.,,,,., 1,) and the right context r by (rı, T2, .2.....,) where I E IRd and rj E R the word embeddings for the left and the right context. is the window size for the context. We > pass and r through their separate biLSTM layers to get the hidden states hj. h for the left context and hr,, hr for the right context.
The hidden states are passed through the attention layer to compute the attention scores. The attention layer is layer feedforward neural network and computes the normalized attention for each of the hidden states Ue E R
a Wa W.(tann( W.
(12)
h
exp (at)
df The scalar values are multiplied with their respective hidden states to get the final context vector representation ve:
(13)
Σ exp (a))
E, exp (a))
v.
h
(14)
1=0
Finally, concatenate the context vector v. and v.n. to get the example representation
18

---

Under review as a conference paper at ICLR 2021
H FINE-GRAINE ENTITY TYPING EVALUATION
Fine-grained entity typing is a multi-label classification task. We follow the standard evaluation metric introduced in Ling & Weld (2012): Strict Accuracy, Loose Micro Fl and Loose Macro Fl.
We denote set of ground truth types as and the set predicted types as P. We use the Fl computed from the precision p and recall for the evaluation metrics mentioned below:
Strict Accuracy. The prediction is considered correct if and only te = fe:
p EccpTT P-T 1(te = te)
(15)
PP|
Ecepmy PnT
1(4e = fe)
(16)
[T]
Loose Micro. The precision and recall scores are computed as:
p Leep Itfe n fel
(17)
EeEp lifl
Ever 11e Vel
p Σeer ICel
(18)
Loose Macro. The precision and recall scores are computed as:
Ite n fel
p =
(19)
P
lfel
PEP
Iee O fe
(20)
fel
EET
I
GRAPH NEURAL NETWORKS ARCHITECTURE DETAILS
Method
Aggregate
Combine
ZSL-KG-GCN
(1) = Mean (hil"),uEAn
h = (W(Oa!")
ZSL-KG-GAT
(1) Autn (
h a EENittt (1) Niu 111-11)
ZSL-KG-RGCN
a 11 ErER EjeN(yY I Exen aar v. (
h = ola, + w!!'! A1 ((1-11)
ZSL-KG-LSTM
a = LSSMM (A (n NN())
h a W 11!! (a(""))a!"")
Table Graph Aggregators
ZSL-KGGGCN uses mean aggregator to learn the neighbourhood structure. ZSL-KG-GAT projects the neighbourhood nodes to a new features h (I-1) Wh((-1) The neighbourhood node features concatenated with self feature and passed through self-attention module for get the attention coefficients. The attention coefficients are multiplied with the neighbourhood features to the get the node embedding for the 1-th layer in the combine function. ZSL-KG-RGCN uses relational aggregator to learn the structure of the neighbourhood. To avoid overparameterizatton from the relational weights, we perform basis decomposition of the weight vector into B bases. We learn [B] relational coefficients and [B] weight vectors in the aggregate function and add with the self feature in combine function. ZSL-KG-LSTMM uses LSTM as an aggregator to combine the neighbourhood features. The nodes in the graph are passed through an LSTM and the last hidden state is taken the aggregated vector. The aggregated vector is concatenated with the node's previous layer feature and passed to the combine function get the node representation.
19

---

Under review as a conference paper at ICLR 2021
J HYPERPARAMETERS
In this section, we detail the hyperparameters used our experiments.
J.1 TRAINING DETAILS
Our framework is built using py Torch and AllenNLP (Gardner al., 2017). In all our experiments. we use Adam (Kingma & Ba 2015) to train our parameters with a learning rate of 0.001. For intent classification, we experiment with a weight decay of Ie-05 and Se-05. We found that weight decay of 5e-05 gives the best performance overall in intent classification for all the baseline graph aggregators. In intent classification, ZSL-KG use weight decay of le-05. We add a weight decay of 5e-05 for the OntoNotes experiments. Finally, all experiments in zero-shot object classification have weight decay of 5e-04.
Task
Inp. dim.
Hidden dim.
Attn. dim.
Low-rank dim.
Intent classification gine-grained entity typing
300
32
20
16
300
100
100
20
Table Hyperparameters for the biLSTM with attention example encoder in the language related tasks
For intent classification and finegrained entity typing, we assume a low-rank for the compatibility matrix W. The matrix W Rd2xd is factorized into A E Rixd and B Rd where h is the low-rank dimension. Table Table summarizes the hyperparameters used in the example encoders which is biL.TMM with attention task-specific variant of it.
In fine-graine ane-grained entity typing, we have two baselines that do not use graph neural networks: OTypper and DZET. OTyper averages the GloVe embedding of 300-dim for the class representations. DZET uses biLSTM with attention for the class encoder with the same hyperparameters as fine-grained entity typing from Table
J.2 GRAPH AGGREGATOR SUMMARY
Task
layer-1
layer-2
Object classification
2048
2049
Intent classification
64
64
Fine-grained entity typing
128
128
Table 10: Output dimensions of the graph neural networks in experiments.
layer 1
layer
d())
d(p)
d(r))
d(p)
AWA2/aPY
100
150
1024
1024
SNIPS-NLU
150
150
32
OntoNotes
150
150
32
64
BBN
250
150
32
64
Table 11: The hyperparameters used in our Transformer Graph Convolutional Network.
Table 10 describes the output dimensions the node embeddings after each graph neural network layer. ZSL-KG-GCN, DGP, GCNZ, and SGN are linear aggregators and learn only one weight matrix in each of the layers. ZSL-KG- GAT learns a weight matrix for the attention where W. E 2d x1 and uses LeakyReLU activation in the attention. LeakyReLU has negative slope of 0.2. ZSL-KG- RGCN learns bases weight vectors in the baseline. We found that = 1 performs the best for fine-grained entity typing and object classification. For intent classification, we use 10 bases, i.e.,
20

---

Under review as a conference paper at ICLR 2021
B 10. In intent classification and fine- tineigrained entity typing, the non-linear activation function after the graph neural network layer is ReLU and in object classification the activation function is LeakyReLU with negative slope of 0.2.
ZSL-KG with the Transformer Graph Convolutional Network is a complex architecture with nu- merous parameters. In transformer module, there are five hyperparameters input dimension d(-1)- output dimension d(-1)- 1)- feedforward layer hidden dimension d(z). and projection dimension d(p)- The input dimension and output dimensions are the same in the aggregator. We tuned the hyperparameters on the held out validation classes for object classification and intent classifica- tion. For tne-grained entity typing we manually tuned the hyperparameters. Table [1 details the hyperparameters used in our experiments for all the datasets.
21


Reviews:
[{"review_id": "fAaVjc-86D2", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposed to leverage a commonsense knowledge graph and an attention GNN based model to aggregate the node features on the graph for the problem of zero-shot learning. \n\nThe main concern is the technical contribution is limited:\n\n- The authors mention \"Existing methods for zero-shot object classification such as GCNZ and DGP cannot be\nadapted to common sense knowledge graphs as they do not scale to large graphs or require a directed\nacyclic graph such as WordNet.\" Can the authors elaborate on why GCNZ and DGP can not scale to large knowledge graphs? To my understanding, GCNZ and DGP use WordNet knowledge graphs which are large-scale. And [a] also proposed to leverage GNN structure and knowledge graph, can the authors explain more comparisons with respect to this paper?\n\n- The proposed GAT model is similar to the previous works, i.e., GCNZ, and DGP. To me, the model GNN and attention (Transformer in the paper) model have already been discussed in these two works and the only difference seems to be the knowledge graph, where the previous paper uses the WordNet while the current submission uses the ConceptNet. (I think WordNet is a more straightforward knowledge graph format, can the authors explain why extra efforts on making a ConceptNet is better?) Also, [e] this paper has already proposed to leverage ConcepNet and GNN structures for ZSL, can the authors explain the difference with it please?\n\n- The authors claim the structure of Transformer while actually, it is the structure of dot product attention. The appliance of the attention of ZSL has been a lot, such as [a,b,c,d]. So I think the technical contribution is limited and it is better to also include the discussions with these works.\n\nBased on the previous three points, I believe the technical contribution is limited for this submission.\n\nThe experiments do not support the claims:\n\n- The authors claim the model can scale to large size while in the experiment section,  the results on ImageNet is not reported. The most related two works GCNZ and DPN both reported their results on ImageNet, thus i believe the performance result on this dataset is necessary. \n\n- I am expected to see performance on the generalized zero-shot learning since this setting is more practical and includes the prediction on both seen and unseen classes. The most recent papers including the previous works listed in the experiment table have the results on GZSL, so do the GCNZ and DPN. Thus, I think the results on GZSL are also necessary to support the claims in this submission.\n\n\nWriting:\nEven the basic format is not well organized, such as the tables are out of columns and spelling errors.\n\nRelated works:\nThe current related works section is a little bit precise, and I would recommend the authors to include the related works mentioned in my previous reviews and discuss the difference and how this submission is different from previous works. Since there is plenty of previous literature that is somewhat related to the current submission, a more detailed explanation of the relationships to previous works is recommended.\n\n[a] Attribute Propagation Network for Graph Zero-shot Learning, AAAI 2021\n\n[b] Attentive Region Embedding Network for ZSL, CVPR19.\n\n[c] Semantic-Guided Multi-Attention Localization for ZSL, NIPS19.\n\n[d] Attribute Attention for Semantic Disambiguation in ZSL, ICCV19.\n\n[e] TGG: Transferable Graph Generation for Zero-shot and Few-shot Learning", "extra_scores": {}, "extra_text": {}}, {"review_id": "XHtjbdrWFor", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper tackles zero-shot learning by leveraging the large-scale knowledge graph i.e., ConceptNet to propagate the knowledge learned from seen classes to unseen classes. The authors propose a novel propagation rule that aggregates node embeddings by the self-attention technique. It is infeasible to run GCN on such large-scale knowledge graph. Therefore they reduce the knolwedge graph by adopting a neighborhood sampling strategy based on random walks. The method is evaluated on multiple zero-shot learning tasks including object classification, intent classification and fine-grained entity typing. The SOTA results are achieved.\n\nPros\n\n-This paper is well motivated. The idea of leveraging a large-scale common sense knowledge graph that contains richer semantic relationship is reasonable and novel. I agree that previous GCN methods for zero-shot learning are limited by the wordnet. \n\n-The self-attention or transformer-based aggregator is novel as well. Applying such non-linear aggregation makes graph capture complex relationships between nodes. Therefore, I believe the technical contribution is significant. \n\n-The evaluation is conducted on multiple zero-shot learning tasks and the results show that the proposed approach generalizes  well to different tasks. \n\n-The paper is written well and easy to follow in general, although some important details and justification of the results are missing (I will points out the issues below).  \n\nCons\n\n-I am particularly not convinced by the object classification results (Table 1). This paper only reports results on two small datasets i.e., AWA2 and aPY, in the zero-shot learning setting, while the GCN baselines i.e., GCNZ, SGCN and DGP, are all evaluated on the large-scale ImageNet in both zero-shot and generalized zero-shot learning settings. I strongly suggest the authors to compare with GCNZ, SGCN and DGP on the ImageNet.\n\n-The results on the BBN dataset (Table 3) are not impressive. The proposed approach performs much worse than DZET \nin two metrics i.e., Loose Mic. and Loose Mac. The authors fail to give some justification for these bad results.  \n\n-The neighborhood sampling technique is rarely evaluated. I believe it is interesting to ablate different sampling strategies e.g., uniformly ramdom sampling, importance sampling (Chen et al., 2018a) and random walks. Implementation details for sampling are not provided either which makes it hard to reproduce the results. \n\n-Writing can be further improved in the following perspectives. A discussion on the difference between the common sense knowledge graph and the standard knowledge like WordNet adopted by previous ZSL works should be included. The justification for the bad results on the BBN dataset is missing. There are not enough implementation details in order to reproduce the results e.g., what are the text features used for the intent classification? How many nodes are selected in the graph sampling step?  \n\nMinor questions\n\n-In the last paragraph of Section 4.1, I do not get why other GCN approaches are referred as general-purpose methods? \n\nJustification of the rating\n\n-Overall, I think this paper has significant technical contributions. However, due to the issues I have pointed out above, my initial score is only 6 (marginally above the acceptance threshold.). If the authors can address my concerns, in particular, providing better results on the ImageNet dataset, I will be very happy to upgrade my score.  \n\n----------------------------------------------------\nPost-rebuttal\n\nThanks for the new ZSL and GZSL results on ImageNet. The results are convincing to me. My other concerns are properly addressed too. I read the concerns from R4 and R5. To my knowledge, using common sense knowledge graph and the GCN with self-attention are novel in the zero-shot learning literature. I decide to increase my score to an accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XC3jVK0gU3Z", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to tackle the zero-shot learning problem by learning class representation from commonsense knowledge graphs. To capture knowledge in graphs, the paper proposes to perform Transformer-based aggregation. \n\nPros:\n- The motivation of using commonsense knowledge graphs to enhance zero-short learning is interesting (although it has been explored in the previous work). \n- The paper is in general clearly written and easy to follow.\n\nCons:\n- In term of methodology, the novelty of paper is very limited---it uses Transformer to aggregate knowledge over graphs. This is pretty straightforward, and the previous work (e.g., (Kampffmeyer et al.)) has tried Graph Attention Networks (GAT)-based models for zero-shot learning. Transformer and GAT are very similar, particularly in the setup of this work (e.g., without sequence/position embedding and the graphs are not fully connected). The differences claimed in this paper between GAT-based and Transformer-based models are rather marginal.\n\n- The empirical comparison to GAT-based models in the experiments is not clear enough. For example, Table 4 shows that the proposed model is better than ZSL-KG-GAT, but detailed analyses were not provided to make this clearer to help understand why that is the case (and hence better understand the contributions of this paper). For example, whether the performance differences are related to the use of layer normalization, the two-layer feed-forward nets in equation (4), or other reasons? Previous work suggested DGP is better than GAT-based models (Kampffmeyer et al.) and in this current submission GAT and transformer-based models are better. More discussions will be helpful. \n\n- It will be more helpful if the paper describes more details about the models in comparison, e.g., details about ZSL-KG-GAT such as the setup of multi-heads.\n\nMore comments:\n- The title of the paper may be made more specific, particularly given much work has been done by using commonsense/knowledge graphs for zero-short learning. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes to leverage a commonsense knowledge graph and an attention GNN based model to aggregate the node features on the graph for the problem of zero-shot learning. It received three reviews two of them recommending rejection, and another review was initially borderline however they moved to acceptance after the rebuttal. The meta reviewer finds that the paper is not yet ready for publication and recommends rejection based on the following observations.

Although the model is interesting, as agreed by the reviewers the initial version of the paper fell short on convincingly evaluating the method, e.g. generalized zero-shot learning (GZSL) setting as pointed out by R5, ImageNet and small scale dataset results as pointed out by R1. Similarly, the main paper (without the annexes) has been found to fall short on providing enough details of the model as pointed out by R1. 

The authors ran additional experiments during the rebuttal phase which showed some promise, however one more review round may be necessary to carefully validate these results. As R1 pointed out, the paper only reports results on two small datasets i.e., AWA2 and aPY, which contain classes similar to ImageNet. It would be interesting to observe the behaviour of the model on more challenging scenarios on other publicly available benchmark datasets of fine-grained nature whose distribution are far from ImageNet. This would indicate the generalisation ability of the model.

Furthermore, as pointed out by R1, moving the details on the implementation and the architecture details from appendix and from python scripts to the main paper may be beneficial. However, this would end up significantly extending the paper. Hence, one more review round may be necessary for this paper.
Decision: Reject