Open Set Domain Adaptation with Zero-shot Learning on Graph
Authors: Xinyue Zhang
Xu Yang
Zhi-yong Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: open set domain adaptation
zero-shot learning
knowledge graph
graph convolutional network
adversarial learning

Abstract: Open set domain adaptation focuses on transferring the information from a richly labeled domain called \emph{source domain} to a scarcely labeled domain called \emph{target domain} while classifying the unseen target samples as one \emph{unknown} class in an unsupervised way. Compared with the close set domain adaptation, where the source domain and the target domain share the same class space, the classification of the unknown class makes it easy to adapt to the realistic environment. Particularly, after the recognition of the unknown samples, the robot can either ask for manually labeling or further develop the classification ability of the unknown classes based on pre-stored knowledge. Inspired by this idea, in this paper we propose a model for open set domain adaptation with zero-shot learning on the unknown classes. We utilize adversarial learning to align the two domains while rejecting the unknown classes. Then the knowledge graph is introduced to generate the classifiers for the unknown classes with the employment of the graph convolution network (GCN). Thus the classification ability of the source domain is transferred to the target domain and the model can distinguish the unknown classes with prior knowledge. We evaluate our model on digits datasets and the result shows superior performance.

Under review as a conference paper at ICLR 2022
OPEN SET DOMAIN ADAPTATION WITH ZERO-SHOT LEARNING ON GRAPH
Anonymous authors Paper under double-blind review
ABSTRACT
Open set domain adaptation focuses on transferring the information from a richly labeled domain called source domain to a scarcely labeled domain called target domain while classifying the unseen target samples as one unknown class in an un- supervised way. Compared with the close set domain adaptation, where the source domain and the target domain share the same class space, the classification of the unknown class makes it easy to adapt to the realistic environment. Particularly, after the recognition of the unknown samples, the robot can either ask for man- ually labeling or further develop the classification abilitty of the unknown classes based on pre-stored knowledge. Inspired by this idea, in this paper we propose model for open set domain adaptation with zero-shot learning the unknown classes. We utilize adversarial learning to align the two domains while rejecting generate the
the unknown classes. Then the knowledge graph is introduced classifiers for the unknown classes with the employment of the graph convolution network (GCN). Thus the classification ability of the source domain is transferred to the target domain and the model can distinguish the unknown classes with prior knowledge. We evaluate our model on digits datasets and the result shows superior performance.
INTRODUCTION
In the last decades, deep learning models have shown good performance in various tasks, especially in visual perception. The training of the deep learning network relies on plenty of labeled data. However, most of the existing large labeled datasets are collected from the Internet. The images in these datasets are normative and unified, which are different from the images relevant for a specific application. Besides, depending on the application, the images may be obtained by different typed of visual sensors or with different perspective of sensors. It costs lot to retrain the classification model in different situations. In some typical applications, the samples in the real world are hard to gather too large to label. Thus it is important to deal with the gap among domains. They should be able to utilize the wellelabeled samples in the source domain to classify the samples in the unlabeled target domain which is related domain adaptation. There are already some researches on domain adaptation, such as [Ganin & Lempitsky (2015). Long et al.] (205)), Long et al. (2016), and Wang & Deng (2018). The alignment of the domain gap makes the robot adapt well dynamic and unstructured environments.
Except for the domain gap among different datasets, the variation of the classes also makes it hard for the model to adapt to a new dataset. Depending on the application and the scale of different datasets, the model may come across classes that are not contained in the source domains. With the traditional domain adaptation methods, the unknown classes are mistakenly aligned due to the absence of training samples of unknown classes in the source domain. The imbalance of the types of classes brings over-fitting problems and is not suitable for classification in the open world. Thus is important for the robot to reject the unknown classes and only align the shared classes. This problem is known as open set domain adaptation, which is first proposed by Panareda Busto & Gall (017)) and followed by for instance Saito et al. (2018). Busto et al. (2018), and Liu et al. (2019). In the setting of the open set domain adaptation, the target domain contains both the classes of the source domain and the additional new classes. The model not only aligns the target domain to the source domain but also rejects the unknown classes.

---

Under review as a conference paper at ICLR 2022
open domain adaptation with zero-shot learning
close set domain adaptation
open set domain adaptation
source
target
unknown
Figure 1: An overview the proposed domain adaptation with zero-shot learning. Close set domain adaptation align the target domain to the source domain. Open set domain adaptation not only align the domain gap, but also reject the unknown classes as one class. Open set domain adaptation with zero-shot learning further gives detailed classification on the unknown classes, which is more complex and valuable.
is worth noting that previous open set domain adaptation methods typically classify all the addi- tional new classes into one unknown class. However, the unknown class may contain classes that are worth learning. It may be more valuable to detect the unknown classes in detail and develop the ability to classify them with the former information. With the process of distinction and transfer- ring, the model can expand its visual recognition ability with little labeled information. Since the unknown classes are not included in the source domain, the model lacks the labeled information for the new classes. Current open set domain adaptation methods can not give detailed classification on the unknown part with no labeled images. This problem is related to zero-shot learning. In the zero-shot learning problem, complementary information is collected to transfer the knowledge from the base classes classify the unknown ones. Inspired by this, with the knowledge stored in the knowledge graph, the classifiers of the unknown classes can be obtained in the target domain with no labeled samples.
Towards this end, we propose generic model to align the gap between the labeled source domain and the unlabeled target domain while classifying the unknown classes in the target domain. The contributions of this paper mainly lies in tackling the following two difficulties.
First, since the unknown classes are not contained in the source domain, we have no labeled samples for supervised training. The lack of labeled data may cause overfitting problem of the model, which means the model only classify the samples as the known classes and can not classify the unknown ones. is necessary to utilize complementary information support the inference. Thus employ the knowledge graph to stores some prior knowledge of the known classes and the unknown classes, which contains the structural relations between different classes, beyond the individual attribute representation of each class. The structural information offers bridge for the inference from the known classes the unknown ones. With the employment of the graph convolution network, the information propagates among the graph and the unknown classes gather the information from their neighbor to generate their classifiers. These inference classifiers work the initial classifiers of the classification model.
The second difficulty is how to adapt the inference classifiers the target domain. Since we only have labeled samples in the source domain, the inference classifiers are suitable to the source do- main. not able classify the unknown samples in the target domain because of the domain gap. Thus we introduce adversarial learning to align the domain gap. The classification model con- sists of two modules, the feature generator, and the classifier. Since the generator works to extracts the features of the samples and the classifier works to output the class probability, we train them simultaneously in an adversarial way. The classifier is trained to found a boundary for the unknown classes while the generator is trained make the samples far from the boundary. With adversarial learning, the generator can deceive the classifier to generate aligned features both domains and re- ject the unknown classes according the unknown boundary. Thus the feature of the shared classes
2

---

Under review as a conference paper at ICLR 2022
is aligned in both domains and the unknown classes are rejected class. With the adaptation in both domain gap and class gap, our model is able to classify objects in the dynamic and complex open world.
generator
classifier
Source domain
Output
G
C
adversarial
N
Target domain
Knowledge graph
Figure 2: An overview of our model for domain adaptation with zero-shot learning. We utilize knowledge graph to infer the classifier of the unknown classes. With the employment of the graph convolutional network, the model generates the initial classifier for all classes and applies it to the classification part. The generator and the classifier are trained in an adversarial way align the domain gap.
We utilize the knowledge graph and the adversarial learning in jointly trained framework. The two parts work together to align the shared classes in two spaces while generating classifiers for the unknown classes the target domain. We further evaluate our method on digits datasets and demonstrate its effectiveness.
2 RELATED WORKS
2.1 OPEN SET DOMAIN ADAPTATION
Open set domain adaptation goes beyond traditional close set domain adaptation. considers a more realistic classification task, in which the target domain contains unknown samples that not present the source domain. Open set domain adaptation is first proposed by Busto et al.) (2018). They measure the distance between the target sample and the center of the source class to decide whether target sample belongs to one of the source classes or the unknown class. However, they require the source domain to have unknown samples as well. Later on, Saito et al., (2018) propose open set back- nnck-propagation (OSBP) for source domain with no unknown samples. They utilize adversarial learning to train the feature generator and classifier. boundary for the unknown classes, the feature generator tries to deceive it. However, both of them only separate the unknown classes the target domain, can not give detailed classification on the unknown ones. The learnable information in the unknown space deserves deep exploitation. We have found few papers that consider the fine- aene-grained classification of the unknown classes in open set domain adaptation, aim to fill in the blanks.
the classifier tries set
2.2 ZERO-SHOT LEARNING
Zero-shot learning aims at generating classifiers for unknown classes with no labeled samples. Sev- eral pieces of research have been done on this area, such as Kipf Welling (2016) Xian et al. 2017). Due to the limitation of the available samples, some researchers extract complementary information from the related known classes to support the inference of the unknown ones. Among these methods, building the relationship between classes in form of graph seems more reasonable. The special geometry of graphs well shows the complicated relationship and the unknown classes gather adequate information from the known ones. Current zero-shot learning please refer to
3

---

Under review as a conference paper at ICLR 2022
knowledge graphs for inference. [Wang et al. (2018) built unweighted knowledge graph com- bined with word embedding upon the graph convolutional network. With information propagation, novel nodes generate predictive classifiers with common sense. [Kamffimeyer al. (2019) improve upon this model and propose a dense graph propagation to prevent dilution of knowledge from dis- tant nodes. Knowledge graphs intuitively present the stored information for the visual cognitive development.
2.3 GRAPH CONVOLUTIONAL NETWORK
Graph convolutional network (GCN) a kind of graph neural network proposed by Kipt & Welling (2016). GCN enables the nodes in the graph to share the intensity of statistics, which improves the efficiency of sampling. GCN can also be employed in non-euclidean space. From the perspective of the spatial domain, GCN iteratively aggregates neighborhood information. The propagation of information on the graph further exploits the structural information on the graph. GCN is first introduced by Bruna et al. (2013). Then GCN is extended with filtering method based the recurrent Chebyshev polynomial. The improvement reduces the computational complexity a lot, which is equivalent common CNNs in image operating. [Kipf & Welling (2016) further simplify the framework to improve the scalability and robustness. They employed their model on the semi- supervised learning on the graphs. Our model employs the framework of GCN to propagate the complementary information among nodes for the inference of classifiers.
3 APPROACH
3.1 PROBLEM DEFINITION
In open set domain adaptation with zero-shot learning, we have source domain D2 [(x], H= which contains n, labeled samples, and a target domain D₂ و- which contains ny unlabeled samples. The class space in the source domain C, which we call known classes. C, also is shared by the class space of the target domain C.. Is worth noting that C, further contains additional unknown classes Cu, that is C. C, V C.. The source domain is sampled from the distribution g.. while the target domain is sampled from the distribution qr- In close set domain adaptation, q. * qt. In open set domain adaptation with zero-shot learning, we also define qa # q²- q refers to the distribution of the known classes in the target domain. Note that the samples in the target domain are all unlabeled and the samples in the source domain are all labeled.
3.2 CLASSIFIER INFERENCE MODULE
With few labeled samples, the human can make good inferences on unfamiliar things with the related information that they obtain from books. Our model also extracts the task-based knowl- edge from a prestored knowledge graph. The inference graph is denoted as G (V, E), where V = (V1, U2, ..., Un,, .... Un, is node-set of all classes C.. n, refers to the number of known classes. ne - n, refers to the number of unknown classes. The nodes in the prestored knowledge contain the attributes of different classes, E = {Ei (1, v;)} } is an edge set referring to the re- lationship among the graph. The edges in the prestored knowledge graph are based on the similarity of the attributes between different classes.
Since we only have labeled data of the source domain. We first train the recognition model on the source domain D.. Specifically, the pre-trained recognition model is denoted C(F(-10))w) The model consists of two parts, feature extractor (..(0) and class classifier C(|ww). and w indicate the parameters of the model trained with D. { v1), ),... (I ya)}. ). The symbol æ; refers the source images of the ith class while y; refers to their label. Feature extractor I Fxx(0) takes an image as input and figures the feature vector of it as zi. The final classification score is computed
[$1, 12, ... sM] [ W1, 2T W2,,.,, 2T w..)
Thus the inference of the classifiers on unknown classes is to inference the classification weights w, on the unknown classes with the inference graph.
With the framework of the graph convolutional network, Our model propagates information among nodes by exploring the class relationship. For one layer in GCN, node aggregates information
4

---

Under review as a conference paper at ICLR 2022
from the neighbors connected it. GCN can also be extended multiple layers to perform deeper spread. Therefore, the unknown classes can utilize the information from the related known classes and predict the classification weights of their own. The mechanism of GCN is described as
H ReLu(D
(2)
where H denotes the output of the 1th layer, while for the first layer H = V. uses Leaky ReLu as the nonlinear activation function. To reserve the self-information of the nodes, self-loops are added among the propagation, E E + I, where E E RNNN is the symmetric adjacency matrix and I E R X represents identity matrix. Dú = Σj Ei normalizes rows in E to prevent the scale of input modified by E. The matrix U is the weight matrix of the 1 layer, which GCN regulates constantly to achieve better performance.
Our model conducts two layers of GCN the inference graph. Unknown classes learn the mecha- nism of end-to-end learning from known classes through propagation. The inference graph is trained to minimize the loss between the predicted classification weights and the ground-truth weights. The ground-truth weights refer to the classifiers of the known classes, which are extracted from the pre- trained model on the source domain.
M E(w M
LGON
where in refers to the output of known classes on GCN. to train denotes the ground truth classifiers of the known classes obtained from the pre-trained model. With the supervision of the known classes, the unknown nodes in the inference graph can also generate classifier weights of their own. Finally, with the employment of GCN, the classifier inference module not only generates predictive classifiers of the unknown classes in the source domain but also provides more general classifiers of the known ones.
3.3 DOMAIN ADAPTATION MODULE
The classifier inference module generates classifiers for the unknown classes. However, these clas- sifiers are suitable to the source domain, since the ground-truth classifiers are extracted from the model trained the labeled samples. Thus the domain adaptation module attempts align the domain gap between the source domain and target domain.
We apply the inference classifiers the pre-trained classification model. The number of the clas- sifiers of the pre-trained classification model changes from n, to 74, As mentioned above, the classification model consists of two parts, the feature generator, and the classifier. To align the do- main gap, we employ adversarial learning on the classifier and the feature generator. The classifier aims at setting boundary for the unknown classes in the target domain. With the boundary, the unknown classes can be picked out. The boundary refers to the proportion of the unknown classes in the target domain.
Pun = p(y = Yun xT.) =
(4)
Yun
Σ ply= = vx(xz)
(5)
1=
where Yun refers to the proportion of unknown classes, t refers to the boundary. The feature gener- ator tries to generate features that deceive the classifier. That is, the objective of the generator is to maximize the error of the classifier. To increase the error, the generator tries to generate features far from the boundary. Besides, the classification ability on the known classes should be reserved, thus we also consider the classification accuracy on the source domain during the training process. We use standard cross-entropy loss for this purpose.
L. yo) lo(pp(yy---- ply velx.) (C(F(x)))).
= velz.))
(6)
(7)
With the cross-entropy loss, the model ensure the classification accuracy on known classes. For the boundary of the unknown classes, follow the settings in the OSBP and utilize binary cross
5

---

Under review as a conference paper at ICLR 2022
entropy loss.
Lxdv( = +log(Pum)
- lt)l - Pun))
(8)
74
Pun
ply yilwz)) i=
Σ
The overall objective of our model is,
min Ls (Fis Ma) Loadu Ladd + LOON
(10)
min T., (1 Ue) Lodu (244 + LGGN
(11)
With the domain adaptation module, the unknown classes in the target domain are separated and the features of both domains aligned. We also suggest iterating the classifier inference module and the domain adaptation module for better performance.
4 EXPERIMENT
In this section, we perform experiments to evaluate the effectiveness of our model. Since there is no related model the open set domain adaptation problem with zero-shot learning, demonstrate our model with current zero-shot learning models and make a detailed analysis of the results.
4.1 DATASETS
We test our model on three digits datasets. Compared to the traditional dataset on domain adaptation. The digits datasets contain a fewer number of classes, which means the number of the known classes fewer. Thus the information that the unknown classes can utilize is fewer. The task turns more difficult zero-shot learning problem. The three digits datasets are MNIST, USPS, and SVHN. MNIST proposed by [LeCun et al. (1998), which contains about seventy thousand images of white digits black background. USPS is proposed by Friedman et al. (2001). which contains seven thousand grayscale images on the handwritten digits. SVHN is proposed by Netzer et al. (011), which contains six hundred thousand images of real-world street view house numbers. We consider each dataset as a different domain. For the class space, we have two settings of unknown classes. In the 3-way setting, the source domain contains seven classes (0-6), while the target domain contains ten classes (0-9). While in the 2-way setting, the source domain contains seven classes (0- 7), while the target domain contains ten classes (0-9). In both settings, our goal is to align the known classes in the target domain to the source domain and have the ability to classify the unknown ones.
4.2 SETTINGS
employ GCN of two layers for the inference of the classifier. The ground truth of the classifiers on the known classes is extracted from the original recognition model trained the source domain. The original model has almost the same structure as the final classification model, except for the number of the classifier. The classification model is trained in 1200 epochs. We use the Adam Da 2014) optimizer for training with the weight decay of 0.0005 and the learning rate 0.001. The boundary of unknown classes is set to 0.5 for better performance. The whole project is under the framework of Py Torch Paszke et al. 2017)
4.3 COMPARISON
To test the performance of our model, we conduct experiments under several settings. However, since there are few models that work the domain adaptation with zero-shot learning, compare our model to zero-shot learning methods. Besides verifying the value of the inference classifiers, we also compare model to open set domain adaptation methods with random initialization unknown classifiers. The results are shown in the following table.
The comparison between model and other exiting methods is reported in table The task names s2m, u2m, and m2u refer to the transfer tasks from SVHN to MNIST, USPS to MNIST, and MNIST to USPS. The 2-way and 3-way settings mean the number of unknown classes is 2 and 3. The class

---

Under review as a conference paper at ICLR 2022
4
2
3
4
3
>>
8
9
MNIST
USPS
Figure 3:
In the task setting m2u, the model is trained on the MNIST dataset with seven eight classes and transferred to the USPS dataset with ten classes.
type setting all and unknown refers the classification accuracy the overall 10 classes and the unknown classes. The performance is evaluated by the average top-1 accuracy.
z-GCN proposed by Wang al. (2018) zero-shot learning model with the employment of graph knowledge. It only considers the classes gap while ignores the domain gap. However, as figure shows, the domain gap between the MNIST and the USPS large. Overfitting on the source domain and the lack of labeled training images in the target domain affect the results a lot. In the s2m tasks with the 2-way setting, the classification accuracy of z-GCN on all classes forty-eight, which twenty percent lower than our model. In the u2m and m2u tasks, the improvement of our model about ten percent as well. The result demonstrates that the adversarial learning employed by our model is able to transfer the classification ability from the source domain to the target domain with no labeled data. The domain adaptation is important for the flexibility the models
Table 1: Comparison results
task
s2m
setting
2-way
3-way
classes
all
unknown
all
unknown
7 7-GCN
48.4%
6.2%
39.5%
13.2%
OSBP
58.2%
17.4%
54.0 %
24.3%
our 67.0%
38.2%
64.3%
46.4%
task
u2m
setting
2-way
3-way
classes
all 60.5%
unknown
all
unknown
z- z-GCN
9.5%
54.1 1
10.4%
OSBP
61.4%
8.5%
62.3 %
12.4%
our 67.4%
26.5%
69.2%
24.1%
task
m2u
setting
2-way
3-way
classes
all
unknown
all
unknown
z- z-GCN
63.4%
8.6%
62.0%
12.3%
OSBP
51.3%
7.6%
42.2%
10.3%
our
73.6%
49.3%
68.2%
23.5%
Although z- Z-GCN aims at the zero-shot problem, is worth noticing that the accuracy of the unknown classes the target domain adaptation around ten percent. The classification accuracy low on the zero-shot learning problem. That is because the z-GCN only considers the gap in the class space, but does not take the domain gap into consideration. With the training process, z-GCN only infers the classifier for the source domain. However, the testing is performed This result shows the importance of domain adaptation in the zero-shot learning problem. With the aligning of the domain gap. the accuracy of the unknown classes is increased by about twenty percent in our model. In the 52m tasks, the improvement is about thirty percent on the unknown classes. The high classification accuracy the unknown classes also confirms that the domain
the target domain.
7

---

Under review as a conference paper at ICLR 2022
adaptation not only aligns the known classes in both domains but also extracted the features of the images, which is domain invariant.
To test the effectiveness of the inference classifiers, further conduct experiments the open set domain adaptation method OSBP proposed by |Saito et al. (2018). OSBP transfers the classification ability from the source domain to the target domain and rejects all of the unknown classes one class. Since we aim at classifying the unknown classes in detail, we expand the OSBP with randomly initialized classifiers on the unknown classes is about twenty percent lower than our method. However, we notice that the OSBP still shows about ten percent accuracy on the unknown classes and sometimes even higher than z-GCN We owe the classification accuracy on the unknown classes to the rejection mechanism. Since OSBP has the ability to reject the unknown classes as one class, the detailed classification in the one class is much easier. The randomly initialized classifier still has the possibility to classify the two or three unknown classes properly.
the unknown classes. From the results in table the classification ability
The accuracy unknown classes for our method is about twenty percent higher than OSBP. Besides, the accuracy on all classes still has about ten percent increase. The result shows that the inference from the word embedding to the unknown classifier makes sense. Besides, the inaccurate classifiers on the unknown classes confuse the classifier on the known classes and result in a decrease in the accuracy. Thus the outperformance of our model is not simply due to the increased number of the classifiers. The inference of the classifiers based on the knowledge graph is valuable and credible. To avoid randomness, we perform three different domain adaptation tasks. We also test the influence of the number of the unknown classes, which is also shown in the above table. From the result showing above, can come to the conclusion that our model shows a good performance on open set domain adaptation with zero-shot learning.
CONCLUSION
In this paper, we propose a model on open set domain adaptation with zero-shot learning. Our model not only makes good performance on the alignment of the domain gap but also give detailed classification on the unknown classes. The ability of the further classification on the unknown classes improves the visual cognitive development ability of the robot, which is important for the robot working in a realistic environment. The experiments show that our model has a good performance on domain adaptation with zero-shot learning. In future work, will test our model on the real scenario dataset, which is more suitable to the task of the visual cognitive development robot.
REFERENCES
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks graphs. Computer Science, 2013.
Pau Panareda Busto, Ahsan lqbal, and Juergen Gall. Open set domain adaptation for image and action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2))413- 429, 2018.
Kingma Da. A method for stochastic optimization. Computer Science, 2014.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning. Journal of the Royal Statistical Society, 1910):122-12, 2001.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of International Conference on Machine Learning, pp. 1180-1189, 2015.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing- Rethinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 11487-11496 14496, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In Proceedings of International Conference on Learning Representations, 2016.
8

---

Under review as a conference paper at ICLR 2022
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-25 -3224, 1998.
Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate to adapt:
Open domain adaptation via progressive separation. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition, pp. 2927-2936, 2019.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Proceeding of International Conference on Machine Learning, pp- 97-105, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Proceedings of Conference on Neural Information Processing Systems, 2016.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Pau Panareda Busto and Juergen Gall. Open set domain adaptation. Proceedings of the IEEE International Conference Computer Vision, pp. 754-763, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adap- tation by backpropagation. In Proceedings of the European Conference on Computer Vision, pp. 153-168, 2018.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey, Neurocomputing, 312 173:113-113, 2018
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6857-6866, 2018
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 4582-4591, 2017.


Reviews:
[{"review_id": "yigdgT7NOtA", "reviewer": "Reviewer_4Ng6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper formulates a novel problem by stepping one step further from the open-set domain adaptation to classifying the unknown classes. Overall, the problem is interesting and can be practically useful. However, the work does not seem to be solid enough for a publication in ICLR.\n\nSome of my concerns are as follows:\n\nIn the last sentence of 3.2, the authors claim \"...but also provides mor general classifiers of the known ones.\", is there any empirical evidence for this statement?\n\nIn eq (4) and (5), it is confusing to use t for both the \"boundary\" and the subsript for the target domain. And I don't understand the meaning of eq (4) and (5)\n\nI don't see how the domain adaptation module can align two domains from the objective in eq (10) and (11). L_s make sure the classification accuracy on source-domain known classes, L_adv enables the separation of known and unknown classes in the target domain, and L_GCN learns a mapping from the knowledge graph node v to the classifier weight w. How is the domain adaptation done?\n\nThe knowledge graph should also be mentioned in the problem definition in section 3.1.\n\nI believe the knowledge graph is important for good zero-shot learning, however, this information is not mentioned in the experiments.\n\nExperiments on more datasets would be helpful to validate the benefit of the proposed method.\n\nThere are some language issues to fix, e.g., \"To align the domain gap\" should be \"to bridge the domain gap\" or something else;", "summary_of_the_paper": "The paper formulates a novel problem in open-set domain adaptation and aims to classify the unknown classes in the target domain. This is different from the traditional open-set domain adaptation problem setting. To do this, additional knowledge of inter-class relations are employed and embedded so that knowledge learned from the shared classes can be transferred to the unknown classes.", "summary_of_the_review": "In summary, the paper provides a solution to a practically useful problem but the novelty is limited and the presentation is not clear enough to understand.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IEm99X81WbY", "reviewer": "Reviewer_ymSC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strong Points*\n1. Their proposed setting is interesting and realistic. \n2. The idea of using graph structure in the target domain sounds reasonable. \n\n*Weak Points* \n1. Though the high-level idea of leveraging a graph in the target domain sounds reasonable, their description of the method is not clear and not convincing enough. In Eq. 3, it is not clear where $w^{inf}$ comes from. Also, what are $M$ in Eq. 3 and Eq. 1? In addition, in Eq. 3, the objective is just to predict known classifier weights. But, why it is enough to get discriminative features for unknown samples? \n\n2. Where is the attribute $v$ come from? This is missing in both method and experiments section. \n\n3. Their evaluation is not enough to support the validity of their method. First, they perform experiments only on digits dataset, but this is clearly not enough. Second, they are lacking several baselines in open-set domain adaptation utilizing clustering methods such as \"Do We Really Need to Access the Source Data\" and \"Universal Domain Adaptation through Self Supervision\".  ", "summary_of_the_paper": "They propose a new setting in open-set domain adaptation, where the goal is to classify known classes into their classes as well as cluster unknown classes well.  A difference from an existing open-set domain adaptation is that it does only require separating unknown instances from known ones whereas this paper aims to cluster unknown instances. For this goal, they propose a model for open set domain adaptation with zero-shot learning on the unknown classes. They combine adversarial learning to align the two domains, and the knowledge graph is introduced to generate the classifiers for the unknown classes with the employment of the graph convolution network (GCN).\nThey provide experiments on digits dataset and show the gain over baselines. ", "summary_of_the_review": "Considering the weak points, I recommend rejecting this paper. There are many missing parts in methods and they need more experiments to justify their ideas. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HQP2tWnZKqu", "reviewer": "Reviewer_fmyC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths\n\ns1. Knowledge graph for learning classifiers: Unlike prior works, the paper aims at learning the classes present even in unknown bucket by using a zero-shot learning\n\ns2. The paper tackles the open-set domain adaptation by generating classifiers that work well for unknown classes by utilizing Graph learning methods.\n\n-------\n\n### Weaknesses\n\nw1. Motivation for a zero-shot learning and a clarification: Typically, Open-Set Domain-Adaptation focussed on \"rejecting\" unknown classes rather than modeling for the same explicitly [ref 1]. Therefore, these methods are useful in rejecting images even it they are not present in the pre-determined group of unknown classes. There are definitely some interesting use-cases such as incremental learning where the model adapts continually and learns new classes. So, it would be interesting to include some additional motivation for the zero-shot learning (on unknown classes) in real world with domain gap.\n\nw2. Limited novelty of the proposed approach: The domain adaptation module and adversarial training procedure is not novel. This idea has been utilized in several papers. [ref 2, 4 etc.]. For ex, [ref 4] utilized $t$-parameter to separate the unknown classes from source class boundaries. So, the novelty of the paper is limited to utilization of GCN for unknown classes. \n\nw3. Evaluation of the method and comparison against recent methods: The experiments are limited to  evaluation of the method on digits datasets such as MNIST. Although the results are encouraging, more experimental validation would be required across datasets that have richer attribute information (for Knowledge Graph). Further, the authors need to include additional comparisons from recent literature such as STA [ref 1], Inheritable models [ref 5] and closely related works such as [ref 3] (one-shot learning). \n\nw4. Experimental protocol: It would be more interesting to consider an evaluation protocol that shows accuracy of unknown classes both as a \"binary-classification\" problem and \"multi-class classification\" problem. \n\n-----\n\n### Other aspects\n\na1. In my opinion, the paper considers an interesting problem but the idea of using exactly $n_t-n_s$ classes is unreasonable for practical reasons. First, there is already a domain shift among source and target distributions. Second, the target dataset is unlabeled. Therefore, it is hard to anchor out a strong distribution in this scenario. This problem has been addressed in a related work by utilizing prototypical learning [ref 3]. \n\na2. As suggested in the above sections, the evaluation protocol should reflect the efficacy of method and its novel elements. This often requires providing additional metrics that have typically not been considered in literature. \n\na3. Including t-SNE plots would help the reader understand the qualitative performance of the proposed approach for the unknown classes.\n\n-----\n\n### Questions\n\nq1. (Approach - Training strategy) Is there a prescribed way to alternate between the two stages for training the model to gain higher performance? Or is this a heuristic that yields higher performance.\n\nq2. (Experiments - Settings section) How long does the model take to converge? The paper mentions that the model has been trained for 1200 epochs for digits dataset. I wonder if this is practical and if it would scale for larger datasets. \n\nq3. (Experiments - Comparison section) It is not clear what the domain invariance means in the context of feature alignment for unknown classes. \n\n-----\n\n** Minor Typos **\n\nm1. Closed-set domain adaptation and not \"close\"-set.\n\nm2. There are few grammatical errors make the paper little hard  to follow.\n\n----\n\n**References**\n\n[ref 1] Separate to Adapt: Open Set Domain Adaptation via Progressive Separation, CVPR'19\n\n[ref 2] Adversarial Discriminative Domain Adaptation, CVPR'17\n\n[ref 3] Class-Incremental Domain Adaptation, ECCV'20\n\n[ref 4] Open Set Domain Adaptation by Backpropagation, ECCV'18\n\n[ref 5] Towards inheritable models for open-set domain adaptation, CVPR'20", "summary_of_the_paper": "The paper considers the problem of open-set domain adaptation where the target domain has additional group of unknown classes and domain-shift with source domain. One interesting aspect of the paper is that the method utilizes a Knowledge Graph for zero-shot learning on the unknown classes. Further, the method utilizes an adversarial learning approach to align the source and target domain.", "summary_of_the_review": "I have provided the review for the paper in the previous sections. Although, the paper considers an interesting problem setting, more experimental validation would be required to judge its claims. Further, the novelty is limited to application of an already existing ideas. I'm happy to take the discussion with authors in the post-review phase to hear their thoughts. At the moment, the paper is simply not in acceptable state. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses open-set DA, where samples from novel classes in the target domain get clustered 
into new (unlabeled) classes. A key novelty in the learning setup is that it is assumed that one 
has access to a knowledge graph over classes (both source and target). That KG is used for grouping 
target samples into novel classes. 

Reviewers were concerned that the method is not explained with sufficient 
details and the experiments lacked comparisons with openset DA baselines. 
No rebuttal was submitted. 

The paper cannot be accepted to ICLR.
Decision: Reject