Open Set Domain Adaptation with Zero-shot Learning on Graph

1 INTRODUCTION . In the last decades , deep learning models have shown good performance in various tasks , especially in visual perception . The training of the deep learning network relies on plenty of labeled data . However , most of the existing large labeled datasets are collected from the Internet . The images in these datasets are normative and unified , which are different from the images relevant for a specific application . Besides , depending on the application , the images may be obtained by different typed of visual sensors or with a different perspective of sensors . It costs a lot to retrain the classification model in different situations . In some typical applications , the samples in the real world are hard to gather or too large to label . Thus it is important to deal with the gap among domains . They should be able to utilize the well-labeled samples in the source domain to classify the samples in the unlabeled target domain which is related to domain adaptation . There are already some researches on domain adaptation , such as Ganin & Lempitsky ( 2015 ) , Long et al . ( 2015 ) , Long et al . ( 2016 ) , and Wang & Deng ( 2018 ) . The alignment of the domain gap makes the robot adapt well to dynamic and unstructured environments . Except for the domain gap among different datasets , the variation of the classes also makes it hard for the model to adapt to a new dataset . Depending on the application and the scale of different datasets , the model may come across classes that are not contained in the source domains . With the traditional domain adaptation methods , the unknown classes are mistakenly aligned due to the absence of training samples of unknown classes in the source domain . The imbalance of the types of classes brings over-fitting problems and is not suitable for classification in the open world . Thus it is important for the robot to reject the unknown classes and only align the shared classes . This problem is known as open set domain adaptation , which is first proposed by Panareda Busto & Gall ( 2017 ) and followed by for instance Saito et al . ( 2018 ) , Busto et al . ( 2018 ) , and Liu et al . ( 2019 ) . In the setting of the open set domain adaptation , the target domain contains both the classes of the source domain and the additional new classes . The model not only aligns the target domain to the source domain but also rejects the unknown classes . It is worth noting that previous open set domain adaptation methods typically classify all the additional new classes into one unknown class . However , the unknown class may contain classes that are worth learning . It may be more valuable to detect the unknown classes in detail and develop the ability to classify them with the former information . With the process of distinction and transferring , the model can expand its visual recognition ability with little labeled information . Since the unknown classes are not included in the source domain , the model lacks the labeled information for the new classes . Current open set domain adaptation methods can not give detailed classification on the unknown part with no labeled images . This problem is related to zero-shot learning . In the zero-shot learning problem , complementary information is collected to transfer the knowledge from the base classes to classify the unknown ones . Inspired by this , with the knowledge stored in the knowledge graph , the classifiers of the unknown classes can be obtained in the target domain with no labeled samples . Towards this end , we propose a generic model to align the gap between the labeled source domain and the unlabeled target domain while classifying the unknown classes in the target domain . The contributions of this paper mainly lies in tackling the following two difficulties . First , since the unknown classes are not contained in the source domain , we have no labeled samples for supervised training . The lack of labeled data may cause overfitting problem of the model , which means the model only classify the samples as the known classes and can not classify the unknown ones . It is necessary to utilize complementary information to support the inference . Thus we employ the knowledge graph to stores some prior knowledge of the known classes and the unknown classes , which contains the structural relations between different classes , beyond the individual attribute representation of each class . The structural information offers a bridge for the inference from the known classes to the unknown ones . With the employment of the graph convolution network , the information propagates among the graph and the unknown classes gather the information from their neighbor to generate their classifiers . These inference classifiers work as the initial classifiers of the classification model . The second difficulty is how to adapt the inference classifiers to the target domain . Since we only have labeled samples in the source domain , the inference classifiers are suitable to the source domain . It is not able to classify the unknown samples in the target domain because of the domain gap . Thus we introduce adversarial learning to align the domain gap . The classification model consists of two modules , the feature generator , and the classifier . Since the generator works to extracts the features of the samples and the classifier works to output the class probability , we train them simultaneously in an adversarial way . The classifier is trained to found a boundary for the unknown classes while the generator is trained to make the samples far from the boundary . With adversarial learning , the generator can deceive the classifier to generate aligned features in both domains and reject the unknown classes according to the unknown boundary . Thus the feature of the shared classes is aligned in both domains and the unknown classes are rejected as one class . With the adaptation in both domain gap and class gap , our model is able to classify objects in the dynamic and complex open world . We utilize the knowledge graph and the adversarial learning in a jointly trained framework . The two parts work together to align the shared classes in two spaces while generating classifiers for the unknown classes in the target domain . We further evaluate our method on digits datasets and demonstrate its effectiveness . 2 RELATED WORKS . 2.1 OPEN SET DOMAIN ADAPTATION . Open set domain adaptation goes beyond traditional close set domain adaptation . It considers a more realistic classification task , in which the target domain contains unknown samples that are not present in the source domain . Open set domain adaptation is first proposed by Busto et al . ( 2018 ) . They measure the distance between the target sample and the center of the source class to decide whether a target sample belongs to one of the source classes or the unknown class . However , they require the source domain to have unknown samples as well . Later on , Saito et al . ( 2018 ) propose open set back-propagation ( OSBP ) for source domain with no unknown samples . They utilize adversarial learning to train the feature generator and classifier . As the classifier tries to set a boundary for the unknown classes , the feature generator tries to deceive it . However , both of them only separate the unknown classes in the target domain , but can not give detailed classification on the unknown ones . The learnable information in the unknown space deserves deep exploitation . We have found few papers that consider the fine-grained classification of the unknown classes in open set domain adaptation , we aim to fill in the blanks . 2.2 ZERO-SHOT LEARNING . Zero-shot learning aims at generating classifiers for unknown classes with no labeled samples . Several pieces of research have been done on this area , such as Kipf & Welling ( 2016 ) Xian et al . ( 2017 ) . Due to the limitation of the available samples , some researchers extract complementary information from the related known classes to support the inference of the unknown ones . Among these methods , building the relationship between classes in form of a graph seems more reasonable . The special geometry of graphs well shows the complicated relationship and the unknown classes can gather adequate information from the known ones . Current zero-shot learning please refer to knowledge graphs for inference . Wang et al . ( 2018 ) built an unweighted knowledge graph combined with word embedding upon the graph convolutional network . With information propagation , novel nodes generate predictive classifiers with common sense . Kampffmeyer et al . ( 2019 ) improve upon this model and propose a dense graph propagation to prevent dilution of knowledge from distant nodes . Knowledge graphs intuitively present the stored information for the visual cognitive development . 2.3 GRAPH CONVOLUTIONAL NETWORK . Graph convolutional network ( GCN ) is a kind of graph neural network proposed by Kipf & Welling ( 2016 ) . GCN enables the nodes in the graph to share the intensity of statistics , which improves the efficiency of sampling . GCN can also be employed in non-euclidean space . From the perspective of the spatial domain , GCN iteratively aggregates neighborhood information . The propagation of information on the graph further exploits the structural information on the graph . GCN is first introduced by Bruna et al . ( 2013 ) . Then GCN is extended with a filtering method based on the recurrent Chebyshev polynomial . The improvement reduces the computational complexity a lot , which is equivalent to common CNNs in image operating . Kipf & Welling ( 2016 ) further simplify the framework to improve the scalability and robustness . They employed their model on the semisupervised learning on the graphs . Our model employs the framework of GCN to propagate the complementary information among nodes for the inference of classifiers . 3 APPROACH . 3.1 PROBLEM DEFINITION . In open set domain adaptation with zero-shot learning , we have a source domain Ds = { ( xsi , ysi ) } ns i=1 , which contains ns labeled samples , and a target domain Dt = { xtj } nt j=1 , which contains nt unlabeled samples . The class space in the source domain is Cs which we call known classes . Cs also is shared by the class space of the target domainCt . It is worth noting thatCt further contains additional unknown classes Cu , that is Ct = Cs ∨Cu . The source domain is sampled from the distribution qs , while the target domain is sampled from the distribution qt . In close set domain adaptation , qs 6= qt . In open set domain adaptation with zero-shot learning , we also define qs 6= qst . qst refers to the distribution of the known classes in the target domain . Note that the samples in the target domain are all unlabeled and the samples in the source domain are all labeled . 3.2 CLASSIFIER INFERENCE MODULE . With few labeled samples , the human can make good inferences on unfamiliar things with the related information that they obtain from books . Our model also extracts the task-based knowledge from a prestored knowledge graph . The inference graph is denoted as G = ( V , E ) , where V = { v1 , v2 , ... , vns , ... , vnt } is a node-set of all classes Ct. ns refers to the number of known classes . nt − ns refers to the number of unknown classes . The nodes in the prestored knowledge contain the attributes vi of different classes . E = { ei , j = ( vi , vj ) } is an edge set referring to the relationship among the graph . The edges in the prestored knowledge graph are based on the similarity of the attributes between different classes . Since we only have labeled data of the source domain . We first train the recognition model on the source domain Ds . Specifically , the pre-trained recognition model is denoted as C ( F ( ·|θ ) |w ) . The model consists of two parts , feature extractor F ( ·|θ ) and class classifier C ( ·|w ) . θ and w indicate the parameters of the model trained withDs = { ( x1 , y1 ) , ... , ( xns , ys ) } . The symbol xi refers to the source images of the ith class while yi refers to their label . Feature extractor F ( x|θ ) takes an image as input and figures out the feature vector of it as zi . The final classification score is computed as [ s1 , s2 , ... , sM ] = [ z Tw1 , z Tw2 , ... , z Twns ] ( 1 ) Thus the inference of the classifiers on unknown classes is to inference the classification weights ws on the unknown classes with the inference graph . With the framework of the graph convolutional network , Our model propagates information among nodes by exploring the class relationship . For one layer in GCN , a node aggregates information from the neighbors connected to it . GCN can also be extended to multiple layers to perform a deeper spread . Therefore , the unknown classes can utilize the information from the related known classes and predict the classification weights of their own . The mechanism of GCN is described as H ( l+1 ) = ReLu ( D̂− 1 2 ÊD̂− 1 2H ( l ) U ( l ) ) ( 2 ) where H ( l ) denotes the output of the lth layer , while for the first layer H0 = V . It uses Leaky ReLu as the nonlinear activation function . To reserve the self-information of the nodes , self-loops are added among the propagation , Ê = E + I , where E ∈ RN×N is the symmetric adjacency matrix and I ∈ RN×N represents identity matrix . Dii = ∑ j Eij normalizes rows in E to prevent the scale of input modified by E. The matrix U l is the weight matrix of the lth layer , which GCN regulates constantly to achieve better performance . Our model conducts two layers of GCN on the inference graph . Unknown classes learn the mechanism of end-to-end learning from known classes through propagation . The inference graph is trained to minimize the loss between the predicted classification weights and the ground-truth weights . The ground-truth weights refer to the classifiers of the known classes , which are extracted from the pretrained model on the source domain . LGCN = 1 M M∑ i=1 ( winfi − w train i ) 2 ( 3 ) wherewinf refers to the output of known classes on GCN.wtrain denotes the ground truth classifiers of the known classes obtained from the pre-trained model . With the supervision of the known classes , the unknown nodes in the inference graph can also generate classifier weights of their own . Finally , with the employment of GCN , the classifier inference module not only generates predictive classifiers of the unknown classes in the source domain but also provides more general classifiers of the known ones .

Summary: They propose a new setting in open-set domain adaptation, where the goal is to classify known classes into their classes as well as cluster unknown classes well.  A difference from an existing open-set domain adaptation is that it does only require separating unknown instances from known ones whereas this paper aims to cluster unknown instances. For this goal, they propose a model for open set domain adaptation with zero-shot learning on the unknown classes. They combine adversarial learning to align the two domains, and the knowledge graph is introduced to generate the classifiers for the unknown classes with the employment of the graph convolution network (GCN). They provide experiments on digits dataset and show the gain over baselines.