Open Set Domain Adaptation with Zero-shot Learning on Graph

1 INTRODUCTION . In the last decades , deep learning models have shown good performance in various tasks , especially in visual perception . The training of the deep learning network relies on plenty of labeled data . However , most of the existing large labeled datasets are collected from the Internet . The images in these datasets are normative and unified , which are different from the images relevant for a specific application . Besides , depending on the application , the images may be obtained by different typed of visual sensors or with a different perspective of sensors . It costs a lot to retrain the classification model in different situations . In some typical applications , the samples in the real world are hard to gather or too large to label . Thus it is important to deal with the gap among domains . They should be able to utilize the well-labeled samples in the source domain to classify the samples in the unlabeled target domain which is related to domain adaptation . There are already some researches on domain adaptation , such as Ganin & Lempitsky ( 2015 ) , Long et al . ( 2015 ) , Long et al . ( 2016 ) , and Wang & Deng ( 2018 ) . The alignment of the domain gap makes the robot adapt well to dynamic and unstructured environments . Except for the domain gap among different datasets , the variation of the classes also makes it hard for the model to adapt to a new dataset . Depending on the application and the scale of different datasets , the model may come across classes that are not contained in the source domains . With the traditional domain adaptation methods , the unknown classes are mistakenly aligned due to the absence of training samples of unknown classes in the source domain . The imbalance of the types of classes brings over-fitting problems and is not suitable for classification in the open world . Thus it is important for the robot to reject the unknown classes and only align the shared classes . This problem is known as open set domain adaptation , which is first proposed by Panareda Busto & Gall ( 2017 ) and followed by for instance Saito et al . ( 2018 ) , Busto et al . ( 2018 ) , and Liu et al . ( 2019 ) . In the setting of the open set domain adaptation , the target domain contains both the classes of the source domain and the additional new classes . The model not only aligns the target domain to the source domain but also rejects the unknown classes . It is worth noting that previous open set domain adaptation methods typically classify all the additional new classes into one unknown class . However , the unknown class may contain classes that are worth learning . It may be more valuable to detect the unknown classes in detail and develop the ability to classify them with the former information . With the process of distinction and transferring , the model can expand its visual recognition ability with little labeled information . Since the unknown classes are not included in the source domain , the model lacks the labeled information for the new classes . Current open set domain adaptation methods can not give detailed classification on the unknown part with no labeled images . This problem is related to zero-shot learning . In the zero-shot learning problem , complementary information is collected to transfer the knowledge from the base classes to classify the unknown ones . Inspired by this , with the knowledge stored in the knowledge graph , the classifiers of the unknown classes can be obtained in the target domain with no labeled samples . Towards this end , we propose a generic model to align the gap between the labeled source domain and the unlabeled target domain while classifying the unknown classes in the target domain . The contributions of this paper mainly lies in tackling the following two difficulties . First , since the unknown classes are not contained in the source domain , we have no labeled samples for supervised training . The lack of labeled data may cause overfitting problem of the model , which means the model only classify the samples as the known classes and can not classify the unknown ones . It is necessary to utilize complementary information to support the inference . Thus we employ the knowledge graph to stores some prior knowledge of the known classes and the unknown classes , which contains the structural relations between different classes , beyond the individual attribute representation of each class . The structural information offers a bridge for the inference from the known classes to the unknown ones . With the employment of the graph convolution network , the information propagates among the graph and the unknown classes gather the information from their neighbor to generate their classifiers . These inference classifiers work as the initial classifiers of the classification model . The second difficulty is how to adapt the inference classifiers to the target domain . Since we only have labeled samples in the source domain , the inference classifiers are suitable to the source domain . It is not able to classify the unknown samples in the target domain because of the domain gap . Thus we introduce adversarial learning to align the domain gap . The classification model consists of two modules , the feature generator , and the classifier . Since the generator works to extracts the features of the samples and the classifier works to output the class probability , we train them simultaneously in an adversarial way . The classifier is trained to found a boundary for the unknown classes while the generator is trained to make the samples far from the boundary . With adversarial learning , the generator can deceive the classifier to generate aligned features in both domains and reject the unknown classes according to the unknown boundary . Thus the feature of the shared classes is aligned in both domains and the unknown classes are rejected as one class . With the adaptation in both domain gap and class gap , our model is able to classify objects in the dynamic and complex open world . We utilize the knowledge graph and the adversarial learning in a jointly trained framework . The two parts work together to align the shared classes in two spaces while generating classifiers for the unknown classes in the target domain . We further evaluate our method on digits datasets and demonstrate its effectiveness . 2 RELATED WORKS . 2.1 OPEN SET DOMAIN ADAPTATION . Open set domain adaptation goes beyond traditional close set domain adaptation . It considers a more realistic classification task , in which the target domain contains unknown samples that are not present in the source domain . Open set domain adaptation is first proposed by Busto et al . ( 2018 ) . They measure the distance between the target sample and the center of the source class to decide whether a target sample belongs to one of the source classes or the unknown class . However , they require the source domain to have unknown samples as well . Later on , Saito et al . ( 2018 ) propose open set back-propagation ( OSBP ) for source domain with no unknown samples . They utilize adversarial learning to train the feature generator and classifier . As the classifier tries to set a boundary for the unknown classes , the feature generator tries to deceive it . However , both of them only separate the unknown classes in the target domain , but can not give detailed classification on the unknown ones . The learnable information in the unknown space deserves deep exploitation . We have found few papers that consider the fine-grained classification of the unknown classes in open set domain adaptation , we aim to fill in the blanks . 2.2 ZERO-SHOT LEARNING . Zero-shot learning aims at generating classifiers for unknown classes with no labeled samples . Several pieces of research have been done on this area , such as Kipf & Welling ( 2016 ) Xian et al . ( 2017 ) . Due to the limitation of the available samples , some researchers extract complementary information from the related known classes to support the inference of the unknown ones . Among these methods , building the relationship between classes in form of a graph seems more reasonable . The special geometry of graphs well shows the complicated relationship and the unknown classes can gather adequate information from the known ones . Current zero-shot learning please refer to knowledge graphs for inference . Wang et al . ( 2018 ) built an unweighted knowledge graph combined with word embedding upon the graph convolutional network . With information propagation , novel nodes generate predictive classifiers with common sense . Kampffmeyer et al . ( 2019 ) improve upon this model and propose a dense graph propagation to prevent dilution of knowledge from distant nodes . Knowledge graphs intuitively present the stored information for the visual cognitive development . 2.3 GRAPH CONVOLUTIONAL NETWORK . Graph convolutional network ( GCN ) is a kind of graph neural network proposed by Kipf & Welling ( 2016 ) . GCN enables the nodes in the graph to share the intensity of statistics , which improves the efficiency of sampling . GCN can also be employed in non-euclidean space . From the perspective of the spatial domain , GCN iteratively aggregates neighborhood information . The propagation of information on the graph further exploits the structural information on the graph . GCN is first introduced by Bruna et al . ( 2013 ) . Then GCN is extended with a filtering method based on the recurrent Chebyshev polynomial . The improvement reduces the computational complexity a lot , which is equivalent to common CNNs in image operating . Kipf & Welling ( 2016 ) further simplify the framework to improve the scalability and robustness . They employed their model on the semisupervised learning on the graphs . Our model employs the framework of GCN to propagate the complementary information among nodes for the inference of classifiers . 3 APPROACH . 3.1 PROBLEM DEFINITION . In open set domain adaptation with zero-shot learning , we have a source domain Ds = { ( xsi , ysi ) } ns i=1 , which contains ns labeled samples , and a target domain Dt = { xtj } nt j=1 , which contains nt unlabeled samples . The class space in the source domain is Cs which we call known classes . Cs also is shared by the class space of the target domainCt . It is worth noting thatCt further contains additional unknown classes Cu , that is Ct = Cs ∨Cu . The source domain is sampled from the distribution qs , while the target domain is sampled from the distribution qt . In close set domain adaptation , qs 6= qt . In open set domain adaptation with zero-shot learning , we also define qs 6= qst . qst refers to the distribution of the known classes in the target domain . Note that the samples in the target domain are all unlabeled and the samples in the source domain are all labeled . 3.2 CLASSIFIER INFERENCE MODULE . With few labeled samples , the human can make good inferences on unfamiliar things with the related information that they obtain from books . Our model also extracts the task-based knowledge from a prestored knowledge graph . The inference graph is denoted as G = ( V , E ) , where V = { v1 , v2 , ... , vns , ... , vnt } is a node-set of all classes Ct. ns refers to the number of known classes . nt − ns refers to the number of unknown classes . The nodes in the prestored knowledge contain the attributes vi of different classes . E = { ei , j = ( vi , vj ) } is an edge set referring to the relationship among the graph . The edges in the prestored knowledge graph are based on the similarity of the attributes between different classes . Since we only have labeled data of the source domain . We first train the recognition model on the source domain Ds . Specifically , the pre-trained recognition model is denoted as C ( F ( ·|θ ) |w ) . The model consists of two parts , feature extractor F ( ·|θ ) and class classifier C ( ·|w ) . θ and w indicate the parameters of the model trained withDs = { ( x1 , y1 ) , ... , ( xns , ys ) } . The symbol xi refers to the source images of the ith class while yi refers to their label . Feature extractor F ( x|θ ) takes an image as input and figures out the feature vector of it as zi . The final classification score is computed as [ s1 , s2 , ... , sM ] = [ z Tw1 , z Tw2 , ... , z Twns ] ( 1 ) Thus the inference of the classifiers on unknown classes is to inference the classification weights ws on the unknown classes with the inference graph . With the framework of the graph convolutional network , Our model propagates information among nodes by exploring the class relationship . For one layer in GCN , a node aggregates information from the neighbors connected to it . GCN can also be extended to multiple layers to perform a deeper spread . Therefore , the unknown classes can utilize the information from the related known classes and predict the classification weights of their own . The mechanism of GCN is described as H ( l+1 ) = ReLu ( D̂− 1 2 ÊD̂− 1 2H ( l ) U ( l ) ) ( 2 ) where H ( l ) denotes the output of the lth layer , while for the first layer H0 = V . It uses Leaky ReLu as the nonlinear activation function . To reserve the self-information of the nodes , self-loops are added among the propagation , Ê = E + I , where E ∈ RN×N is the symmetric adjacency matrix and I ∈ RN×N represents identity matrix . Dii = ∑ j Eij normalizes rows in E to prevent the scale of input modified by E. The matrix U l is the weight matrix of the lth layer , which GCN regulates constantly to achieve better performance . Our model conducts two layers of GCN on the inference graph . Unknown classes learn the mechanism of end-to-end learning from known classes through propagation . The inference graph is trained to minimize the loss between the predicted classification weights and the ground-truth weights . The ground-truth weights refer to the classifiers of the known classes , which are extracted from the pretrained model on the source domain . LGCN = 1 M M∑ i=1 ( winfi − w train i ) 2 ( 3 ) wherewinf refers to the output of known classes on GCN.wtrain denotes the ground truth classifiers of the known classes obtained from the pre-trained model . With the supervision of the known classes , the unknown nodes in the inference graph can also generate classifier weights of their own . Finally , with the employment of GCN , the classifier inference module not only generates predictive classifiers of the unknown classes in the source domain but also provides more general classifiers of the known ones .

Summary: The paper considers the problem of open-set domain adaptation where the target domain has additional group of unknown classes and domain-shift with source domain. One interesting aspect of the paper is that the method utilizes a Knowledge Graph for zero-shot learning on the unknown classes. Further, the method utilizes an adversarial learning approach to align the source and target domain.