DOMAIN ADAPTATION VIA DISTRIBUTION AND REPRESENTATION MATCHING: A CASE STUDY ON TRAINING DATA SELECTION VIA REINFORCEMENT LEARNING

1 INTRODUCTION . Learning with massive data suffers from “ Pyrrhic victory ” where huge amounts of resource , e.g. , computation , annotation , storage , etc. , are consumed with many issues . One well-known issue for data-driven approaches is that data quality considerably affects the performance of learned models . Especially in NLP , such phenomenon is incredibly significant where noise and inaccurate annotations are demolishing models ’ robustness when applying them to various scenarios ( Akopyan & Khashba , 2017 ; Zhang et al. , 2016 ) , one of which is cross-domain NLP tasks . In this scenario , many data and their labels are considered noisy and inaccurate when they are used across domains , resulting in the mismatch of distribution between training and test data . To reduce the impact of the noise in data and annotations , reasonable selection of training samples is proved to be an useful strategy that can help supervised models achieving equivalent performance with less computational efforts ( Fan et al. , 2017 ; Feng et al. , 2018 ) . Particularly , it is also confirmed that appropriate data selection strategy can benefit domain adaptation ( Ruder & Plank , 2017 ) by preventing negative transfer ( Rosenstein et al. , 2005 ) from irrelevant data and noisy labels , meanwhile providing a more efficient learning process compared with other domain adaptation methods such as sample reweighing ( Borgwardt et al. , 2006 ) , feature distribution matching ( Tzeng et al. , 2014 ) and representation learning ( Csurka , 2017 ) . Although various data selection-based domain adaptation approaches were proposed for NLP tasks ( III , 2007 ; Blitzer et al. , 2007 ; Søgaard , 2011 ) , most of them only consider scoring or ranking learning samples under a certain metric over the entire dataset , then select the top n ( or a proportion , which is usually a predefined hyper-parameter ) items to learn . However , such pre-designed metrics is not always able to cover effective characteristics for transferring domain knowledge and meanwhile suffers from their inflexibility . Even though some methods learn measures across domains and models ( Ruder & Plank , 2017 ) , their hyper-parameter setting , such as the n , still demands further explorations . Fortunately , recent studies confirmed that RL is an effective solution for data selection in training neural networks with improving their efficiency ( Fan et al. , 2017 ) and accuracy ( Feng et al. , 2018 ) . A natural extension of using RL for data selection can be considered for the cross-domain scenario . Yet , to apply RL to data selection for domain adaptation is not a straightforward mission as those RL application for in-domain tasks . To tackle this mission , in this paper , we propose a general data selection approach for domain adaptation via RL that jointly learns data representations and matches them from the source and target domain . There are two major components in our approach : a selection distribution generator ( SDG ) and a task-specific predictor with a feature extractor and a classifier1 . The SDG and the predictor are pipelined by taking each others ’ output as input and optimized accordingly via RL . As a result , in our approach , two major problems in domain adaptation are addressed : some redundant or noisy data across domains gives little or negative contribution to adaptation ; useful information from the source domain is usually not properly organized and exploited for the target domain . Experimental results from three NLP tasks , namely , part-of-speech ( POS ) tagging , dependency parsing and sentiment analysis , illustrate that our approach achieve state-of-the-art performance , which confirm the validity and effectiveness of our approach . 2 THE APPROACH . Our approach is in the line of the unsupervised domain adaptation ( Roark & Bacchiani , 2003 ) , where one has no access to any labels of target domain data . Particularly , for a task T , we take labeled data instances for the task from a source domain DS as the input to our approach , and some unlabeled data from a target domain DT as the guidance . The approach is then expected to generate an optimal selected data set and a model trained on the set for T . An overview illustration of the entire architecture for our approach is demonstrated in Figure 1 . In detail , for each step , the key component for selection , SDG , produces a distribution vector based on the representation of the selected source data from last step , then data instances are selected according to the vector and new reward is generated for next round data selection . To update the SDG , different measurements can be used to assess the discrepancy between the representations of the selected source data and the guidance set and then approximates the value function for updating . For the predictor , the loss from the classifier on the domain-specific task is used to guide the parameter updating for the feature extractor and the classifier . The details is unfolded in the following subsections , in which we firstly give the details of the two major components and then the joint training algorithm . 2.1 THE PREDICTOR . The predictor is the main component to perform a particular task T . In our approach we decompose the predictor into two parts , the feature extractor and a classifier , and use them separately . The feature extractor serves as the representation learning module that transform input data to dense representations , while the classifier takes the representations for T in the target domain . In doing so , the predictor is a neural model in this study so that the aforementioned separation can be done . Therefore , the feature extractor is normally the first n-1 layers of the predictor given that it has n layers in total . As a result , the classifier is the last layer dedicated for model output . The Feature Extractor Data in its original form , especially as natural language , is usually difficult to be exploited in neural computation . The feature extractor is thus serving as a critical component in our approach to transform the data into distributed representations for their efficient use and computation . There are two-way inputs for the feature extractor . One is the guidance set XTg = { xT1 , xT2 , ... , xTm } , a collection of unlabeled data drawn from the target domain , serving as the reference for the selection process . The other input is the selected data from the source domain in a “ data bag ” , which is a batch of a certain amount of samples to facilitate the selection in this study . Let XS = { xS1 , xS2 , ... , xSn } , ∀xSi ∈ XS denote the data from the source domain , we uniformly partitions the entire data set into N disjoint data bags marked as { B1 , B2 , ... , BN } , with Bj = { xS ( j−1 ) n/N+1 , x S ( j−1 ) n/N+2 , ... , x S jn/N } , j ∈ { 1 , 2 , ... , N } . Through the feature extractor , the outputs of the guidance set and the selected data are two collections of distribution vectors.2 The Classifier Once data selection is done , the classifier is combined with the feature extractor and trained on the selected data for a particular task . Note that the training is conducted in the source domain and the output of the classifier is used to generate gradients for back-propagation . Then the parameters of the classifier and the feature extractor are updated accordingly ( with a learning rate β ) . 2.2 THE SELECTION DISTRIBUTION GENERATOR . A multi-layer perceptron ( MLP ) model is used as the SDG , which learns the selection policy optimized by the reward from the representations of the guidance set and the selected data by RL . In doing so , at each time step , the SDG is fed by a collection of data representations for a data bag from the feature extractor . We denote the collection as ΦBj = { r j 1 , r j 2 , ... , r j |Bj | } , where r j l 1It is not necessarily a classifier , e.g. , such as a tagger . However we use the term classifier for simplicity . 2The formal description of such representations are given in the next section . ( l = 1 , 2 , ... , |Bj | ) is the vector of the l-th sample3 in Bj .4 Then ΦBj is mapped by the SDG to a vector DBj = ( p j 1 , p j 2 , ... , p j |Bj | ) , p j l ( l = 1 , 2 , ... , |Bj | ) , which represents the probability for each sample that whether it should be selected individually . To learn the SDG , each ΦBj is measured with Φt to give the reward in an RL framework , which is described as follows . RL Components in Learning the SDG . • State ( s1 , s2 , ... sj , ... sN ) includes a collection of states for all j with respect to N data bags , where each sj indicates a status including data bag B̂j and parameters of the feature extractor for the selected instances B̂j from Bj . For simplicity we use ΦB̂j and Φt to represent state sj . • Action For each state , the action space A is a 0-1 judgment to decide if selecting an instance ( 1 ) or not ( 0 ) . An action a = { ak } |Bj | k=1 ∈ { 0 , 1 } |Bj | , which is obtained from DB̂j . After each action , data selection gives new ΦB̂j , then transforms state s into s ′ . Thus we define the policy as PW ( a|s ) . • Reward Since the data selection process is to ensure that the selected data fitting the distribution of the target domain , we set a reward r ( s , a , s′ ) to assess the distance between ΦB̂j and Φt in the current state ( s′ ) and its previous state ( s ) : r ( s , a , s′ ) = d ( Φs B̂j−1 , Φst ) − γd ( Φs ′ B̂j , Φs ′ t ) ( 1 ) where d ( · , · ) is a distribution discrepancy measurement , which can be implemented by different information-bearing functions . γ ∈ ( 0 , 1 ) is a discounting constant that decreases the impact from future distribution difference . Note that Eq . ( 1 ) is conducted in a sequence based on two adjacent data bags Bj−1 and Bj , of which Φs ′ B̂j are impacted by Φs B̂j−1 via parameters Ψ of the feature extractor updated by B̂j−1 . Consequently , the state transition probability P ( s′|s , a ) is determined by stochastic optimization and other randomness in training , e.g. , dropout ( Srivastava et al. , 2014 ) . When better instances are selected , the reward is then expected to produce a higher value because the measurement for the previous state d ( Φs B̂j−1 , Φst ) supposes to give a larger distance between ΦB̂j−1 and Φt than that for the current state . Distribution Discrepancy Measurement To measure each B̂j and the XTg , let P be the elementwise average of ΦB̂j and Q the average of Φt , we use the following measurements for d ( · , · ) in this paper : • JS : The Jensen-Shannon divergence ( Lin , 1991 ) . d ( P , Q ) = 12 [ DKL ( P ||M ) + DKL ( Q||M ) ] where DKL ( P ||Q ) = ∑n i=1 pi log pi qi , with M = 12 ( P +Q ) . • MMD : The maximum mean discrepancy ( Borgwardt et al. , 2006 ) . d ( P , Q ) = ‖P −Q‖ . • RÉNYI : The symmetric Rényi divergence ( Rényi , 1961 ) . d ( P , Q ) = 12 [ Ry ( P , M ) +Ry ( Q , M ) ] , Ry ( P , Q ) = 1α−1 log ( ∑n i=1 pαi qα−1i ) . We set α=0.99 following Van Asch & Daelemans ( 2010 ) . 3Representations in the collection follow the same order of their corresponding data instances in the bag . 4Similarly , the representation collection of the guidance set is denoted as Φt . Algorithm 1 : Joint training algorithm of our approach Input : Epochs L ; Training data B = { B1 , B2 , ... , BN } . W ( SDG ) , Ψ ( Predictor ) , Θ ( feature extractor ) ; Loss function of the predictor F ( Ψ , B̂j ) ; nJ ; d ( · , · ) ; γ . Output : Updated W and Ψ ( Θ ) . Initialize W and Ψ ( Θ ) with standard Gaussian distribution ; for epoch l = 1 to L do Σ = 0 ; for k = 1 to nJ do Σr = 0 ; Shuffle { B1 , B2 , ... , BN } ; for each Bj ∈ B do Φ sj Bj ← Θj−1 ( Bj ) ; Φ sj t ← Θj−1 ( XTg ) ; On current bag state sj , Dj ←W ( Φ sj t ) ; select Φ sj B̂j from ΦsjBj via Dj ( take action aj ) ; r ( sj−1 , aj , sj ) ← d ( Φ sj−1 B̂j−1 , Φ sj−1 t ) − γd ( Φ sj Bj , Φ sj t ) Σr ← Σr + γj−1r ( sj−1 , aj , sj ) Ψ← Ψ− β∇ΨF ( Ψ , B̂j ) ; ( Θj ← Θj−1 − β∇ΘF ( Ψ , B̂j ) ) ; end Σ← Σ + ∑N j=1∇W log πW ( a k j |skj ) Σr ; end ∇WJ̃ ( W ) ← 1 nJ Σ ; W←W + τ∇WJ̃ ( W ) ; end • LOSS : The guidance loss . d = − 1m ∑m i=1 ∑ yt∈Yt yt log pΦ ( yt|x T i ) where yt is the label of instance t from the guidance set , and pφ the learned conditional probability of the predictor . Note that , different from aforementioned measurements , LOSS requires labels from the target domain , thus is only set as a comparison to our approach with using annotated data through the predictor . Optimization The following object is optimized to obtain the optimal distribution generation policy . J ( W ) = EPW ( a|s ) [ N∑ j=1 γj−1r ( sj , aj ) ] ( 2 ) Then the parameters of the SDG , i.e. , W , is updated via policy gradient ( Sutton et al. , 1999 ) by W←W + τ∇WJ̃ ( W ) ( 3 ) where τ is the discounting learning rate5 , the gradient ∇WJ ( W ) is approximated by ∇WJ̃ ( W ) = 1 nJ ∑nJ k=1 ∑N j=1∇W log πW ( akj |skj ) ∑N j=1 γ j−1r ( skj , a k j ) , where j refers to the j-th step ( corre- sponding to the j-th data bag ) in RL ; i denotes the i-th selection process to estimate∇WJ ( W ) . Note that∇WJ ( W ) is estimated after nJ times of selection over all N data bags , where nJ is a predefined hyper-parameter .

Summary: The paper aims to address issues with Domain Adaptation by using RL approaches. Domain Adaptation is an actively studied area in NLP research and so this paper is relevant and timely. This paper proposes and algorithm that is in line with work that aims at selecting data smartly when performing Domain Adaptation. The proposed algorithm learns representations for text in the source and target domains jointly. The proposed algorithm has two components i) a selection distribution generator (SDG) and ii) a task specific prediction for tasks being POS tagging, Dependency parsing and Sentiment Analysis.