Year: 2018

Abstract:
Supervised models suffer from domain shifting where distribution mismatch across domains greatly affect model performance. Particularly, noise scattered in each domain has played a crucial role in representing such distribution, especially in various natural language processing (NLP) tasks. In addressing this issue, training data selection (TDS) has been proven to be a prospective way to train supervised models with higher performance and efficiency. Following the TDS methodology, in this paper, we propose a general data selection framework with representation learning and distribution matching simultaneously for domain adaptation on neural models. In doing so, we formulate TDS as a novel selection process based on a learned selection distribution from the input data, which is produced by a trainable selection distribution generator (SDG) that is optimized by reinforcement learning (RL). Then, the model trained by the selected data not only predicts the target domain data in a specific task, but also provides input for the value function of the RL, which exactly helps update the parameters of SDG in a policy gradient way. Experiments are conducted on three typical NLP tasks, namely, part-of-speech tagging, dependency parsing, and sentiment analysis. Results demonstrate the validity and effectiveness of our approach.

heading: 1 INTRODUCTION
text: Learning with massive data suffers from “Pyrrhic victory” where huge amounts of resource, e.g., computation, annotation, storage, etc., are consumed with many issues. One well-known issue for data-driven approaches is that data quality considerably affects the performance of learned models. Especially in NLP, such phenomenon is incredibly significant where noise and inaccurate annotations are demolishing models’ robustness when applying them to various scenarios (Akopyan & Khashba, 2017; Zhang et al., 2016), one of which is cross-domain NLP tasks. In this scenario, many data and their labels are considered noisy and inaccurate when they are used across domains, resulting in the mismatch of distribution between training and test data. To reduce the impact of the noise in data and annotations, reasonable selection of training samples is proved to be an useful strategy that can help supervised models achieving equivalent performance with less computational efforts (Fan et al., 2017; Feng et al., 2018). Particularly, it is also confirmed that appropriate data selection strategy can benefit domain adaptation (Ruder & Plank, 2017) by preventing negative transfer (Rosenstein et al., 2005) from irrelevant data and noisy labels, meanwhile providing a more efficient learning process compared with other domain adaptation methods such as sample reweighing (Borgwardt et al., 2006), feature distribution matching (Tzeng et al., 2014) and representation learning (Csurka, 2017).
Although various data selection-based domain adaptation approaches were proposed for NLP tasks (III, 2007; Blitzer et al., 2007; Søgaard, 2011), most of them only consider scoring or ranking learning samples under a certain metric over the entire dataset, then select the top n (or a proportion, which is usually a predefined hyper-parameter) items to learn. However, such pre-designed metrics is not always able to cover effective characteristics for transferring domain knowledge and meanwhile suffers from their inflexibility. Even though some methods learn measures across domains and models (Ruder & Plank, 2017), their hyper-parameter setting, such as the n, still demands further explorations. Fortunately, recent studies confirmed that RL is an effective solution for data selection in training neural networks with improving their efficiency (Fan et al., 2017) and accuracy (Feng et al., 2018). A natural extension of using RL for data selection can be considered for the cross-domain scenario.
Yet, to apply RL to data selection for domain adaptation is not a straightforward mission as those RL application for in-domain tasks. To tackle this mission, in this paper, we propose a general data selection approach for domain adaptation via RL that jointly learns data representations and matches them from the source and target domain. There are two major components in our approach: a selection distribution generator (SDG) and a task-specific predictor with a feature extractor and a classifier1. The SDG and the predictor are pipelined by taking each others’ output as input and optimized accordingly via RL. As a result, in our approach, two major problems in domain adaptation are addressed: some redundant or noisy data across domains gives little or negative contribution to adaptation; useful information from the source domain is usually not properly organized and exploited for the target domain. Experimental results from three NLP tasks, namely, part-of-speech (POS) tagging, dependency parsing and sentiment analysis, illustrate that our approach achieve state-of-the-art performance, which confirm the validity and effectiveness of our approach.

heading: 2 THE APPROACH
text: Our approach is in the line of the unsupervised domain adaptation (Roark & Bacchiani, 2003), where one has no access to any labels of target domain data. Particularly, for a task T , we take labeled data instances for the task from a source domain DS as the input to our approach, and some unlabeled data from a target domain DT as the guidance. The approach is then expected to generate an optimal selected data set and a model trained on the set for T . An overview illustration of the entire architecture for our approach is demonstrated in Figure 1. In detail, for each step, the key component for selection, SDG, produces a distribution vector based on the representation of the selected source data from last step, then data instances are selected according to the vector and new reward is generated for next round data selection. To update the SDG, different measurements can be used to assess the discrepancy between the representations of the selected source data and the guidance set and then approximates the value function for updating. For the predictor, the loss from the classifier on the domain-specific task is used to guide the parameter updating for the feature extractor and the classifier. The details is unfolded in the following subsections, in which we firstly give the details of the two major components and then the joint training algorithm.

heading: 2.1 THE PREDICTOR
text: The predictor is the main component to perform a particular task T . In our approach we decompose the predictor into two parts, the feature extractor and a classifier, and use them separately. The feature extractor serves as the representation learning module that transform input data to dense representations, while the classifier takes the representations for T in the target domain. In doing so, the predictor is a neural model in this study so that the aforementioned separation can be done. Therefore, the feature extractor is normally the first n-1 layers of the predictor given that it has n layers in total. As a result, the classifier is the last layer dedicated for model output.
The Feature Extractor Data in its original form, especially as natural language, is usually difficult to be exploited in neural computation. The feature extractor is thus serving as a critical component in our approach to transform the data into distributed representations for their efficient use and computation. There are two-way inputs for the feature extractor. One is the guidance set XTg = {xT1 , xT2 , ..., xTm}, a collection of unlabeled data drawn from the target domain, serving as the reference for the selection process. The other input is the selected data from the source domain in a “data bag”, which is a batch of a certain amount of samples to facilitate the selection in this study. Let XS = {xS1 , xS2 , ..., xSn}, ∀xSi ∈ XS denote the data from the source domain, we uniformly partitions the entire data set into N disjoint data bags marked as {B1, B2, ..., BN}, with Bj = {xS(j−1)n/N+1, x S (j−1)n/N+2, ..., x S jn/N}, j ∈ {1, 2, ..., N}. Through the feature extractor, the outputs of the guidance set and the selected data are two collections of distribution vectors.2
The Classifier Once data selection is done, the classifier is combined with the feature extractor and trained on the selected data for a particular task. Note that the training is conducted in the source domain and the output of the classifier is used to generate gradients for back-propagation. Then the parameters of the classifier and the feature extractor are updated accordingly (with a learning rate β).

heading: 2.2 THE SELECTION DISTRIBUTION GENERATOR
text: A multi-layer perceptron (MLP) model is used as the SDG, which learns the selection policy optimized by the reward from the representations of the guidance set and the selected data by RL. In doing so, at each time step, the SDG is fed by a collection of data representations for a data bag from the feature extractor. We denote the collection as ΦBj = {r j 1, r j 2, ..., r j |Bj |}, where r j l
1It is not necessarily a classifier, e.g., such as a tagger. However we use the term classifier for simplicity. 2The formal description of such representations are given in the next section.
(l = 1, 2, ..., |Bj |) is the vector of the l-th sample3 in Bj .4 Then ΦBj is mapped by the SDG to a vector DBj = (p j 1, p j 2, ..., p j |Bj |), p j l (l = 1, 2, ..., |Bj |), which represents the probability for each sample that whether it should be selected individually. To learn the SDG, each ΦBj is measured with Φt to give the reward in an RL framework, which is described as follows.

heading: RL Components in Learning the SDG
text: • State (s1, s2, ...sj , ...sN ) includes a collection of states for all j with respect to N data bags, where each sj indicates a status including data bag B̂j and parameters of the feature extractor for the selected instances B̂j from Bj . For simplicity we use ΦB̂j and Φt to represent state sj . • Action For each state, the action space A is a 0-1 judgment to decide if selecting an instance (1) or not (0). An action a = {ak} |Bj | k=1 ∈ {0, 1}|Bj |, which is obtained from DB̂j . After each action, data
selection gives new ΦB̂j , then transforms state s into s ′. Thus we define the policy as PW(a|s).
• Reward Since the data selection process is to ensure that the selected data fitting the distribution of the target domain, we set a reward r(s, a, s′) to assess the distance between ΦB̂j and Φt in the current state (s′) and its previous state (s):
r(s, a, s′) = d(Φs B̂j−1 ,Φst )− γd(Φs ′ B̂j ,Φs ′ t ) (1)
where d(·, ·) is a distribution discrepancy measurement, which can be implemented by different information-bearing functions. γ ∈ (0, 1) is a discounting constant that decreases the impact from future distribution difference. Note that Eq. (1) is conducted in a sequence based on two adjacent data bags Bj−1 and Bj , of which Φs ′
B̂j are impacted by Φs B̂j−1 via parameters Ψ of the feature
extractor updated by B̂j−1. Consequently, the state transition probability P(s′|s, a) is determined by stochastic optimization and other randomness in training, e.g., dropout (Srivastava et al., 2014). When better instances are selected, the reward is then expected to produce a higher value because the measurement for the previous state d(Φs
B̂j−1 ,Φst ) supposes to give a larger distance between
ΦB̂j−1 and Φt than that for the current state.
Distribution Discrepancy Measurement To measure each B̂j and the XTg , let P be the elementwise average of ΦB̂j and Q the average of Φt, we use the following measurements for d(·, ·) in this paper: • JS: The Jensen-Shannon divergence (Lin, 1991). d(P,Q) = 12 [DKL(P ||M) + DKL(Q||M)]
where DKL(P ||Q) = ∑n i=1 pi log pi qi
, with M = 12 (P +Q). • MMD: The maximum mean discrepancy (Borgwardt et al., 2006). d(P,Q) = ‖P −Q‖. • RÉNYI: The symmetric Rényi divergence (Rényi, 1961). d(P,Q) = 12 [Ry(P,M) +Ry(Q,M)], Ry(P,Q) = 1α−1 log( ∑n i=1 pαi qα−1i ). We set α=0.99 following Van Asch & Daelemans (2010).
3Representations in the collection follow the same order of their corresponding data instances in the bag. 4Similarly, the representation collection of the guidance set is denoted as Φt.
Algorithm 1: Joint training algorithm of our approach Input: Epochs L; Training data B = {B1, B2, ..., BN}. W (SDG), Ψ (Predictor), Θ (feature extractor); Loss function of the predictor F (Ψ, B̂j); nJ ; d(·, ·); γ. Output: Updated W and Ψ (Θ). Initialize W and Ψ(Θ) with standard Gaussian distribution; for epoch l = 1 to L do
Σ = 0; for k = 1 to nJ do
Σr = 0; Shuffle {B1, B2, ..., BN}; for each Bj ∈ B do
Φ sj Bj ← Θj−1(Bj); Φ sj t ← Θj−1(XTg ); On current bag state sj , Dj ←W(Φ sj t ); select Φ sj
B̂j from ΦsjBj via Dj (take action aj);
r(sj−1, aj , sj)← d(Φ sj−1 B̂j−1 ,Φ sj−1 t )− γd(Φ sj Bj ,Φ sj t ) Σr ← Σr + γj−1r(sj−1, aj , sj) Ψ← Ψ− β∇ΨF (Ψ, B̂j); ( Θj ← Θj−1 − β∇ΘF (Ψ, B̂j) ) ;
end Σ← Σ + ∑N j=1∇W log πW(a k j |skj )Σr;
end ∇WJ̃(W)← 1
nJ Σ;
W←W + τ∇WJ̃(W); end
• LOSS: The guidance loss. d = − 1m ∑m i=1 ∑ yt∈Yt yt log pΦ(yt|x T i ) where yt is the label of
instance t from the guidance set, and pφ the learned conditional probability of the predictor. Note that, different from aforementioned measurements, LOSS requires labels from the target domain, thus is only set as a comparison to our approach with using annotated data through the predictor.
Optimization The following object is optimized to obtain the optimal distribution generation policy. J(W) = EPW(a|s)[ N∑
j=1
γj−1r(sj , aj)] (2)
Then the parameters of the SDG, i.e., W, is updated via policy gradient (Sutton et al., 1999) by W←W + τ∇WJ̃(W) (3) where τ is the discounting learning rate5, the gradient ∇WJ(W) is approximated by ∇WJ̃(W) = 1
nJ
∑nJ k=1 ∑N j=1∇W log πW(akj |skj ) ∑N j=1 γ j−1r(skj , a k j ), where j refers to the j-th step (corre-
sponding to the j-th data bag) in RL; i denotes the i-th selection process to estimate∇WJ(W ). Note that∇WJ(W ) is estimated after nJ times of selection over all N data bags, where nJ is a predefined hyper-parameter.

heading: 2.3 JOINT TRAINING
text: We jointly train the SDG and the predictor with policy gradient method (Sutton et al., 1999), which favors actions with high rewards from better selected samples. The entire joint training process is described in Algorithm 1. Following Feng et al. (2018), we choose the whole Bj for the predictor to compute r(sj−1, aj , sj) when B̂j = ∅ at time step j to help exclude noisy bags.

heading: 3 EXPERIMENT
text: To evaluate our approach, we conduct experiments on three typical NLP tasks: POS tagging, dependency parsing, and sentiment analysis. Details about our experiments are described as follows.

heading: 3.1 DATASETS
text: Two popular datasets are used in our experiments. For POS tagging and dependency parsing, we use the dataset from the SANCL 2012 shared task (Petrov & McDonald, 2012), with six different domains. For sentiment analysis, we use the product review dataset from Blitzer et al. (2006), with
5τ and the aforementioned β can be self-adapted by the optimizer, such as Adam (Kingma & Ba, 2014).
four domains. Note that for all datasets, there exists both labeled and unlabeled samples. The statistics and the domains for the aforementioned datasets are reported in Table 1.

heading: 3.2 COMMON SETTINGS
text: A major difference between our approach and other data selection methods is that the number of instances to be selected, n, is not fixed in our approach. Instead, it chooses the most effective one automatically. For fair comparison, we record the resulted n from our approach in different tasks and use it in other methods to guide their selection. In all experiments, the source domain includes all labeled data from the dataset except that for the target domain, i.e., we take turns selecting a domain as the target domain, and use the union of the rest as the source domain. N is set according to a uniform bag size of around 1K. For the guidance set, we follow Ruder & Plank (2017) and randomly select half of the instances from all the test data in the target domain discarding their labels.
Considering that the starting reward needs to be calculated from a reliable feature extractor, before the regular training, we pre-train the predictor on all source data for 2 epochs, then initialize parameters of SDG with Gaussian variables. Later the predictor and SDG follows ordinary updating paradigm in each training epoch. In all experiments, we use Adam (Kingma & Ba, 2014) as the optimizer, and set γ to 0.99 following Fan et al. (2017) and nJ to 3.All the deep learning based methods are configured with the same neural networks as our corresponding experiments.

heading: 3.3 POS TAGGING
text: The Predictor We use the Bi-LSTM tagger proposed in Plank et al. (2016) as the predictor.
Baselines Following Ruder & Plank (2017), we compare our approach to five baselines: 1) JS-E: top instances selected using Jensen-Shannon divergence. 2) JS-D: top instances selected from the most similar source domain, where the similarity between domains are determined by Jensen-Shannon divergence. 3) Bayesian optimization (Brochu et al., 2010) with the following settings: T-S, term distribution similarity; TO-S, topic distribution similarity; T+TO-S, joint term and topic distribution similarity; T-S+D, term distribution similarity and diversity; TO-S+D, topic distribution similarity and diversity. 4) MDAN: a recent multiple source domain adaptation with adversarial learning (Zhao et al., 2018). 5) RANDOM: a random selection model that selects the same number of instances according to our proposed approach. 6) ALL: The predictor is trained on all available source data.
Results POS tagging results are reported in Table 2. Overall, our approach with different distribution discrepancy metrics outperforms all baselines based on the same feature extractor. This observation demonstrates the excellent adaptability of our approach on this tasks with complicated structural variance in sentences. Among the four metrics, Rényi divergence achieve the best overall performance, which is slightly surpassed by MMD in the ANSWER and WEBLOGS domain. It is observed that the convergence of our approach results in around 50% of the source data selected.
3.4 DEPENDENCY PARSING
A EM N R WB WSJ
JS-E 81.02 80.53 83.25 84.66 85.36 82.43 JS-D 82.80 79.93 81.77 83.98 83.44 80.61 T-S 83.79 81.09 82.68 84.66 84.85 82.57 TO-S 82.87 81.43 82.07 83.98 84.98 82.90 T+TO-S 82.87 81.13 82.97 84.65 84.43 82.43 T-S+D 83.72 81.60 82.80 84.62 85.44 82.87 TO-S+D 82.60 80.83 84.04 84.45 85.89 82.33 MDAN 83.56 84.45 84.12 84.96 85.38 84.91
RANDOM 81.28 83.41 81.03 82.67 82.46 80.74 ALL 85.65 87.78 86.07 87.27 85.51 85.56
SDG (JS) 84.03 85.98 84.17 86.25 86.22 85.24 SDG (MMD) 84.19 86.25 84.87 86.80 85.57 84.37 SDG (RÉNYI) 84.55 85.11 85.27 86.93 85.65 85.79 SDG (LOSS) 83.97 85.86 84.05 86.21 86.03 84.98
Table 3: Dependency parsing results (LAS) for each domain.
The Predictor The Bi-LSTM parser proposed by Kiperwasser & Goldberg (2016) is the predictor.
Baselines For dependency parsing, we use the same baselines introduced in the POS tagging task.
Results The performance (labeled attachment scores, LAS) of dependency parsing is reported in Table 3. Similar to POS tagging, MDAN stand out among the baselines and the term distribution-based method (T-S) as well as its combination with diversity features (T-S+D) surpass other Bayesian optimization baselines, while our models outperform the measurementbased as well as neural network
based ones significantly in most domains. However, different from POS tagging, in this task, the predictor trained on the entire source data still performs the best on some domains, which can be explained by the complexity of the task. Yet, our models, e.g., the SDG (JS) and SDG (RÉNYI), outperform the ALL model in the last two domains, with only half of the source data used. 3.5 SENTIMENT ANALYSIS
The Predictor We use the CNN classifier proposed by Kim (2014) as the predictor in this task.
Baselines In addition to the baselines used in POS tagging and dependency parsing, we use a series of extra baselines from previous studies: 1) SCL, the structural correspondence learning proposed by Blitzer et al. (2006); 2) SST, the sentiment sensitive thesaurus method (Bollegala et al., 2011); 3) DAM, a general-purpose multiple source domain adaptation method proposed by Mansour et al. (2008); 4) SDAMS-LS and SDAMS-SVM, the specially designed sentiment domain adaptation approach (Wu & Huang, 2016) for multiple sources with square loss and hinge loss, respectively.
Results Table 4 presents the results of sentiment analysis. Similar to previous tasks, it is observed that our approach still performs competitively in this task, even though compared with the algorithms dedicatedly designed for sentiment analysis (e.g., SDAMS) and a general deep learning based multiple source domain adaptation method (MDAN). A potential reason for our weaker re-
sults on Electronics target domain is that dedicatedly designed methods include relation graphs among key words as prior knowledge, while our model aims for a wider application without task-specific consideration. Slightly different from previous tasks, in this task, around 40% source data are selected upon the convergence of our models.

heading: 3.6 DISCUSSION
text: In all three tasks, our approach achieve the best overall performance when there are half or less than half source data selected to train the predictor. The comparisons between our approach and the basic distribution measure-based methods, the general-purpose multiple source approach as well as the previous models (in sentiment analysis) across all tasks illustrate the effectiveness of our approach in selecting the most useful instances for the target domain while eliminating the noise. However,
domain variance still plays an important role affecting model performance, where it is also taskspecific. Compared to POS tagging and dependency parsing, in sentiment analysis there exists more significant bias across domains, e.g., “small” could be positive in one domain but negative in another.
Consequently, topic related domains express similar sentiment expressions. The investigation of the selected data indicates that our approach chooses more instances from the similar domains in sentiment analysis (e.g., BOOK ⇒ DVD), while the selected instances in POS tagging and dependency parsing are more balanced across domains. This observation suggests the effectiveness of our approach in adapting different tasks with the
most appropriate strategy. In addition, there still exist side effects while leveraging noise filtering and relevant instance selection, which can be observed from the slightly weaker results on Electronics domain in sentiment analysis and the fact that our approach is outperformed by training on all of the sources in parsing task. Such phenomenon implies that filtering some irrelevant instances may lose intrinsic beneficial information for target domain classification, and policy gradient method with partial data sometimes converges to a local optima given structured data with many indirect relations.

heading: 4 ABLATION STUDIES

heading: 4.1 PERFORMANCE AND EFFICIENCY OF DIFFERENT MODELS
text: To better understand the behavior of our model with different measurements, we investigate their performance using a case study on the DVD domain in sentiment analysis. We record the accuracy curves of different models with respect to their training epochs, as shown in Figure 2(a). In general, our models show similar performance and are significantly better than the RANDOM one. Interestingly, their curves are also similar to the ALL model but shows a much stable performance with the epoch increasing. This observation demonstrates that there exist severe noise when using all source data directly, while our models are able to overcome such limitation.
Another investigation is to study how much data are selected by different measurements in our approach. We display the number of instances selected by the four measurements in Figure 2(b), using the same domain and task setting as that in Figure 2(a). Overall our models with different measurements share similar behavior in selecting source data in terms of selection numbers. They tend to select more data at the beginning stage, i.e., before 10 epochs, then reduces selected instances to a smaller set and keeps the performance of the predictor (comparing with the curve in Figure 2(a)). Among all measurements, Rényi divergence tend to select less data instances while achieving a better performance when we match the numbers with the results reported in Table 4.

heading: 4.2 DISTRIBUTION VISUALIZATION
text: To better demonstrate the effect of our approach, we use the sentiment analysis for the DVD domain with SDG (RÉNYI) for visualized comparison among the distributions of instances in different scenarios. Following Tzeng et al. (2014), we plot in Figure 3 the t-SNE visualizations of the feature vectors learned from the feature extractor in four settings: vectors before training (learning with initialized weights) (Figure 3(a)), trained with SDG (Figure 3(b)), trained with JS-E (Figure 3(c)), and directly trained on all source data (Figure 3(d)). It is observed that, in original vectors, DVD and BOOKS are similar, while ELECTRONICS and KITCHEN are different from them as well as to each other. When trained with all source data, feature vectors could show some change in their distributions where instances from different domains are mixed and closer to the target domain. In the case where JS divergence is minimized for each domain, we can see a further mixture with closer representation matching. However, their domain adaptation capability is limited since the vectors are not optimized for the target domain. On the contrary, when trained with our approach, the selected instances result in a highly similar distribution as that in the target domain, as shown in Figure 3(b) the matched shape between the points in red and other colors, which is more beneficial than the structure in non-RL based embeddings (3(c)). Consider that it is a dynamic selection process in our
approach, the observation from Figure 3 confirms that such process ensures that the selected data could shape a target domain alike distribution without redundant instances.

heading: 5 RELATED WORK
text: Many studies have been conducted recently for domain adaptation with neural networks (Long et al., 2015; 2016; Shu et al., 2018; Shankar et al., 2018). Their methodologies include representation learning (Glorot et al., 2011; Chen et al., 2012; Baktashmotlagh et al., 2013; Zhao et al., 2018), reweighing samples from the source domain (Borgwardt et al., 2006; III, 2007; Gong et al., 2013), and seeking a way for explicit feature space transformation (Gopalan et al., 2011; Pan et al., 2011; Long et al., 2013), etc.
Normally, the transferable knowledge across domains are derived from some particular data, while others contribute less and are costly to be learned from (Axelrod et al., 2011; Ruder & Plank, 2017). Thus, previous studies conduct domain adaptation through selecting relative and informative source data according to the nature of the target domain, such as entropy-based source data selection (Song et al., 2012), Bayesian optimization based selection (Ruder & Plank, 2017), etc. Especially in various NLP tasks, Training data selection has been widely used, such as multilingual NER (Murthy et al., 2018), machine translation (Chen et al., 2016; van der Wees et al., 2017), and language model (Moore & Lewis, 2010). Consider that data selection in general is a combinatorics’ optimization problem with exponential complexity, it thus needs to be reduced to a series of sequential decisions, while RL provides a good solution in this case. For example, in Feng et al. (2018), they used sequential one-step actions for each single instance where every action is decided based on the previous one. In doing so, the selection is a consuming process where the complexity is related to the amount of the source data. Fan et al. (2017) sought to allocate appropriate training data at different training stages, which helps achieving comparative accuracy with less computational efforts compared with the model trained on the entire data.
Another line of research related to this work is the representation adaptation across domains (Mansour et al., 2008), which is the main strategy of domain adaptation for neural network models. Compared to the aforementioned studies, the proposed approach in this paper combines dynamic data selection and transferable representation learning in a unified RL framework, and is conducted in an effective way on data batches. Especially different from previous work that requires a predefined selection number, our approach not only ensures the selection of related instances in a dynamic manner, but also effectively controls negative transfer (Rosenstein et al., 2005) with limited noise.

heading: 6 CONCLUSION
text: In this paper, we proposed a general data selection framework for domain adaptation via reinforcement learning. In detail, the framework matches the representations of the selected data from the source domain and the guidance set from the target domain and pass their similarity in different time steps as rewards to guide the selection distribution generator. Through the generator, different instances from the source domain are selected to train the task-specific predictor. Experiments were conducted on POS tagging, dependency parsing, and sentiment analysis. Results from all tasks demonstrate that our approach outperforms different baselines, especially the one trained on all source data. Ablation study was also provided in analyzing the factors that affects the performance of our approach. Investigations on model convergence, selection numbers, as well as distribution visualizations confirm the validity and effectiveness of our approach.

Review summaries:
The paper aims to address issues with Domain Adaptation by using RL approaches. Domain Adaptation is an actively studied area in NLP research and so this paper is relevant and timely. This paper proposes and algorithm that is in line with work that aims at selecting data smartly when performing Domain Adaptation. The proposed algorithm learns representations for text in the source and target domains jointly. The proposed algorithm has two components i) a selection distribution generator (SDG) and ii) a task specific prediction for tasks being POS tagging, Dependency parsing and Sentiment Analysis.