Title: DOMAIN ADAPTATION VIA DISTRIBUTION AND REP-RESENTATION MATCHING: A CASE STUDY ON TRAINING DATA SELECTION VIA REINFORCEMENT LEARNING

Abstract: Supervised models suffer from domain shifting where distribution mismatch across domains greatly affect model performance. Particularly, noise scattered in each domain has played a crucial role in representing such distribution, especially in various natural language processing (NLP) tasks. In addressing this issue, training data selection (TDS) has been proven to be a prospective way to train supervised models with higher performance and efficiency. Following the TDS methodology, in this paper, we propose a general data selection framework with representation learning and distribution matching simultaneously for domain adaptation on neural models. In doing so, we formulate TDS as a novel selection process based on a learned selection distribution from the input data, which is produced by a trainable selection distribution generator (SDG) that is optimized by reinforcement learning (RL). Then, the model trained by the selected data not only predicts the target domain data in a specific task, but also provides input for the value function of the RL, which exactly helps update the parameters of SDG in a policy gradient way. Experiments are conducted on three typical NLP tasks, namely, part-of-speech tagging, dependency parsing, and sentiment analysis. Results demonstrate the validity and effectiveness of our approach.

1. INTRODUCTION:

Learning with massive data suffers from "Pyrrhic victory" where huge amounts of resource, e.g., computation, annotation, storage, etc., are consumed with many issues. One well-known issue for data-driven approaches is that data quality considerably affects the performance of learned models. Especially in NLP, such phenomenon is incredibly significant where noise and inaccurate annotations are demolishing models' robustness when applying them to various scenarios (Akopyan & Khashba, 2017; Zhang et al., 2016) , one of which is cross-domain NLP tasks. In this scenario, many data and their labels are considered noisy and inaccurate when they are used across domains, resulting in the mismatch of distribution between training and test data. To reduce the impact of the noise in data and annotations, reasonable selection of training samples is proved to be an useful strategy that can help supervised models achieving equivalent performance with less computational efforts (Fan et al., 2017; Feng et al., 2018) . Particularly, it is also confirmed that appropriate data selection strategy can benefit domain adaptation (Ruder & Plank, 2017) by preventing negative transfer (Rosenstein et al., 2005) from irrelevant data and noisy labels, meanwhile providing a more efficient learning process compared with other domain adaptation methods such as sample reweighing (Borgwardt et al., 2006) , feature distribution matching (Tzeng et al., 2014) and representation learning (Csurka, 2017) .

Although various data selection-based domain adaptation approaches were proposed for NLP tasks (III, 2007; Blitzer et al., 2007; Søgaard, 2011) , most of them only consider scoring or ranking learning samples under a certain metric over the entire dataset, then select the top n (or a proportion, which is usually a predefined hyper-parameter) items to learn. However, such pre-designed metrics is not always able to cover effective characteristics for transferring domain knowledge and meanwhile suffers from their inflexibility. Even though some methods learn measures across domains and models (Ruder & Plank, 2017) , their hyper-parameter setting, such as the n, still demands further explorations. Fortunately, recent studies confirmed that RL is an effective solution for data selection in training neural networks with improving their efficiency (Fan et al., 2017) and accuracy (Feng et al., 2018) . A natural extension of using RL for data selection can be considered for the cross-domain scenario.

Yet, to apply RL to data selection for domain adaptation is not a straightforward mission as those RL application for in-domain tasks. To tackle this mission, in this paper, we propose a general data selection approach for domain adaptation via RL that jointly learns data representations and matches them from the source and target domain. There are two major components in our approach: a selection distribution generator (SDG) and a task-specific predictor with a feature extractor and a classifier1 . The SDG and the predictor are pipelined by taking each others' output as input and optimized accordingly via RL. As a result, in our approach, two major problems in domain adaptation are addressed: some redundant or noisy data across domains gives little or negative contribution to adaptation; useful information from the source domain is usually not properly organized and exploited for the target domain. Experimental results from three NLP tasks, namely, part-of-speech (POS) tagging, dependency parsing and sentiment analysis, illustrate that our approach achieve state-of-the-art performance, which confirm the validity and effectiveness of our approach.

2. THE APPROACH:

Our approach is in the line of the unsupervised domain adaptation (Roark & Bacchiani, 2003) , where one has no access to any labels of target domain data. Particularly, for a task T , we take labeled data instances for the task from a source domain D S as the input to our approach, and some unlabeled data from a target domain D T as the guidance. The approach is then expected to generate an optimal selected data set and a model trained on the set for T . An overview illustration of the entire architecture for our approach is demonstrated in Figure 1 . In detail, for each step, the key component for selection, SDG, produces a distribution vector based on the representation of the selected source data from last step, then data instances are selected according to the vector and new reward is generated for next round data selection. To update the SDG, different measurements can be used to assess the discrepancy between the representations of the selected source data and the guidance set and then approximates the value function for updating. For the predictor, the loss from the classifier on the domain-specific task is used to guide the parameter updating for the feature extractor and the classifier. The details is unfolded in the following subsections, in which we firstly give the details of the two major components and then the joint training algorithm.

2.1. THE PREDICTOR:

The predictor is the main component to perform a particular task T . In our approach we decompose the predictor into two parts, the feature extractor and a classifier, and use them separately. The feature extractor serves as the representation learning module that transform input data to dense representations, while the classifier takes the representations for T in the target domain. In doing so, the predictor is a neural model in this study so that the aforementioned separation can be done. Therefore, the feature extractor is normally the first n-1 layers of the predictor given that it has n layers in total. As a result, the classifier is the last layer dedicated for model output.

The Feature Extractor Data in its original form, especially as natural language, is usually difficult to be exploited in neural computation. The feature extractor is thus serving as a critical component in our approach to transform the data into distributed representations for their efficient use and computation. There are two-way inputs for the feature extractor. One is the guidance set X T g = {x T

1 , x T 2 , ..., x T m }, a collection of unlabeled data drawn from the target domain, serving as the reference for the selection process. The other input is the selected data from the source domain in a "data bag", which is a batch of a certain amount of samples to facilitate the selection in this study. Let X S = {x S 1 , x S 2 , ..., x S n }, ∀x S i ∈ X S denote the data from the source domain, we uniformly partitions the entire data set into N disjoint data bags marked as {B 1 , B 2 , ..., B N }, with B j = {x S (j-1)n/N +1 , x S (j-1)n/N +2 , ..., x S jn/N }, j ∈ {1, 2, ..., N }. Through the feature extractor, the outputs of the guidance set and the selected data are two collections of distribution vectors. 2The Classifier Once data selection is done, the classifier is combined with the feature extractor and trained on the selected data for a particular task. Note that the training is conducted in the source domain and the output of the classifier is used to generate gradients for back-propagation. Then the parameters of the classifier and the feature extractor are updated accordingly (with a learning rate β).

2.2. THE SELECTION DISTRIBUTION GENERATOR:

A multi-layer perceptron (MLP) model is used as the SDG, which learns the selection policy optimized by the reward from the representations of the guidance set and the selected data by RL. In doing so, at each time step, the SDG is fed by a collection of data representations for a data bag from the feature extractor. We denote the collection as Φ Bj = {r j 1 , r j 2 , ..., r j |Bj | }, where r j l (l = 1, 2, ..., |B j |) is the vector of the l-th sample3 in B j . 4 Then Φ Bj is mapped by the SDG to a vector D Bj = (p j 1 , p j 2 , ..., p j |Bj | ), p j l (l = 1, 2, ..., |B j |), which represents the probability for each sample that whether it should be selected individually. To learn the SDG, each Φ Bj is measured with Φ t to give the reward in an RL framework, which is described as follows.

RL Components in Learning the SDG:

• State (s 1 , s 2 , ...s j , ...s N ) includes a collection of states for all j with respect to N data bags, where each s j indicates a status including data bag Bj and parameters of the feature extractor for the selected instances Bj from B j . For simplicity we use Φ Bj and Φ t to represent state s j . • Action For each state, the action space A is a 0-1 judgment to decide if selecting an instance (1) or not ( 0). An action

a = {a k } |Bj | k=1 ∈ {0, 1} |Bj | , which is obtained from D Bj .

After each action, data selection gives new Φ Bj , then transforms state s into s . Thus we define the policy as P W (a|s).

• Reward Since the data selection process is to ensure that the selected data fitting the distribution of the target domain, we set a reward r(s, a, s ) to assess the distance between Φ Bj and Φ t in the current state (s ) and its previous state (s):

EQUATION

where d(•, •) is a distribution discrepancy measurement, which can be implemented by different information-bearing functions. γ ∈ (0, 1) is a discounting constant that decreases the impact from future distribution difference. Note that Eq. ( 1) is conducted in a sequence based on two adjacent data bags B j-1 and B j , of which Φ s Bj are impacted by Φ s Bj-1 via parameters Ψ of the feature extractor updated by Bj-1 . Consequently, the state transition probability P(s |s, a) is determined by stochastic optimization and other randomness in training, e.g., dropout (Srivastava et al., 2014) .

When better instances are selected, the reward is then expected to produce a higher value because the measurement for the previous state d(Φ s Bj-1

, Φ s t ) supposes to give a larger distance between Φ Bj-1 and Φ t than that for the current state.

Distribution Discrepancy Measurement To measure each Bj and the X T g , let P be the elementwise average of Φ Bj and Q the average of Φ t , we use the following measurements for d(•, •) in this paper:

• JS: The Jensen-Shannon divergence (Lin, 1991) .

d(P, Q) = 1 2 [D KL (P ||M ) + D KL (Q||M )] where D KL (P ||Q) = n i=1 p i log pi qi , with M = 1 2 (P + Q).

• MMD: The maximum mean discrepancy (Borgwardt et al., 2006) . (Rényi, 1961) .

d(P, Q) = P -Q . • R ÉNYI: The symmetric Rényi divergence

d(P, Q) = 1 2 [Ry(P, M ) + Ry(Q, M )], Ry(P, Q) = 1 α-1 log( n i=1 p α i q α-1 i

). We set α=0.99 following Van Asch & Daelemans (2010) . for each Bj ∈ B do Φ

s j B j ← Θj-1(Bj); Φ s j t ← Θj-1(X T g ); On current bag state sj, Dj ← W(Φ s j t ); select Φ s j Bj from Φ s j B j via Dj (take action aj); r(sj-1, aj, sj) ← d(Φ s j-1 Bj-1 , Φ s j-1 t ) -γd(Φ s j B j , Φ s j t ) Σr ← Σr + γ j-1 r(sj-1, aj, sj) Ψ ← Ψ -β∇ Ψ F (Ψ, Bj); ( Θj ← Θj-1 -β∇ Θ F (Ψ, Bj) ) ; end Σ ← Σ + N j=1 ∇ W log π W (a k j |s k j )Σr; end ∇ W J(W) ← 1 nJ Σ; W ← W + τ ∇ W J(W); end • LOSS: The guidance loss. d = -1 m m i=1 yt∈Yt y t log p Φ (y t |x T i )

where y t is the label of instance t from the guidance set, and p φ the learned conditional probability of the predictor. Note that, different from aforementioned measurements, LOSS requires labels from the target domain, thus is only set as a comparison to our approach with using annotated data through the predictor. Optimization The following object is optimized to obtain the optimal distribution generation policy.

J(W) = E P W (a|s) [ N j=1 γ j-1 r(sj, aj)]

(2)

Then the parameters of the SDG, i.e., W, is updated via policy gradient (Sutton et al., 1999) by

W ← W + τ ∇ W J(W) (3)

where τ is the discounting learning rate5 , the gradient

∇ W J(W) is approximated by ∇ W J(W) = 1 n J n J k=1 N j=1 ∇ W log π W (a k j |s k j ) N j=1 γ j-1 r(s k j , a k j )

, where j refers to the j-th step (corresponding to the j-th data bag) in RL; i denotes the i-th selection process to estimate ∇ W J(W ). Note that ∇ W J(W ) is estimated after n J times of selection over all N data bags, where n J is a predefined hyper-parameter.

2.3. JOINT TRAINING:

We jointly train the SDG and the predictor with policy gradient method (Sutton et al., 1999) , which favors actions with high rewards from better selected samples. The entire joint training process is described in Algorithm 1. Following Feng et al. (2018) , we choose the whole B j for the predictor to compute r(s j-1 , a j , s j ) when Bj = ∅ at time step j to help exclude noisy bags.

3. EXPERIMENT:

To evaluate our approach, we conduct experiments on three typical NLP tasks: POS tagging, dependency parsing, and sentiment analysis. Details about our experiments are described as follows.

3.1. DATASETS:

Two popular datasets are used in our experiments. For POS tagging and dependency parsing, we use the dataset from the SANCL 2012 shared task (Petrov & McDonald, 2012) , with six different domains. For sentiment analysis, we use the product review dataset from Blitzer et al. (2006) , with four domains. Note that for all datasets, there exists both labeled and unlabeled samples. The statistics and the domains for the aforementioned datasets are reported in Table 1 .

3.2. COMMON SETTINGS:

A major difference between our approach and other data selection methods is that the number of instances to be selected, n, is not fixed in our approach. Instead, it chooses the most effective one automatically. For fair comparison, we record the resulted n from our approach in different tasks and use it in other methods to guide their selection. In all experiments, the source domain includes all labeled data from the dataset except that for the target domain, i.e., we take turns selecting a domain as the target domain, and use the union of the rest as the source domain. N is set according to a uniform bag size of around 1K. For the guidance set, we follow Ruder & Plank (2017) and randomly select half of the instances from all the test data in the target domain discarding their labels. 

3.3. POS TAGGING:

The Predictor We use the Bi-LSTM tagger proposed in Plank et al. (2016) as the predictor.

Baselines Following Ruder & Plank (2017), we compare our approach to five baselines: 1) JS-E: top instances selected using Jensen-Shannon divergence. 2) JS-D: top instances selected from the most similar source domain, where the similarity between domains are determined by Jensen-Shannon divergence. 3) Bayesian optimization (Brochu et al., 2010) with the following settings: T-S, term distribution similarity; TO-S, topic distribution similarity; T+TO-S, joint term and topic distribution similarity; T-S+D, term distribution similarity and diversity; TO-S+D, topic distribution similarity and diversity. 4) MDAN: a recent multiple source domain adaptation with adversarial learning (Zhao et al., 2018) . 5) RANDOM: a random selection model that selects the same number of instances according to our proposed approach. 6) ALL: The predictor is trained on all available source data.

Results POS tagging results are reported in Table 2 . Overall, our approach with different distribution discrepancy metrics outperforms all baselines based on the same feature extractor. This observation demonstrates the excellent adaptability of our approach on this tasks with complicated structural variance in sentences. Among the four metrics, Rényi divergence achieve the best overall performance, which is slightly surpassed by MMD in the ANSWER and WEBLOGS domain. It is observed that the convergence of our approach results in around 50% of the source data selected. The Predictor The Bi-LSTM parser proposed by Kiperwasser & Goldberg (2016) is the predictor.

3.4. DEPENDENCY PARSING:

Baselines For dependency parsing, we use the same baselines introduced in the POS tagging task.

Results:

The performance (labeled attachment scores, LAS) of dependency parsing is reported in The Predictor We use the CNN classifier proposed by Kim (2014) as the predictor in this task.

Baselines In addition to the baselines used in POS tagging and dependency parsing, we use a series of extra baselines from previous studies: 1) SCL, the structural correspondence learning proposed by Blitzer et al. (2006) ; 2) SST, the sentiment sensitive thesaurus method (Bollegala et al., 2011) ; 3) DAM, a general-purpose multiple source domain adaptation method proposed by Mansour et al. (2008) ; 4) SDAMS-LS and SDAMS-SVM, the specially designed sentiment domain adaptation approach (Wu & Huang, 2016) for multiple sources with square loss and hinge loss, respectively.

Results Table 4 presents the results of sentiment analysis. Similar to previous tasks, it is observed that our approach still performs competitively in this task, even though compared with the algorithms dedicatedly designed for sentiment analysis (e.g., SDAMS) and a general deep learning based multiple source domain adaptation method (MDAN). A potential reason for our weaker results on Electronics target domain is that dedicatedly designed methods include relation graphs among key words as prior knowledge, while our model aims for a wider application without task-specific consideration. Slightly different from previous tasks, in this task, around 40% source data are selected upon the convergence of our models.

3.6. DISCUSSION:

In all three tasks, our approach achieve the best overall performance when there are half or less than half source data selected to train the predictor. The comparisons between our approach and the basic distribution measure-based methods, the general-purpose multiple source approach as well as the previous models (in sentiment analysis) across all tasks illustrate the effectiveness of our approach in selecting the most useful instances for the target domain while eliminating the noise. However, domain variance still plays an important role affecting model performance, where it is also taskspecific. Compared to POS tagging and dependency parsing, in sentiment analysis there exists more significant bias across domains, e.g., "small" could be positive in one domain but negative in another. Consequently, topic related domains express similar sentiment expressions. The investigation of the selected data indicates that our approach chooses more instances from the similar domains in sentiment analysis (e.g., BOOK ⇒ DVD), while the selected instances in POS tagging and dependency parsing are more balanced across domains. This observation suggests the effectiveness of our approach in adapting different tasks with the most appropriate strategy. In addition, there still exist side effects while leveraging noise filtering and relevant instance selection, which can be observed from the slightly weaker results on Electronics domain in sentiment analysis and the fact that our approach is outperformed by training on all of the sources in parsing task. Such phenomenon implies that filtering some irrelevant instances may lose intrinsic beneficial information for target domain classification, and policy gradient method with partial data sometimes converges to a local optima given structured data with many indirect relations.

4. ABLATION STUDIES:



4.1. PERFORMANCE AND EFFICIENCY OF DIFFERENT MODELS:

To better understand the behavior of our model with different measurements, we investigate their performance using a case study on the DVD domain in sentiment analysis. We record the accuracy curves of different models with respect to their training epochs, as shown in Figure 2 (a). In general, our models show similar performance and are significantly better than the RANDOM one. Interestingly, their curves are also similar to the ALL model but shows a much stable performance with the epoch increasing. This observation demonstrates that there exist severe noise when using all source data directly, while our models are able to overcome such limitation.

Another investigation is to study how much data are selected by different measurements in our approach. We display the number of instances selected by the four measurements in Figure 2 (b), using the same domain and task setting as that in Figure 2 (a). Overall our models with different measurements share similar behavior in selecting source data in terms of selection numbers. They tend to select more data at the beginning stage, i.e., before 10 epochs, then reduces selected instances to a smaller set and keeps the performance of the predictor (comparing with the curve in Figure 2(a) ). Among all measurements, Rényi divergence tend to select less data instances while achieving a better performance when we match the numbers with the results reported in Table 4 .

4.2. DISTRIBUTION VISUALIZATION:

To better demonstrate the effect of our approach, we use the sentiment analysis for the DVD domain with SDG (R ÉNYI) for visualized comparison among the distributions of instances in different scenarios. Following Tzeng et al. (2014) , we plot in Figure 3 When trained with all source data, feature vectors could show some change in their distributions where instances from different domains are mixed and closer to the target domain. In the case where JS divergence is minimized for each domain, we can see a further mixture with closer representation matching. However, their domain adaptation capability is limited since the vectors are not optimized for the target domain. On the contrary, when trained with our approach, the selected instances result in a highly similar distribution as that in the target domain, as shown in Figure 3 (b) the matched shape between the points in red and other colors, which is more beneficial than the structure in non-RL based embeddings (3(c)). Consider that it is a dynamic selection process in our approach, the observation from Figure 3 confirms that such process ensures that the selected data could shape a target domain alike distribution without redundant instances.

5. RELATED WORK:

Many studies have been conducted recently for domain adaptation with neural networks (Long et al., 2015; 2016; Shu et al., 2018; Shankar et al., 2018) . Their methodologies include representation learning (Glorot et al., 2011; Chen et al., 2012; Baktashmotlagh et al., 2013; Zhao et al., 2018) , reweighing samples from the source domain (Borgwardt et al., 2006; III, 2007; Gong et al., 2013) , and seeking a way for explicit feature space transformation (Gopalan et al., 2011; Pan et al., 2011; Long et al., 2013) , etc.

Normally, the transferable knowledge across domains are derived from some particular data, while others contribute less and are costly to be learned from (Axelrod et al., 2011; Ruder & Plank, 2017) . Thus, previous studies conduct domain adaptation through selecting relative and informative source data according to the nature of the target domain, such as entropy-based source data selection (Song et al., 2012) , Bayesian optimization based selection (Ruder & Plank, 2017) , etc. Especially in various NLP tasks, Training data selection has been widely used, such as multilingual NER (Murthy et al., 2018) , machine translation (Chen et al., 2016; van der Wees et al., 2017) , and language model (Moore & Lewis, 2010) . Consider that data selection in general is a combinatorics' optimization problem with exponential complexity, it thus needs to be reduced to a series of sequential decisions, while RL provides a good solution in this case. For example, in Feng et al. (2018) , they used sequential one-step actions for each single instance where every action is decided based on the previous one.

In doing so, the selection is a consuming process where the complexity is related to the amount of the source data. Fan et al. (2017) sought to allocate appropriate training data at different training stages, which helps achieving comparative accuracy with less computational efforts compared with the model trained on the entire data.

Another line of research related to this work is the representation adaptation across domains (Mansour et al., 2008) , which is the main strategy of domain adaptation for neural network models. Compared to the aforementioned studies, the proposed approach in this paper combines dynamic data selection and transferable representation learning in a unified RL framework, and is conducted in an effective way on data batches. Especially different from previous work that requires a predefined selection number, our approach not only ensures the selection of related instances in a dynamic manner, but also effectively controls negative transfer (Rosenstein et al., 2005) with limited noise.

6. CONCLUSION:

In this paper, we proposed a general data selection framework for domain adaptation via reinforcement learning. In detail, the framework matches the representations of the selected data from the source domain and the guidance set from the target domain and pass their similarity in different time steps as rewards to guide the selection distribution generator. Through the generator, different instances from the source domain are selected to train the task-specific predictor. Experiments were conducted on POS tagging, dependency parsing, and sentiment analysis. Results from all tasks demonstrate that our approach outperforms different baselines, especially the one trained on all source data. Ablation study was also provided in analyzing the factors that affects the performance of our approach. Investigations on model convergence, selection numbers, as well as distribution visualizations confirm the validity and effectiveness of our approach.

:

It is not necessarily a classifier, e.g., such as a tagger. However we use the term classifier for simplicity.

The formal description of such representations are given in the next section.

Representations in the collection follow the same order of their corresponding data instances in the bag.

Similarly, the representation collection of the guidance set is denoted as Φt.

τ and the aforementioned β can be self-adapted by the optimizer, such as Adam(Kingma & Ba, 2014).

Figures and Tables:

Figure 1: The architecture of a predictor with a selection distribution generator. All black solid arrows refer to data flow, while the red dashed arrow denotes reward and the orange dotted arrows indicating back-propagation of gradients from training the predictor.

Joint training algorithm of our approach Input: Epochs L; Training data B = {B1, B2, ..., BN }. W (SDG), Ψ (Predictor), Θ (feature extractor); Loss function of the predictor F (Ψ, Bj); nJ ; d(•, •); γ. Output: Updated W and Ψ (Θ). Initialize W and Ψ(Θ) with standard Gaussian distribution; for epoch l = 1 to L do Σ = 0; for k = 1 to nJ do Σr = 0; Shuffle {B1, B2, ..., BN };

(a) Accuracies of different models against training epochs. (b) % of data selected by different models against training epochs.

Figure 2: Investigation of using different models on the DVD domain for sentiment classification.

the t-SNE visualizations of the feature vectors learned from the feature extractor in four settings: vectors before training (learning with initialized weights) (Figure 3(a)), trained with SDG (Figure 3(b)), trained with JS-E (Figure 3(c)), and directly trained on all source data (Figure 3(d)). It is observed that, in original vectors, DVD and BOOKS are similar, while ELECTRONICS and KITCHEN are different from them as well as to each other.

Figure 3: t-SNE visualization of learned features for sentiment analysis on the DVD domain in different settings. Red cross, blue triangle, green star, and orange circle represent DVD, BOOKS, ELECTRONICS, and KITCHEN (best viewed in color with enlarged figures).

The statistics of all datasets used in our experiments, with the number presenting labeled (LBL.) or unlabeled (UNL.) samples in each domain. The domain abbreviations in different tasks are explained as follows. A:Answer, EM:Email, N:News, R:Reviews, WB:Weblogs, WSJ:Wall Street Journal, and B:Book, D:DVD, K:Kitchen, E:Electronics.

Table content: <table><tr><td>TASK</td><td/><td colspan="4">POS TAGGING/DEPENDENCY PARSING</td><td/><td colspan="3">SENTIMENT ANALYSIS</td></tr><tr><td>DOMAIN</td><td>A</td><td>EM</td><td>N</td><td>R</td><td colspan="2">WB WSJ</td><td>B</td><td>D</td><td>K</td><td>E</td></tr><tr><td>LBL.</td><td>3.5K</td><td>4.9K</td><td>2.4K</td><td>3.8K</td><td colspan="2">2.0K 3.0K</td><td>2K</td><td>2K</td><td>2K</td><td>2K</td></tr><tr><td>UNL.</td><td colspan="5">27K 1,194K 1,000K 1,965K 525K</td><td colspan="5">30K 4.5K 3.6K 5.7K 5.9K</td></tr></table>

Dependency parsing results (LAS) for each domain.

Table content: <table/>

in most domains. However, different from POS tagging, in this task, the predictor trained on the entire source data still performs the best on some domains, which can be explained by the complexity of the task. Yet, our models, e.g., the SDG (JS) and SDG (R ÉNYI), outperform the ALL model in the last two domains, with only half of the source data used.

Table content: <table><tr><td>. Similar to POS tagging, MDAN</td><td/><td/><td/></tr><tr><td>stand out among the baselines and</td><td/><td/><td/></tr><tr><td>the term distribution-based method</td><td/><td/><td/></tr><tr><td>(T-S) as well as its combination</td><td/><td/><td/></tr><tr><td>with diversity features (T-S+D)</td><td/><td/><td/></tr><tr><td>surpass other Bayesian optimiza-</td><td/><td/><td/></tr><tr><td>tion baselines, while our mod-</td><td/><td/><td/></tr><tr><td>els outperform the measurement-</td><td/><td/><td/></tr><tr><td>based as well as neural network</td><td/><td/><td/></tr><tr><td>based ones significantly 3.5 SENTIMENT ANALYSIS</td><td/><td/><td/></tr><tr><td/><td>B</td><td>D</td><td>E</td><td>K</td></tr><tr><td>JS-E</td><td colspan="4">72.49 68.21 76.78 77.54</td></tr><tr><td>JS-D</td><td colspan="4">75.28 73.75 72.53 80.05</td></tr><tr><td>T-S</td><td colspan="4">75.39 76.27 81.91 83.41</td></tr><tr><td>TO-S</td><td colspan="4">76.07 75.92 81.69 83.06</td></tr><tr><td>T+TO-S</td><td colspan="4">75.75 76.62 81.74 83.39</td></tr><tr><td>T-S+D</td><td colspan="4">76.20 77.60 82.66 84.98</td></tr><tr><td>TO-S+D</td><td colspan="4">77.16 79.00 81.92 84.29</td></tr><tr><td>MDAN</td><td colspan="4">78.65 80.72 84.46 86.04</td></tr><tr><td>RANDOM</td><td colspan="4">76.78 75.28 78.25 82.27</td></tr><tr><td>ALL</td><td colspan="4">78.48 79.68 80.58 84.50</td></tr><tr><td>SCL</td><td colspan="4">74.57 76.30 78.93 82.07</td></tr><tr><td>SST</td><td colspan="4">76.32 78.77 83.57 85.19</td></tr><tr><td>DAM</td><td colspan="4">75.61 77.57 82.79 84.23</td></tr><tr><td>SDAMS-LS</td><td colspan="4">77.95 78.80 83.98 85.96</td></tr><tr><td colspan="5">SDAMS-SVM 77.86 79.02 84.18 85.78</td></tr><tr><td>SDG (JS)</td><td colspan="4">79.37 81.06 82.38 85.78</td></tr><tr><td>SDG (MMD)</td><td colspan="4">79.57 81.08 82.68 85.69</td></tr><tr><td>SDG (R ÉNYI)</td><td colspan="4">80.07 82.07 82.28 86.18</td></tr><tr><td>SDG (LOSS)</td><td colspan="4">79.57 80.58 81.88 85.08</td></tr></table>

Sentiment analysis results (accuracy %).

Table content: <table/>

Peer review (iclr2019):
 Response to author comments: Unfortunately I am still significantly unclear on why RL is useful here. The author response attempts to clarify that by pointing me to paragraph 2 of the intro, which states that RL has been used for data selection in other settings in the past. What would help me (and I believe, the paper) more is a reason why greedy selection isn*t sufficient for this particular problem. Even just a single motivating example would be extremely helpful. R3 mentioned similar concerns in their review, saying that the paper lacks explanation for why RL would win over non-RL for e.g. sentiment analysis. Likewise, while I appreciate the authors comparing against a stronger baseline in Figure 3, I don*t know how to interpret the figure. Why is Figure 3(b) better than Figure 3(c), and why does using RL cause that difference to arise? Original review: Domain adaptation is an interesting task, and new methods for it would be welcome. This paper appears to have technical depth and the experimental results are promising. However, the presented approach is complex, and I found it very hard to understand -- both in terms of how exactly it works, and in terms of why the chosen techniques were chosen. More detail on my questions and confusions follows. First, I never understood the motivation for using RL here. If minimizing the distance between selected data from the source domain and data in the target domain is the objective (equation 1), how does RL help? The reward seems like it is immediate in each time step. How does the *order* in which we add source examples to our collection matter? I never understood the crucial difference that made the RL approach outperform the baselines that just select examples that minimize e.g. JS divergence. Neither the paper*s discussion of motivation nor the experimental analysis clarifies this. The paper says in Section 2.1 that a formal description of the representations is to follow. I didn*t see this description (I do not see a formal definition of how the feature extractor works, and e.g. how it produces vectors that are *distributions* that can be used within e.g. JS divergence). The paper also says it follows (Ruder and Plank, 2017) in using JS as a baseline, but as I understand that work the JS baseline is computed over words, not learned representations. What is done in the submission, is the JS baseline over words in the instances, or the representations from the feature extractor? What is the reason for partitioning the source data into disjoint "data bags"? Why not just select the best source domain examples (from among all the source data) using RL? The experiments are generally over enough tasks and compare against several baselines, and although the empirical wins are not that large I feel that they would be sufficient for publication if not for my other concerns. The analysis (sec 4) did not make it clear to me why the RL approach works. The visualization in Figure 3 only contrasts the proposed approach with a weak baseline of selecting all source data -- what we really need to see is an analysis that reveals why the learning of a policy with RL is better than simply greedily minimizing JS for each source data selection, for up to some limit of n selections. Minor The paper has a number of typos The citations in the paper are mis-formatted -- seem to use shortcite where they shouldn*t (e.g. "scenarios Akopyan and Khashba (2017)" should be "scenarios (Akopyan and Khashba, 2017)"). When the policy .pi_w(a | s) is introduced at the start of Sec 2.2, it uses symbols (a, s) that have not been defined, also that policy variable is not really utilized in the text so it could be deleted.