Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: It is proposed to speed up the existing SDA methods by selecting a subset of the source dataset that is highly "similar" to the target dataset and training only using the source subset. Since model training is performed on smaller set (given budget), the computational effort is lessened. Also, since the subset is more aligned to the target SDA is explicitly performed.

Subset similarity is measured through a SMI, with pairwise data similarity defined as cosine similarity of the gradient of loss at the points wrt. the model parameter. 

The overall algorithm alternates between loss optimization (training/learning) and optimal subset selection under SMI. 

Using simulations on benchmarks it is argued that computational efficiency is achieved without much compromise on accuracy.
strengths_and_weaknesses: The basic strategy of restricting training to merely a subset inherently seems to have some limitations:

1. The valuable information in the complement of the selected subset is completely ignored.
2. The extent of complement's usefulness is not only determined by its similarity to the target set. Perhaps, the complement can be used in other subtle ways: for example training domain-invariant features etc.
However, such aspects and corresponding trade-off have not been discussed.

Since the proposed methodology explicitly performs SDA, it seems very critical to empirically compare against few state-of-the-art SDA approaches. Especially, those which are scalable. However, the simulations do not seem to compare even wrt one SDA method. the baselines are simply other alternative subset methods like random/core, and training with full dataset (which is also NOT an SDA method). Without these critical comparisons the practical usefulness of the method cannot be established.

----------
rebuttal resolved some concerns. thanks.
questions: 1. How does the initialized subset effect the final accuracy? Can this be experimentally studied via simulations?

2. Since the initialization of subset is random, reporting mean and std div over multiple initializations seems a better way of presenting the results.

limitations: Memory limitation is mentioned. However some of the above written limitations can be discussed in more detail.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: For SDA (Supervised Domain Adaptation), authors propose a framework to select a subset from source domain data so that the training process can be speeded up and better performance can be achieved.
strengths_and_weaknesses: Strengths:

- The problem setting is novel, interesting and realistic. It is great to see ideas focusing on utilizing data instead of building fancy models which provides marginal improvement of performance. 

- Comprehensive experiments are conducted to demonstrate the effectiveness of this framework. 

- The paper is well-written and presented. 

Weaknesses:

- The calculation of the similarity matrix is quite straightforward. Can it be learned or calculated more properly?
questions: Can this framework be extended to UDA?

limitations: Stated in weaknesses.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors propose to employ submodular mutual information measures for dataset selection in domain adaptation. Starting with a random subset of data, the proposed algorithm iterates between training on the currently selected dataset and selecting a new dataset based on submodular measure and gradient similarity. Authors claim that the proposed algorithm is much more efficient than full-data training. When combined with d-SNE, ORIENT outperforms full-data training baseline on three adaptation directions and underperforms on three directions, out of 12 directions.
strengths_and_weaknesses: Quality, Significance: The paper builds upon well-established techniques such as submodular mutual information and submodular optimization, which makes the proposed method principled. For example, this allows the selection part of the algorithm to borrow well-known algorithms in submodular optimization literature, and provide approximation guarantee. The proposed framework is also quite general, which will allow future researchers to implement novel algorithms based on it, for example by proposing new similarity measures or mutual information metrics.

Clarity: Authors employ the abstract framework of submodular mutual information, which allows authors to describe their proposed method in high-level. This makes the narrative of the paper quite clear, but this also makes the paper a bit superficial in some of the technical details. For example, it is unclear how four SMIs proposed in this paper have advantages and disadvantages against others. Even intuitive ideas behind these SMIs are not discussed in the main body of the paper. Neither does experiment section discuss how SMIs compare against each other.

Quality: The main contribution of this paper is mostly empirical, as SMIs have been established in prior papers. However, I am not sure  empirical results are very convincing. In Figure 3, Full data selection seems to be quite competitive in most of the adaptation directions even from the first evaluation. Also, there seems to be quite a bit of random errors in this plot, which makes me not sure we can conclude anything other than ORIENT (G), Random, and CRAIG being much worse than others. Results in Table 2 & 3 are mixed. In Table 2, Full outperforms ORIENT 3/6 directions, and ties with ORIENT on 2/6. In table 3, FULL outperforms on 3/12, and ORIENT outperforms on 3/12. Hence we can't really say ORIENT performs better. The only benefit is the reduction of training time, but it is unclear how authors computed the speedup. Therefore, I am not sure ORIENT would be actually computationally more efficient than FULL. 
questions: How do authors compute speedup in Figure 4? I am suspecting authors are fixing the number of passes on the dataset, because speed-up numbers in Figure 4 are very uniform. If so, then I am not sure it is fair to say ORIENT achieves 3x speed-up. FULL could've achieved performance similar with ORIENT when the number of gradient descent steps are the same. Therefore, speed-up should be computed based on how much time it takes to achieve a target performance level.

Can authors provide some discussion on what are main ideas behind proposed SMIs, why they were chosen, what are their conceptual pros & cons, and how do experimental results validate/invalidate these intuitions & assumptions?

How should the size of selected dataset chosen? Can we benefit from more or less frequent dataset selection than authors currently do?

Shouldn't authors compare against dataset selection baselines? Authors' main claim is that their proposed SDA algorithms provide 3x speedup, but maybe we can achieve 3x speedup alternatively by just employing dataset selection on the full data without considering alignment with target domain data.
limitations: Domain Adaptation via Pseudo In-Domain Data Selection (Axelrod et al, https://aclanthology.org/D11-1033.pdf ) should've been cited, as this paper proposes data selection for domain adaptation. They don't use submodular functions for selection, and they rely on unsupervised model (language model) for selection, so it'd be difficult for authors to compare against Axelrod et al. However, such an experiment would certainly make the paper stronger, by including additional task (machine translation) and comparing against non-submodular baseline (authors don't compare against any non-submodular baseline).

In line 118, authors do not assume monotonicity of submodular function when discussing (1-1/e) approximation guarantee of simple greedy algorithm.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces ORIENT, a novel subset selection framework to speed up learning in supervised domain adaptation (SDA). Based on submodular mutual information (SMI) functions, ORIENT selects a small diverse subset of training data in the source domain that is similar to data in the target domain. The authors show that ORIENT generalizes the known SDA methods such as GLISTER, GRADMATCH, and a variant of CRAIG. Numerical experiments demonstrated that ORIENT successfully reduces the learning time while achieving the accuracy same as baseline methods.
strengths_and_weaknesses: - *Originality*: Novel. This paper unifies the several known methods with SMI functions. To the best of my knowledge, this seems to be the first study in SDA from the viewpoint of SMI functions.
- *Quality*: Good. The presented results seem sound.
- *Clarity*: While I spotted a few typos in the manuscript, the writing is mostly clear.
- *Significance*: This paper has several interesting contributions. I particularly like that the several known methods in SDA are special cases of ORIENT with different submodular functions. The proposed framework can be applied to a wide variety of machine learning models including neural nets, as long as the (stochastic) gradient descent is possible. So adapting ORIENT in the existing ML framework (like PyTorch or TensorFlow) seems quite easy. Furthermore, numerical experiments demonstrate that the proposed method indeed reduces the learning time: sometimes 2-3x speed-ups. I believe that ORIENT is useful and valuable for practitioners.
questions: - In Algorithm 1, line 4, $\theta$ is updated as $\theta \gets \arg\min \mathcal{L}(A)$. Isn't it a single or a few (stochastic) gradient steps? Otherwise, you get the same $\theta$ for $L$ times until $A$ is changed.
limitations: N/A
review_title: null

Meta-review: metareview: The paper contributes to an important research direction: reducing the computing resources needed for training deep learning models while achieving state-of-the-art results. The proposed method is a principled strategy for domain adaptation problems. While based on existing notions, they are used cleverly. The diverse and extensive experiments show that relying on submodular mutual information to select target points is a promising strategy (although reporting the error bars more systematically would be most appreciated). The activation maps are a nice addition to commonly reported accuracy metrics. 

For the reasons mentioned above, I recommend accepting the paper.

I strongly encourage the authors to consider the reviewer's discussion to provide an improved version. In addition to reviewers' suggestions for improvements, I would like to see in the revised version:
- Line 118: set operations $A \cup x$ written as $A \cup \lbrace x \rbrace$ ;
- Figures 3 and 4: error bars obtained by repeating the experiments with random data splits ;
- Line 278: the name of the missing author (H. Shimodaira) to reference [1] ;
- Section A.5: a clear explanation of how are computed the standard deviations reported in Tables 5,6,7,8.
confidence: Certain
Decision: Accept