Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses universal domain adaptation (UniDA) for object recognition: target sample is classified as either one of the "known" classes or "unknown". This work proposes an add-on strategy that helps improve results of existing approaches, reaching SOTA performance in UniDA.

** Analysis **
For UniDA, one wants to increase the domain-invariant score (DIS) between source/target while trying to alleviate the potential negative-transfer risk (NTR) between "known" and "unknown" classes. However, the two measures are often at odds with each other. Quantifying NTR as known-unknown classification accuracy using entropy threshold (like [17]) and DIS as the $\mathcal{A}-$distance (like [1]), the paper provides an analysis on the NTR/DIS trade-off of resnet features at different depths. Empirically, feature of the res3-block balances NTR and DIS. Based on this finding, res3-block feature is used throughout experiments.

** Proposal 1 **
To make domain alignment possible while avoiding the risk of negative-transfer, the paper proposes to align in the Bag-of-visual-words space. The intuition is that BoW contains visual primitives that are universally shared among known and unknown classes. To this end, res3-block features $h\in\mathbb{R}^{H\times W\times N_d}$ are passed through a soft-assignment module including a vocabulary matrix $V\in\mathbb{R}^{N_d \times K}$ ($K$ is the vocabulary size) and a soft-quantization operation (Eqn. 4). For both source and target samples, one minimizes the BoW-wise entropy of the corresponding soft-assignment tensor $\in\mathbb{R}^{H\times W \times K}$. That is to implement the sparsity assumption of the word-histogram space.

** Proposal 2**
A novel pretext task that help encourage prototype-alignment: a pretext dataset is composed of images stitched by a varying number of random image crops from different images. Pretext task is to predict the number of crops, based on the global word-histogram of the stitched image.

** Experiment **
On several benchmarks (Office-31, DomainNet, OfficeHome), the two proposed strategies complement with existing approaches and improve performance, achieving SOTAs in UniDA.
strengths_and_weaknesses: - The paper is well-written, easy to follow. Most arguments are sound with adequate empirical supports; there are only a few concerns (see below).

- Originality: the two proposed strategies are novel in the context of UniDA. 

- Extensive ablation studies validate the proposed strategies and experimental choices.

- Complementary to existing approaches. The combinations with either OVANet [32] or OCC [20] obtain SOTA results.
questions: - What's unclear to me is the sparsity assumption on the word-histogram space. Could the authors elaborate more the intuitions?

- As discussed in L216-220, there are risks of word-level misalignment when minimizing the BoW-entropy. Could the authors provide some real examples when the misalignment happen? How does the proposed pretext task help mitigate the misalignment risks?
limitations: Limitations and potential negative societal impact are given in the supplementary material. I find those adequate.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper studies the universal domain adaptation problem. Unlike partial DA or open-set DA, UniDA does not require knowing relation between source and target domain label sets. The authors start by analyzing the behaviors of source-trained classifier on target domain data. Specifically, the authors propose a new metric called negative-transfer-risk (NTR). It measures the class-specificity of the features in a specified layer via a shared-vs-unknown binary classifier. The other metric is the inverse of A-distance. Motivated by the analysis, the authors propose the do the adaptation in the mid-level layer of the network. Inspired by the BOW, the authors design a word-prototypes in the selected mid-layer representation, and regularize such a BOW-like representation via a subsidiary prototype-space alignment. To realize SPA, the authors design a novel word-histogram based pretext task with UniDA objectives. The experiments are conducted on three DA benchmarks, and competitive results are reported.
strengths_and_weaknesses: Strength
* The studied problem Universal DA is a much practical setting, where much weaker assumption is posed compared with standard, partial or open-set DA.

* The technical detail part are generally well written, such as the design of word-prototype.

* The reported results seem competitive. It shows the benefit by applying SPA to both OVAnet and DCC. 

Weakness
* The introduction and motivation of NTR are unclear to me and not easy to follow. For detailed questions, please see the below question section.

* Experiments. To support the key statement of mid-level adaptation, the authors miss an ablation study of applying SPA on different layers and all layers.

* From Office-31, [49] seems the best existing method. I’d enough the authors to apply SPA to [49] as [32,20] to show the generalizability of SPA.

* The proposed SPA is based on convolution operator. As vision transformer is becoming more popular and powerful, the applicability of the proposed method may be limited to ConvNet only.

questions: 
* As defined in Eq 1, NTR measures the class-specificity of the features in a source-trained specified layer via a shared-vs-unknown binary classifier. So Fig 2 blue solid curve only suggests that in deeper layers (e.g. Res4/FC), it is easier to tell apart the known/unknown given the feature of the source-domain trained network, compared with the feature extracted from the shallow layers. I am not clear why this observation suggests “(line 131: adaptation should be performed at a shallower layer.)”, since the final DA classification task will be performed in the final deep layer.

    Also, I’m not clear if a higher NTR score is better or the other way around. Intuitively, I think higher NTR the better. While when I see the dashed blue curve (post adapt) in Fig 2, it looks that the lower the better. Please clarify.

* Similarly, DIS score only shows the expressive capability of each layer in terms of domain-invariance. Then the conclusion is drawn that the adaptation should be performed in the mid-level layer, which seems contradictory to the existing multi-level adaptation works such as [ref-1].   
[ref-1] Xie et al., Multi-Level Domain Adaptive Learning for Cross-Domain Detection, 2019

* Please conduct an ablation study of applying SPA on different layers and all layers.

limitations: No potential negative societal impact identified.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors tackle the problem of universal domain adaptation where the source domain consists of fully-labeled images and unknown category images while the target domain consists of unlabeled images of known and potentially unknown categories. To tackle this problem, they first study the trade-off relation between the negative transfer risk (NTR), i.e., class-level misalignment between the shared and private classes, and domain invariance. Motivated by the trade-off between negative transfer risk and domain invariance, they propose to align mid-level features of deep neural networks. To this end, they propose to align bag-of-word style features with self-entropy regularization to encourage the sparser BoW features. They further regularize the model training by adding a pretext task of entropy-bin classification on the grid-shuffled images from multiple instances. Plugged into the existing UniDA methods, the proposed method shows favorable performance on universal and open-set domain adaptation settings on three public benchmarks.
strengths_and_weaknesses: The strengths of this paper are as follows.
S1: The problem tackled in this work, universal domain adaptation is challenging and interesting. 
S2: Aligning mid-level features for the task of UniDA seems reasonable. Intuitively, high-level features are often specific to classes, therefore prone to category-shift. In contrast, low-level features are robust to category shift. However, they are prone to domain-shift. Therefore, intuitively mid-level features such as BoW could be a good balance between category-shift and domain-shift.
S3: The proposed method shows favorable performance on three public benchmarks. The proposed components, the BoW-style architecture, entropy regularization, and pretext task, contribute to the performance.

However, this paper has several weaknesses as well.
W1: My biggest concern is the study on the relation between NTR and DIS (domain invariance score). Fig. 2 shows Res1 block has the lowest NTR and highest DIS. Compared to Res1 block, Res3 block shows higher NTR and lower DIS. If NTR and DIS are the only criteria, why do we need to use Res3 block feature for alignment? Why we cannot align Res2 and Res1 as well? 

W2: The proposed architecture modification does not seem to align the mid-level feature only. By minimizing L_{em} we align the mid-level feature. However, minimizing L_{s,n} and L_{t,n} also align the high-level feature as f_n is attached on top of Res4 and GAP. This is a self-contraction with the claim made from Fig. 2.

W3: The improvement from each component is not very surprising. BoW-style mid-level feature + L_{em} only improves 0.4 ~ 0.5 points on top of OVANet. The results make me doubt that the proposed BoW-style feature alignment is really optimal. In addition, the proposed pretext task improves only 0.4 ~ 0.6 points compared to the rotation task.

W4: Presentation quality should be improved. In general, the paper is not straightforward to read. For example, it is hard to understand why NTR in eq. (1) measures negative transfer risk. Another confusion is that do we need to more align the layers with higher DIS or layers with lower DIS.



questions: Please address all of my concerns in the weaknesses part above. Also, I do have a few more questions for the authors here.

Q1: Why the NTR in Eq (1) does measure negative transfer risk? Basically, it is the accuracy of the pseudo labeling of the shared-private label using entropy thresholding. If we have a high NTR value, we know what are known samples and what are unknown samples. Intuitively it would reduce class-level misalignment.

Q2: In L142-144, the authors claim the domain adaptation should be performed at a deeper layer to encourage domain invariance. However, Fig 2. Shows deeper layers show lower DIS. What is a reasonable conclusion here? Should we more align the deeper layers with lower DIS as they are less domain invariant? Or should we more align the shallower layers with higher DIS as they are domain more promising?

Q3: How do we train a linear domain classifier to measure DIS? 

Q4: In Table 6, are the other pretext baselines equipped with the proposed BoW-style architecture modification and L_{em} as well? If not, we need to compare with “our pretext task” baseline in Table 5 (64.9% and 72.8%) instead of full SPA (65.5% and 73.2%).

Q5: In L271-272, why the word prototypes are learnt only through the pretext task? The main task classifier is also attached on top of the word-histogram feature as shown in Fig. 3.


limitations: The authors did not really address the limitations of their work. Please describe what the failure modes of the proposed method are and when the failures happen. They have not discussed the potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This submission deals with universal domain adaptation for object recognition. The authors propose to extend existing strategies with an original and effective complementary strategy, thus achieving SOTA performance in this context. Their first proposal aims to align domains while avoiding the risk of negative-transfer, working in the Bag-of-visual-words space. Their second proposal is a new pretext task which seeks to predict the number of crops on images stitched by a varying number of random image crops. This should favor prototype-alignment.

This submission received diverging ratings. Reviewers have raised several concerns, to which the authors have provided detailed answers. The reviewers appreciated the answers and the additional experiences provided. Following the discussions, the final scores of the reviewers have increased and are clearly positive on this submission, on the express condition that all the improvements discussed are integrated in a very careful way.

The AC agrees that the strengths in this case outweigh the weaknesses, but strongly recommend that all the improvements are fully reflected in the final version.
confidence: Less certain
Decision: Accept