Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes Okapi, a method for domain adaptation / domain generalization based on the idea of matching unlabeled examples to labeled in-distribution examples and enforcing consistency in predictions. The authors propose an elaborate strategy for matching examples, and show promising performance on two datasets from the Wilds benchmark.
strengths_and_weaknesses: Strength 1. The paper tackles a challenging problem of using unlabeled data for improving out-of-distribution performance. Prior work suggested that it is not trivial to achieve performance improvements over standard ERM by using unlabeled data. The authors show promising results.

Strength 2. On a high level, the method makes intuitive sense. The authors also perform an ablation evaluating the importance of some parts of the method.

Weakness 1. The method is quite complex, with a lot of moving parts. It involves two versions of an encoder (regular and with smoothed weights), propensity score classifier based on the embeddings, regular classifier, memory banks, two different criteria for discarding datapoints from matching (referred to as calipers in the paper). The method is not easy to understand, and it is not presented very clearly. The two calipers (methods for discarding outlier datapoints), std and fixed, are not described in sufficient detail in the main text. The matching strategy is also only described somewhat vaguely. In particular, what is $\xi$ in CaliperNN$_{\xi}$? How exactly are propensity scores used? 

Given the complexity of the approach, it would be useful to have an algorithm box explaining precisely what each stage of the procedure is doing.

Weakness 2. The empirical evaluation is somewhat limited. The authors focus on only two (challenging) datasets from the Wilds benchmark, comparing to two baselines: ERM and FixMatch. FixMatch was shown by prior work to perform poorly on these datasets, as mentioned by the authors. It is hard to know how significant the improvement is relative to the baselines: on poverty map ERM performs comparably to Okapi in terms of MSE. On iWildCam FixMatch (which was reported to work quite poorly on this dataset according to the authors) performs comparably to Okapi.

On both datasets, Okapi performs slightly better, but it is hard to tell if the improvement is significant, especially given that the method is more or less designed to work well on these two datasets, while FixMatch was shown to perform particularly poorly on iWildsCam. 

Another complication with the evaluation is that the authors chose to use a different backbone compared to prior work, ConvNext. Consequently, we cannot directly compare the numbers reported by the authors to the numbers reported in the prior work. Why was this choice made?

The Calipers in table 1 appear to provide a fairly minor improvement compared to Okapi no caliper.

Weakness 3. It seems like the method may not work well in the cases when the domains are sufficiently different, such as in the common domain generalization benchmarks. The idea of the method is to reject datapoints which can be differentiated from the in-distribution datapoints (based on propensity scores). On datasets such as PACS the domains can be clearly differentiated, so my understanding is that the method would reject all the images from other domains. Is this interpretation correct?
questions: Question 1. Please see my questions in the previous section. 

Question 2. Is there a reason to train the propensity classifier online, along with the encoder / main classifier? It appears that you can train it once before training the main classifier, and compute all matchings, avoiding the complications with the memory bank and online training? Or is the fact that the propensity classifier using the same features as the main classifier important to the method?
limitations: No issues.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a general and robust semi-supervised learning algorithm called Okapi that works for data from various domains. It is based on the simple idea of supplementing the supervised loss with a cross-domain consistency loss that encourages the outputs of an encoder network to be similar for k-nearest neighbor samples belonging to different domains. To make the online matching in a runtime- and memory-efficient way, they incorporates a statistical matching framework and fixed-size memory bank. The results show that Okapi outperforms the baseline methods in terms of out-of-distribution (OOD) generalization on iWildCam and PovertyMap datasets. The work suggests that it is possible to effectively incorporate unlabelled data and domain information into a training algorithm to improve OOD generalization. 
strengths_and_weaknesses: Originality: Making use of the domain information and using propensity score to predict domain for efficient matching is novel. 

Quality: The experiments are done for comparison with the existing methods, and for ablation study are solidly done. 

Clarity: The paper is well written.

Significance: Most of the results in Table 1 are statistically significant.

questions: 1. What is the intuition behind the CaliperNN algorithm to mimic the standard nearest neighbor algorithm? 

The first stage is to remove the samples with a propensity score exceeding a predefined threshold. If I understand correctly, that means we want to remove those samples with high predictive confidence in predicting domains. In other words, we want to keep those samples that lie on the domain decision boundary. How could those samples be more likely the nearest neighbors?

The second stage of filtering is to keep paired samples from different domains only if their Euclidean distance between their respective propensity scores is below a threshold. Do you assume that the small distance in propensity score will reflect small distance in their representations?

2. There are many moving parts and hyperparameters in the algorithm. Appendix C studies the effect of choosing the number of nearest neighbours k, the use of the fixed- and std-calipers in CaliperNN, and which encoder (online or target) is used to generate the queries for statistical matching. How did you choose the threshold t_f and t_theta, and are the results sensitive to the threshold values? Moreover, since the ablation is done in PovertyMap dataset, is the conclusion transfer to other datasets like iWildCam?


limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper uses a modified nearest-neighbors method to match unlabeled data with the labeled data in the latent feature space for semi-supervised learning. The authors conduct experiments on iWildCam and PovertyMap datasets. They compare the performance of their proposed method to two current semi-supervised learning methods (i.e., FixMatch and ERM). 
strengths_and_weaknesses: # Originality
The general idea of using some similarity measure to pair labeled and unlabeled images for semi-supervised learning is not new (e.g., Chen et al. (2022)). In this paper, the authors use nearest-neighbors as a measure of similarity.
# Quality
## Strengths
The paper is well written. Apart from some grammatical errors, the quality of writing is good.

## Points of concern
The proposed nearest-neighbors matching algorithm uses the feature vectors stored in a memory bank for all the previous unlabeled input images. I have concerns regarding the memory requirement for such a memory bank. Also, given the sheer size of modern unlabeled datasets (over 100k samples in the datasets studied here), using KNN to find a match for each input image among all the previous unlabeled images during training seems like a very computationally expensive approach. To address these concerns, I suggest that the authors report the memory and computation requirements for their method. Additionally, I suggest that they compare their method to a supervised learning approach not just in terms of OOD performance, but also in terms of memory and computation costs. Then plot a trade-off curve between the added costs and gain in performance for their proposed method compared to the supervised method. Then we can see whether Okapi has practical significance.

The concern about the practical significance of the proposed method is further exacerbated by the marginal improvements in the benchmarked datasets (0.9 improvement in macro F1 score for the iWildCam and 0.05 improvement in Pearson correlation for the PovertyMap). Again, extensive analysis of the added memory and computational costs of the proposed method is required to see whether it is justified to burden such additional costs in exchange for the minor improvements in performance.


# Clarity
The overall clarity of the paper is good. There are some grammatical errors that need to be fixed (e.g., line 333: “each which…”)
# Significance
Proposing new algorithms without proper cost analysis proves little practical significance.

questions: Did you record the memory, computation costs, and carbon footprints of your experiments? If so, will you be able to add the analysis to the paper for the camera-ready version?
limitations: The computational and memory costs in addition to the carbon footprint of the proposed method is not discussed. As the proposed method, theoretically, seems to be computationally intensive, analysis of these costs become crucial for the assessment of the method.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors studied semi-supervised learning across multiple domains. For each input example, they search for its k-nearest neighbors from other domains and enforce their feature representations to be similar. To filter outliers, the authors develop cross-domain matching based on the propensity score.

The authors applied the proposed algorithm to two datasets: iWildCam and PovertyMap. Empirical results show that their proposed method outperform previous baselines.
strengths_and_weaknesses: Strengths:
1. The idea of enforcing feature consistency on the unlabeled data is interesting.


Weaknesses:
1. The empirical results (OOD 36.1 vs. 35.2) are far from satisfying. Moreover, the in-domain performance of the proposed method drops from 51.1 to 50.6. With 819K unlabeled examples (almost 8 times the size of the labeled data), I don't feel that this matching algorithm will be practically useful.
2. The presentation quality needs to be improved.
    + The introduction focused too much on those common sense knowledge and only talked about their proposed method in the last paragraph.
    + Confusing notation. For example in Eq (2), what are the arguments of $NN$? In Eq (4), what is $\text{CaliperNN}_\xi(z, \mathbf{z}, \mathbf{e}, s, k))$?
    + Given the complexity of the algorithm, I think the authors should include a pseudo-code in the main text.
questions: In the algorithm, you tried to push the representations of the query example to be close to its unalbeled neighbors. Why would it help generalization? Aren't their representations already very close (since they are selected by k-NN)?
limitations: No. The authors claimed that there is no negative societal impacts.

All papers have limitations. I think it is important for researchers to discuss the weakness of their proposed method so that people can better position the paper.
review_title: null

Meta-review: metareview: in this paper, the authors propose an algorithm called Okapi that improves out-of-domain generalization by ensuring the representation similarity between in-domain (labelled) examples and out-of-domain (unlabelled) examples. despite the algorithm's high complexity, which initially concerned the reviewers, the empirical results (both in the original and response) as well as the authors' successful rebuttal convinced the reviewers that this manuscript is worth publication. therefore, i recommend acceptance.

authors, please take into account the comments from the reviews when preparing your camera-ready version.
confidence: Less certain
Decision: Accept