Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose U-WILDS, which extends the WILDS benchmark (typically used for domain generalization or subpopulation shift) to the unsupervised domain adaptation scenario. They select eight datasets from WILDS, spanning a variety of data modalities, and add in additional unlabelled examples from a variety of data sources. They benchmark a comprehensive set of algorithms which make use of unlabelled data, and find that most methods do not significantly outperform ERM, except for some limited cases which the authors characterize in detail.
main_review: The problem is well-motivated, and the use of realistic problems to benchmark unsupervised adaptation methods presents a major gain over prior datasets which rely on different stylized images (e.g. PACS). The datasets used span a wild variety of modalities and domains, and are fairly robust with large sample sizes. The paper is easy to read and easy to follow, and the experimental evaluations are robust.

I would suggest the following improvements:

- In order to present an informal upper bound for the performance of algorithms on the OOD domain, the authors should consider adding in two additional "oracle" baselines which have access to labelled OOD-domain data: 1) training an ERM model only on the OOD domain, 2) training an ERM model on all available data. 

- The authors should justify why the OOD domain was the one selected in each dataset. If possible, the authors could consider allowing for the ability to swap around the various training/validation/test domains, though it might be impractical if some domains do not have sufficient data.

- The authors should clarify the model selection procedure for each of the algorithms. For example, how does each method make use of the Validation (ID) labelled examples versus the Validation (OOD) unlabelled examples for model selection?
summary_of_the_review: Though the paper does not propose any novel methodology, I believe that it is a solid step towards the use of real-world datasets for benchmarking unsupervised domain adaptation algorithms, and would be a valuable contribution to the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce an extension to the new, but popular, data shift benchmark WILDS data sets called U-WILDS.  U-WILDS increases the number of instances in these data sets significantly (by a factor 3.5-14x times, depending on the data set), but includes no additional labels.  Such an extension allows unsupervised domain adaptation techniques to now be evaluated on the various WILDS data sets.  The authors evaluate a sampling of the state of the art in unsupervised domain adaptation on these new data sets.  The main finding is that most of the techniques performed poorly, except data augmentation in the image problems.  The authors suggest that this could motivate the need for better data augmentation techniques for other modalities.
main_review: Strengths:
1) The problem of unsupervised domain adaptation is well-grounded in a practical learning setting where labels are scarce, but observations are not.  This makes the extension of the WILDS data set to this setting a very useful contribution that can facilitate impactful future work.
2) The empirical evaluation was fairly extensive in it's inclusion of a variety of unsupervised domain adaptation techniques.  I think it is important to validate the trend established in prior work that these techniques typically perform poorly in more realistic domain adaptation problems.  I also think the finding that data augmentation techniques work relatively well in image domains to be an interesting one that can motivate future work.
3) Perhaps a necessary consequence of this kind of work is that the paper serves as a nice survey of modern unsupervised domain adaptation, both in terms of data sets and methods.  Someone interested in the topic could use this paper to begin a literature search.

Weaknesses:
1) Ultimately, the novelty of this work is low.  The main novel contributions are the extension to the existing WILDS data set (using already established data sets used by the original WILDS authors) and some new empirical results using previously published results.  However, I do not think for this kind of work that novelty is paramount.  
2) The descriptions of the data sets are relatively shallow.  Besides raw increase in instances, not much else is reported about the data sets.  Is there anything interesting to say about the instances that were chosen to be labeled by the original WILDS authors versus the ones added by U-WILDS?  Namely, is there anything unique about the new instances?  Can you characterize and measure how different the new instances are from the original ones?
3) Data preparation was not discussed at all.  The major practical concern I have about this endeavor is standardization across new and old instances.  If the original WILDS data set was released with some effort to process the data before repackaging and release, the U-WILDS data set would need to undertake the same process.  I would like to see some comments on whether effort was taken to ensure that the U-WILDS instances were preprocessed for consumption in the same way the WILDS data set were, or why that was not a necessary step.
summary_of_the_review: While I recognize the low novelty of the work, I believe the contribution of the extended WILDS data set as well as the insights provided in the empirical evaluation are of significant enough value to the machine learning community to warrant acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ### New large scale dataset for unsupervised transfer learning

The paper proposed an extension to the popular WILDS benchmark dataset by augmenting the different domain data with additional unlabeled examples. The dataset consists of data of various modalities including images, graph and text from various domains. Additionally, many recent methods that leverage unlabeled images to achieve generalization are bench marked on the proposed datasets. 
main_review: ### Strengths 

- The dataset is a useful addition to the WILDS dataset, and the provided unlabeled data would be very useful to design many unsupervised generalization algorithms.
- The paper is very well written, easy to understand and well organized, although the experimental evaluation could be explained in greater detail in main paper.

### Required Clarifications

- The models chosen for benchmarking the methods are not suitable. While domain adaptation methods like DANN and CORAL work strictly with an assumption of single source to target domains, semi-supervised and self-supervised works are well-known to work only for within-domain samples. In this respect, the observations found with respect to the benchmarking, although useful, aren't surprising. 
- self-supervised, semi-supervised and DA methods make different assumptions about the availability of target data. self-supervised requires target labels for fine-tuning, while DA methods only require unlabeled target data during training and nothing more. So, comparing both of them together using a common yard-stick is, in my opinion, not appropriate.  
- The main paper needs to have more experimental detail such as what are the exact labeled and unlabeled data from source and target for each of the experiments.
- The DA methods considered are quite old and primitive. The authors are encouraged to consider more recent benchmarks like CDAN, CAN, AFN etc.
- The authors are also encouraged to use multi-source/multi-domain adaptation benchmarks [1,2,3] which seem most appropriate for the dataset proposed. at least for the visual recognition datasets. While self-supervised and semi-supervised methods aren't particularly designed keeping in mind cross-domain transfer, UDA methods like DANN and CORAL aren't designed to handle multi-domains. 
- The authors are also encouraged to compare and contrast with [1] in terms of the datasets proposed in both papers (although these can be considered contemporary works).
- The key takeaways from sec 7 are not that surprising, and things like lack of augmentation strategies for many modalities and model selection for DA  are already well known to the community, while ineffectiveness of pre-training seems interesting.

1. Dubey, Abhimanyu, et al. "Adaptive Methods for Real-World Domain Generalization." _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2021.
2. Li, Ya, et al. "Deep domain generalization via conditional invariant adversarial networks." _Proceedings of the European Conference on Computer Vision (ECCV)_. 2018.
3. Li, Da, et al. "Deeper, broader and artier domain generalization." _Proceedings of the IEEE international conference on computer vision_. 2017.
summary_of_the_review: Although the dataset itself is very useful and important, the analysis and bench marking needs some improvement. Nevertheless, this paper definitely has merit, and if the authors could clarify the questions raised, I would be happy to update the score. I haven't gone through the supplementary material in detail. If any of the questions above have a direct answer in the suppl. material, the authors can directly point to that and I would be happy to update my comments. 

Post Rebuttal 
****************

I thank the authors for answering all my queries patiently, which answered most of the questions I had regarding suitability of self-supervised and semi-supervised algorithms to the task. I would like to raise my score, and suggest the authors to include these discussions in the main paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present U-WILDS, an extension of the multi-task, large-scale domain-shift dataset WILDS. They provide a large quantity of unlabelled data complementing 8 of the existing multidomain labeled datasets in WILDS. They propose an extensive array of experiments evaluating the ability of a wide variety of algorithms to leverage the unlabelled data to address domain-shift. They present reasoned conclusions, and open-source datasets and implementations.

main_review: Strength:
- The work behind the data collection and baseline computation appears sizeable and very valuable
- The authors implemented many of the top-performing algorithms meant to address domain shift with unlabelled data. They provide their implementation, as well as a unified, didactic, and detailed explanation.
- The conclusions are both measured and essential: we lack a definitive answer to domain shift in the wild, and unlabelled only provides a very partial solution.
- The writing is remarkably clear and to the point.

Weakness:
I am struggling to fault the paper.  The extensive appendix answered most of my questions.
- The paper does not have a methodological contribution per se, but the answers provided from the meta-analysis are new - at least to me, even if the fact that current methods would not hold too well for in-the-wild data was suspected
- I have some questions about missing details, detailed below

Questions:
- "Models are trained on labeled data from the source domains, as well as unlabeled data from one or more of the other sources, depending on what is realistic for the application." I couldn't understand which problems were allowed to use OOD unlabelled data for training and which ones were not. From my understanding, unlabelled data from all domains (source, val, target, extra) are merged and used for training; is this correct? And what would be the use of unlabelled data if not for helping the training?
- As a follow-up, are unlabelled data from all domains (ID/OOD) used identically by all methods? DANN, for example, may work better if only using target domain unlabelled data, if available.
- The justification to not use augmentation on GlobalWheat are weak: translating and rotating bounding boxes seems easy enough?
- It seems that you did not try approaches that explicitly use the domain as weak labels or adversarially, such as "Adversarial Multiple Source Domain Adaptation, Zhao et al. NeurIPS2018. I am not suggesting that you do (the comparison is already more than sufficient), but you should mention these approaches. Note that these would only work when the domains are not too numerous and meaningful.

Details:
The domains of PovertyMap are missing in Fig2
summary_of_the_review: An impressive and exemplary dataset paper. The data collection and baseline implementation could be an important stepping stone for the ML community to address its biggest challenge yet: domain shift in the Wild. This paper highlights that we are not there yet, and that unlabelled data give an encouraging venue that has not reached maturity yet.
review_title: null

Meta-review: This paper presents U-WILDS, an extension of the multi-task, large-scale domain-shift dataset WILDS. The authors propose an extensive array of experiments evaluating the ability of a wide variety of algorithms to leverage the unlabelled data to address domain-shift problems. The vision behind sounds quite ambitious and convincing to me, namely, the proposed U-WILDS benchmark would be a useful and well-motivated resource for the ML community, and their experiments were very comprehensive. Although they did not introduce any new methods in this paper, U-WILDS significantly expands the range of modalities, applications, and shifts available for studying and benchmarking real-world unsupervised adaptation.

The clarity, vision and significance are clearly above the bar of ICLR. While the reviewers had some concerns on the novelty, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to strongly accept this paper for publication! Please include the additional rebuttal discussion in the next version.
Decision: Accept (Oral)