Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The Wild-Time collects 7 temporal out-of-distribution datasets, among which 2 are image datasets, 1 tabular dataset, 1 graph molecular dataset, and 3 text datasets. The paper aims at time domains and use/define temporal distribution shift of multiple time steps for each dataset. They test the in-distribution and out-of-distribution performance on multiple methods, including comparison of different continual learning and invariant learning algorithms. The paper also discusses the observations from the experimental results and analyze the comparisons to give brief conclusions, that no existing continual/invariant learning approach generally outperforms ERM on temporal distribution shifts.
strengths: 1. Out-of-distribution problem is an essential field increasingly gaining attention, and temporal out-of-distribution datasets are not discussed much by existing work, therefore collecting datasets for this field is meaningful contribution.
2. Methods specifically designed for temporal distribution shifts are majorly underdeveloped. The paper evaluate performances of different continual learning and invariant learning methods on their datasets, providing useful data to guide the development of future temporal OOD methods.
3. The paper is generally easy to follow and understand.
weaknesses: 1. My major concern is in the data split. For the Eval-Stream setting, which should be closer to the acknowledged domain design in OOD datasets and models can incrementally learn between time steps, the split seems questionable. For example the authors "allocate 10% of the data at each timestep for test, and the rest for training. For OOD testing, all samples in each year are used", this means that the ID training has 90% data, ID test 10% but OOD test 100% data. I suppose this means that they use the data other than the current tilmestep to do OOD test, or it would not fit my understanding of OOD test setting. But this unbalanced ID/OOD train/test split would make the results incomparable with Eval-Fix. And I'm not sure if this would negatively affect the best performance of incrementally learning strategies. The time steps could be designed as domains like general OOD datasets.
2. Selection of methods and models lack motivation reasoning. The paper evaluated algorithms on different model backbones but give no comparison of other model backbones. Since the conclusion includes that no existing continual/invariant learning approach generally outperforms ERM on temporal distribution shifts, careful selection and reasoning of methods and models should be vital. Also some important OOD-related methods are not compared or discussed. Frequently used methods in OOD generalization recently, such as mixup (Zhang, Hongyi, et al. "mixup: Beyond empirical risk minimization." arXiv preprint arXiv:1710.09412 (2017)), is not compared. Also, image and text data have no image/text-specified OOD-related methods compared.
3. Some experimental details are not provided. The paper lack some necessary data to prove some claims convincing. For example, algorithms such as IRM has hyperparameters that can have a wide range of value selection, the paper did not provide the fine-tuning process or the reason for parameter selection, and the hyperparameters can largely affect the performance and observations.
4. The GitHub code documentation are not organized much. As datasets are primarily for community use, it should be easily usable and the documentation should be user-friendly. The GitHub repo has few starting instructions, so it is difficult to use, for example, the modules for the datasets specifically. I would suggest the GitHub documentation be actually maintained.
correctness: Mostly correct. A few statements have issues or are not well-supported.
clarity: The paper is generally easy to follow and understand.
relation to prior work: Relation to previous work is discussed in the paper, the main point is that temporal out-of-distribution datasets are not discussed much by existing work.
additional feedback: The suggestions are provided along with the comments in the Weaknesses part.
review_title: Extensive temporal OOD benchmark but can be improved

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper advocate dealing with the issue of distribution shift over time for robust model performance. The author curated 7 datasets from existing datasets that reflect realistic temporal distribution shifts.  Conventional domain generalization methods, as well as continual learning-based methods, are benchmarked under two different settings (Eval-Fix, Eval-Steam). The experiments demonstrate that the benchmarked methods are limited in dealing with temporal distribution shifts across all proposed settings. This work hopes to assist the community in tackling the problem of temporal distribution shift by proving benchmarking datasets and baseline models.
strengths: - This paper is motivated and clearly presented

- This work provides a set of curated Time-OOD datasets, and results of conventional results. It would be a suitable starting point for the researcher interested to work on.

weaknesses: 
While the realistic temporal issue needs to be addressed, there are some other issues to consider:


- The temporal shift distribution curated in this paper are often intermixed with other types of distribution shifts. For example, in the FMOW dataset, the authors directly partitioned the dataset by year, ignoring location consistency across sets, which would result in performance degradation possibly due to shifts caused by locations. In the dataset Yearbook, whether the male-to-female division is consistent between time periods 1930-1970 and 1971-2013 also affects the performance difference. Thus an important question is how to ensure that the performance degradation is purely due to time distribution shift and not due to inconsistencies of other domains.

- The purpose of this paper is to propose datasets with time distribution shift, however, some settings are not quite consistent with the reality. For example, under the main Eval-Fix setting, the time span between train and OOD test for many sub-datasets is huge, for example, in task 1, the training set is on average 20 years older than the test set, and this setting may be uncommon in the standard  ML development pipeline. What we do know is that model updates are frequent.  It would be helpful if you could resolve my doubts about this, namely that some of the proposed dataset settings are not necessarily reality-oriented.

correctness: Yes, it sounds good but I still have some considerations, please see the Weaknesses part. 
clarity: Yes
relation to prior work: Yes
additional feedback: No
review_title: This work presents a benchmark of out of distribution shift generalization over the time

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors are examining distributional shifts in time, also known as concept drift. They construct a benchmark from a series of 7 existing datasets, develop two evaluation strategies and evaluation a range of methods augmentation, invariant learning and continuers learning methods for improving robustness on this benchmark. The authors show that no methods consistently outperform ERM, that augmentation can sometimes be detrimental, and that invariant learning does better than continuous / incremental learning approaches. The authors clearly show that current tools are insufficient for combatting distributional shift in time. 

This is a good paper, but the documentation is lacking, and I will be happy to revise my assessment when the authors provide the necessary bits, as described below. 
strengths: The authors have provided a valuable and diverse benchmark which focuses exclusively on distributional shift in time. This effect occurs in many services and practical applications, and therefore have broad implications for both research and for industrial practitioners. The work itself examines a broad range of applications, tasks and modalities. There are no negative ethical or social implications in this work.

weaknesses: The current work does not have any significant technical weaknesses. However, it would be interesting to examine time-shift in complex, structured prediction tasks, such as segmentation, autoregressive classification and regression. Additionally, it would be interesting to also consider Bayesian-and-related methods, such as Ensembles, and how much improvement in robustness they are able to provide. Finally, exploring the choice of model architecture and its impact on robustness. 

However, there is a weakness in the support documentation. It is unclear how the dataset it to be supported, under what licenses they are shared / re-used, how the long-term support of the dataset will be carried out. The GitHub (which I know is being refactored) is inconsistent in how the data is to be obtained (either via notebooks or via google drive, etc...). This presents a challenge to accessing the data. These issues needs to be properly addressed for this paper to be accepted. I will be very happy to revise my assessment of the paper when the authors provide adequate documentation.
correctness: The dataset seems to have been partitioned sensibly, the evaluation and experiment design and also sound. The claim of the paper that ERM presents a strong baseline and all other approaches don't work well under distributional shift in time are consistent with previously published results regarding other forms of distributional shift. 
clarity: The paper is clearly written and easy to follow. However, the giant table is _extremely_ hard to follow and spending time interpreting it greatly breaks the flow of the story and the paper. I would very strongly recommend breaking it down into far smaller tables which tell well-contained stories or choosing a different form which can convey the same information more clearly.
relation to prior work: Relation to previous work is detailed well. However, I am unclear how the next Eval-Fix setting in the current dataset is different from the data partitioning of the Shifts Weather dataset, other than not covering different climate-zones. Is it the same?
additional feedback: None
review_title: Reviews of the WilD-Time benchmark.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a benchmark for addressing the temporal shifts in machine learning systems. 
(++) The motivation of the paper is clear. The authors bring up a slight problem of large-scale machine learning systems where the data changes over time. 
 (+) The authors bring a practical suggestion for evaluating new models over time-based shifts.
(-) The definition of temporal shifts is very brief, so the importance of these shifts in practice is not well explained. 
(--) Since this paper aims to propose a benchmark for mitigating the temporal shifts, It should contain a brief definition of the meaning and effects of this type of shift in each dataset. If there is any difference and more detail, clarify the difference between this type of shift and other data shifts. 
strengths: Addressing a real-world problem in machine learning is important to be solved for industrial machine learning systems. 
The purpose of the paper is clear. 
There is no synthetic data creation or changes since the domains are based on actual time frames over time. 
The scope of the paper is highly relevant to current machine learning research borders. 
The evaluation method is described clearly. 
Results are reflected in the paper in a clean and understandable way.
The dataset contains various domains with two different tasks. 

 


weaknesses: Although the contribution is clear, there is no real-world scenario in the introduction for introducing the importance of the problem to the reader. 
All variables in the formula were not defined. 
There is no practical sample for how to use this benchmark or any practical guideline for users. 
There is no introduction for any framework or library which can implement the benchmark in practical projects. 
I couldn't find any link to the dataset or benchmark tool. 
The evaluation metrics part is not well described. 
There is no ethics section or any consideration about that. 




correctness: The evaluation method has been defined completely and tested for evaluating different existing methods. 

clarity: The paper is clear and easy to understand. 

relation to prior work: The contribution is clearly mentioned however there is not enough explanation about previous works on time-based shifts. 
There is not enough explanation on how temporal shift is completely a new and not addressed field in the data shift area. 

additional feedback: -
review_title: Well addressing an important practical issue in machine learning, however the paper is not practical enough 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: 
- This paper proposes a dataset for distribution shifts that incorporates the passage of time using timestamp metadata. 
This reflects natural temporal distribution shifts. Consequently, the proposed dataset, Wild-Time, can be viewed as a more plausible dataset for real-world situations. 
- While the previous dataset (Wild) concentrated on image datasets, this paper deals with various datasets including drug discovery, health care and news classification. 
- Moreover, they suggest assessment of the proposed dataset based on the manipulation of timestamp metadata known as Eval-Fix; time-series setting  and Eval-Stream; continual learning setting.
- To demonstrate the efficacy of the proposed dataset, the authors conducted extensive experiments. They consistently tell us that the proposed dataset represents distinct properties of both in-distribution and out of distribution dataset.
strengths: 
- So far, the domain of continual learning has been mainly used to employ traditional datasets such as CIFAR and ImageNet datasets by gradually increasing class information and manipulating available datapoints. Many researchers have been encouraged by this point to easily pursue continual learning studies. However, this setting is somewhat far from real-world circumstances. Accordingly the authors provide datasets that matches the real-world problems.

- The proposed dataset was evaluated by several baselines recently utilized in distribution shift and continuous learning to confirm the properties by comparing their performance, allowing researchers who need to use the data to do so effectively.

- Through the baselines and extensive experiments, the proposed datasets clearly follow characteristics of both continual learning and distributional shifts. 

weaknesses: - This work tells that invariant learning algorithms have a high level of performance. I guess that the hyper-parameters; K and T, were well calibrated. If the appendix results are provided in the manuscript, there appears to be no meaningful difference as compared to continuous learning. I'm wondering how invariant learning algorithms properly work when decreasing values of K and T. In this regard, I have a doubt whether the invariant learning algorithms observes more data-points in the same batch or epoch comparing the cases of continual learning baselines. 

- I might think that the provided data adequately describe the distribution shift. However, when I look at the table 1 and table 12, it seems that the "Empirical Risk Minimization" work well, allowing the out of distribution dataset to be covered with in-distribution data. When the number of available datapoints is reduced, it would be possible to demonstrate that ERM is ineffective. If it could be demonstrated that the existing distributional shift and continuous learning methods perform better than ERM on smaller datasets, then ERM would be rendered obsolete. I believed that it was originally intended.
correctness: - While the baselines in this paper were picked in consideration of various domains, self-supervised learning approaches that recently get high attention due to the best performance are not considered. Despite the fact that these methods are limited to use for image classification tasks, at least, the author had to apply those to the "yearbook" and "FMow" datasets considering the present studies.

- The Eval Fix and Eval-Stream evaluation metrics are well-designed. It is because the majority of continuous learning has shown no interest in time-variants, and distributional shift does not place a priority on out-of-sample data due to the time-effect.
clarity: The writing and presentation are quite adequate, with explanation of each dataset in the appendix.
However, it takes multiple reads to get a sense of the contributions, and the figures and details are not provided to facilitate quick comprehension of datasets. Specifically, they must demonstrate graphically the reason why the proposed datasets such as drug discovery, climate dataset and health care dataset follow distributional shifts like yearbook and FMoW. Although this is not my major factor in my evaluation of this paper, I believe that authors should dedicate to improve the paper readability.
relation to prior work: The previous dataset, Wild, focused on image classification tasks, meanwhile the proposed dataset, Wild-Time includes multiple domains including drug discovery, news classification (arxiv dataset) and climate issues; precipitation.
additional feedback: I'm awaiting the outcomes of self-supervised approaches and ablation studies based on the reduction of available datapoints. I'm able to to increase my score to 7 based on author's response to my comment.


---

**After rebuttal :** After interacting with the authors and considering the comments and revisions, I am happy to raise my score. The paper can make a good contribution to this venue.
review_title: Official Review 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper proposes Wild-Time benchmark of seven datasets to benchmark methods under temporal distributional shift. Two evaluation settings are proposed. The first evaluation setting is Eval-Fix, which uses a fixed training and testing split as in supervised learning, which is usable to broader research communities. The second evaluation setting is Eval-Stream, which is closer to the continual learning evaluation setting. The paper benchmarks two categories of approaches, domain generalization methods and continue learning methods. The results show that existing methods cannot address the temporal distributional shift problem well, leaving room for future methods to address this problem.
strengths: * While most previous distributional shift benchmarks artificial distributional shifts (e.g., image corruptions [21], background [63], color [3]), the proposed Wile-Time benchmark focuses on the natural temporal distributional shift, which is a problem that naturally exists in the real-world. This is a significant contribution to this community and can facilitate future works to develop robust algorithms against temporal distribution shift. 

* The new benchmark is comprehensive in terms of evaluation metrics, evaluation methods, and diversity of datasets. 
weaknesses: It will be better to discuss potential research directions to address this problem.
correctness: In general, the dataset is constructed in a sound way and evaluations are designed properly. 

However, I am confused by the adaptation of the invariant learning methods (L241-254). If I understand correctly, data from each time window will be regarded as one domain. However, as shown in Fig. 3, the time windows are overlapped with each other. Therefore, does it mean that one data point can belong to multiple domains? This is quite different from the traditional definition of domain, e.g., on Waterbirds [63], an image of waterbird on water background is regarded as one domain in GroupDRO [63] and does not belong to other domains. Can authors clarify why extending the invariant learning methods this way and why not using non-overlapping time windows? 
clarity: Yes. The paper is well-written. 

 

Typo: L248: “domain”. -> “domain.” (The period should be inside of the quote; the right quotation mark is wrong)
relation to prior work: Yes. The authors discuss the difference between Wild-Time and two types of benchmarks—distributional shift benchmarks and continual learning benchmarks.
additional feedback: No additional feedback.
review_title: A paper with good contributions to the distributional shift problem

Meta-review: metareview: I hereby sort the strengths and weaknesses that were mentioned together:

strengths
- the usefulness to the community, and need to develop methods that address OOD data. 
- paper well-written

weaknesses
- there are some questions about the datasplit, which seems major, but also addressed
- adding locational shift op top of the temporal shift, which will be a confounding factor in experiments
- there are concerns about the time between certain batches of the data (one reviewer claims it's too big in a certain setting, another mention overlap in another setting)
- information on experimental choices and details and calibration of hyperparameters of the algorithms
- user friendliness (of documentation, Github repo, etc)
Various reviewers have increased their score during the rebuttal. 

All together, the reviewers have been very thorough, nicely picking up on these matters. While I appreciate their through job, I want to place the weaknesses into context of the usefulness to the community, that was also consistently mentioned. Some reviewers mention that they are still not 100% convinced, which is an actionable point to the authors: please do keep an open mind and seek to further improve on these points. I see no dealbreakers in the weaknesses, and due to the generally positive sentiment, I am very inclined to give a positive recommendation. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept