Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper makes two contributions. 

1. It presents a simulator to generate synthetic visual data. Various parameters such as the number of classes, lighting and weather conditions, can be changed procedurally to introduce domain shifts across time. 
2. It presents an analysis of several continual learning approaches as novel classes are introduced, etc. 


strengths: The idea of using simulators to evaluate synthetic learning might provide a systematic way of evaluating approaches as good benchmarks are lacking for this task. Since the generative model is controllable in interpretable ways, analysis can be done along different dimensions providing more insight into ways to improve continual learning. Some of these aspects are illustrated in their study on model's robustness to new classes and weather conditions.


weaknesses: Similators have their own limitations due to the narrow range of visual appearances and other shortcuts employed by graphics engines (e.g., coarse geometry but detailed texture) for realism. This is also a reason for skeptisism for the proposed conjecture. In particular, it would be useful to characterize what good performance on this dataset tells us about perhaps a real-world use case. 


correctness: The paper appears to be correct.
clarity: The clarity can be improved. Several pages in the paper appear as a block of text, which could be shorted into bullet points. It would help to include some figure to illustrate the variety of the dataset and the types of mistakes the models makes in the main paper. 
relation to prior work: Prior work is well discussed.
additional feedback: None.
review_title: Interesting use of simulators to evaluate continual learning

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper proposes a very useful and carefully crafted simulator to the computer vision community, in the context of autonomous driving and continual learning over dynamic content. 
strengths: This is exactly the kind of work that this track is meant to encourage -- very difficult to design and architect simulator, well thought-through and clearly documented. The authors have also took care to make the simulator available as an executable, thus bundling 3D assets that may be difficult to accept in their raw form. The value of raising awareness of this useful tool outweighs any possible flaws in the submission details.
weaknesses: As any handcrafted CG simulator, results obtained using this simulator may not transfer well to real-world scenarios. Yet, this is tool is a clear step forward. 
correctness: Yes, the dataset construction method is well-explained.
clarity: Yes.
relation to prior work: Yes.
additional feedback: N/A.
review_title: Very useful dataset tool for the community

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a graphical model for generating datasets for continual learning. The data are synthetic video sequences of urban scenes that can occur in driving, generated by a computer graphics framework. The users can set different configurations for the objects in the scene, the weather condition, illumination intensity, etc. The authors generated three different types of video sequences and benchmarked several existing continual learning algorithms.
strengths: I think overall the contribution is significant in the continual learning community. The lack of realistic datasets has been a problem in the continual learning community for a long time, and this paper proposes one option to resolve it. The datasets and implementations are released and accessible to the general public. I did not see any ethical and social issues. Detailed comments:

1) The proposed framework is flexible. Users can use it to generate different types of video sequences (e.g. different weather condition) to evaluate continual learning algorithms in different applications. The tasks look more realistic than several commonly used datasets such as the split CIFAR and rotation MNIST.

2) In addition to classification, the dataset also supports other tasks such as image segmentation. The authors did not provide experiments on tasks other than classification, but as the authors mentioned, other tasks can be evaluated using this dataset.
Moreover, since the dataset is video, it also allows the study of the temporal relationship between the frames, which has not been explored enough in continual learning community.
weaknesses: I think the weakness of this paper is mainly in the experiment section (Section 4).

1) Although this paper proposes a more realistic data generation framework for continual learning, I think some datasets that the authors used to benchmark existing algorithms are not realistic enough. For example, the incremental class task does not seem to be realistic in the real urban driving scenario. In practice, it's unlikely that in the first video sequence we only observe trees and in the second sequence we only observe street-lamps, etc. Another suggestions is that for weather condition change, I can imagine that different weather conditions (e.g. clear vs fog) can appear with different probability and some bad condition appears less likely but are important for driving. Right now in the benchmark different weather conditions appear equally likely.

2) I am a bit confused by the results shown in Figure 4. I am not sure what the y-axis "Accuracy" mean here. Is it the accuracy over all the distributions (e.g. in the left figure, all the objects?) If that's the case, then after the first task, the algorithm has only seen trees and not other objects, how can the algorithms all have >80% accuracy (over all the objects) after the first task in the left figure?

3) I am also a bit skeptical on the results that many other methods do not outperform naive continual learning baseline. Intuitively, at least for the methods that stores replay data from previous sequences (e.g. GEM) should not be worse than the naive baseline since in the worst case they can simply discard the replay data and recover the performance of the naive algorithm, and in the literature the replay-based algorithms usually have good performance. I am wondering if the algorithms are carefully tuned or not.
correctness: I think the dataset is constructed in a sound way. The graphical model is quite complicated (see Figure 2), and I think it's a bit hard to follow all the details of this generative model. However, I do think it seems to be a reasonable model for generating urban driving videos. I have some concerns on the experimental results as mentioned in the "Weakness" section.
clarity: The paper is overall clearly written. As mentioned in the "Correctness" section, I think it's hard to follow all the details of the generative model. In addition, as mentioned in the "Weakness" section, I think there are several places that are not clear enough in the experimental results.

Typo: Line 238 its -> it’s 
relation to prior work: The discussion on prior work is clear and sufficient.
additional feedback: See the review in other sections.
review_title: A good paper for generating continual learning datasets

Meta-review: This paper proposes a well-designed and well-documented simulator to help study continual learning. Like all reviewers I recommend acceptance.
Decision: Accept