Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new cross-season monocular depth-prediction dataset which contains a large(r) variety of environments under different seasonal conditions. The dataset is a derivation of the CMU Visual Localization dataset, obtained using SfM. 

The authors evaluate the performance of their dataset on a large set of pre-trained SOTA models, which represent the different model categories including supervised methods, self-supervised, and domain adaptation methods. New metrics are also proposed to provide further insight into the performance of the models on the new dataset. 

The experimental results showcase the challenge of the different seasons and adverse environmental conditions on the performance of SOTA models, with some preliminary insights into what model categories may be potential candidates for refinment. 

strengths: The dataset is a valuable contribution and along with the toolkit which the authors have provided, should spark followup research in this area. 

The depth of evaluation is also appreciated with comparisons of different (pre-trained) methods. 

weaknesses: The dataset is a derived from the CMU Visual Localization dataset using SfM which reduces the novelty of the work. 

While the breadth of evaluation (number of model variations) is commendable, it is hard to obtain any actionable insights from the experiments regarding promising avenues of future exploration. Some commentary of how follow-up research directions may have been useful. 

The authors also largely rely on pretrained models for their experiments. 
correctness: Seems ok.
clarity: The paper will benefit from an edit by a native english speaker
relation to prior work: Seems ok.
additional feedback: ** Update 7/19 ** 

The authors have done a commendable job in responding to reviewer comments. 
While my concerns on the somewhat incremental nature of the work remain (which cannot be addressed during a rebuttal phase), I have raised my score as the authors have addressed the other concerns I raised.
review_title: A cross-season monocular depth prediction datastet derived from the CMU Visual Localization dataset, comprehensive evaluation on several SOTA methods, and proposal of new metrics 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: A dataset for the estimation of depth from monocular images under different environments (seasons) is introduced. The dataset is build on top a previous dataset, applying a pipeline that includes structure from motion. A comprehensive experimental evaluation is performed using existing methods that have been evaluated in other datasets. 
strengths: The dataset seems to have some advantages existing ones (yet it is not clear to me how important the differences are, see below)

The experimental evaluation considers several methods from the state of the art, yet I cannot assess the impact of the findings derived form the experimental evaluation
weaknesses: * I think the motivation for the need of this dataset and benchmark is weak, basically, lines 28-38 (whose writing needs to be improved considerably) are the only motivation. This is a critical aspect to evaluate in the paper, how the problem is approached currently? What other uses for the data set the authors foresee (e.g., it could be used for style transfer with GANs), why is the approached problem relevant nowadays? Etc.

* the methods adopted in the pipeline are not described in the paper, while these may be trivial concepts for people working in this subfield, other readers may struggle with these. Also, it is not clear why is this the best method to use for this process, are not other options for this? Why is this pipeline better? What are its benefits and limitations?

* To me is a bit subjective that the dataset is called real, because the new information with respect to the dataset introduced in [2] has been generated by an algorithm, 

* Removing dynamic objects is described as an advantage in lines 131-133, I see this more like a limitation, can the authors clarify this?, 


* To me is not clear what are the distinctive features of the proposed data set with respect to others shown in Table 1, what makes unique this data set?  Datasets from references  [43, 45, 46] seem similar, or even better than the proposed one. 

* From Section 4.1 is not clear to me why no the metrics that have been used for other datasets cannot be used in the proposed dataset/benchmark?

* I think that the experimental evaluation is a valuable contribution from this paper, however, I am not sure what is the impact of the findings? For example, from 5.1, the main finding seems to be that some algorithms perform better than others in different datasets. Overall, I find this kind of expected/obvious, can the authors argue what is the impact of the findings from that section?, do they comprise new results that have not been published elsewhere? In what sense? 


correctness: As far as i can tell, the submissions is correct
clarity: The writing is good in general, yet there are many typos and it can be improved. However, I do not think the paper is clear in motivating the need for a dataset considering this particular scenario (approached problem), what are the benefits/limitations with respect to existing datasets and what is the potential impact of the findings derived from the experimental evaluation.  
relation to prior work: Somehow, please see my comment on clarity 
additional feedback: ****** After author' feedback *********

I appreciate the effort authors dedicated to clarify some of my concerns, based on this I have raised my score.
review_title: I could not get what are the distinctive features of this dataset/benchmark and what is its potential impact

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work proposes a dataset for monocular depth prediction in different environmental conditions. The dataset is derived from the CMU Visual Localization dataset by computing depth map via SFM. It features two cameras mounted approximately 45 degrees to the left and right of the heading of a car driving in suburban environments. The paper evaluates multiple methods from the KITTI benchmark on the new dataset without fine-tuning and compares these approaches with common metrics with additional emphasis on their variance.
The contributions are:
- New (scaleless) depth ground truth for a subset of the CMU Visual Localization data
- Evaluation and comparison of various methods on this dataset
strengths: 1) Dataset focuses on the robustness of Monocular depth estimation under different environmental conditions which is not the focus of existing datasets. However, for real-world use robustness is important.
2) Evaluation of a variety of methods ranging from supervised to self-supervised methods and domain adaptation.
3) Dataset is available and seems to be ready for use (documentation and code are there).
weaknesses: 1) The work purely evaluates methods from KITTI (forward-facing camera) without fine-tuning on SeasonDepth (this dataset) which features cameras looking at approximately 45 degrees to the side. Therefore, data is heavily biased with respect to KITTI in terms of camera view. As a result, it is unclear whether the environmental conditions or the different camera views are responsible for the performance gap. It is likely, the camera view plays a major role in the performance because no fine-tuning on this dataset was proposed. Instead, it is claimed that the different conditions are predominantly responsible for the performance drop.
2) No experiments with fine-tuning on the training set of the proposed dataset. If seasonal changes are a challenge, it might be just enough to add more variety to the training data (add more seasonal variability). In this case, it can be argued that the issue lies with the limited variability of training data rather than a methodological vulnerability.
3) No ground truth for dynamic objects. It is very difficult to compute ground truth for dynamic objects in real-world datasets. However, it is still a limitation because (e.g.) KITTI does have such ground truth and dynamic objects are highly relevant for driving.
4) While the ground truth is relatively dense, it is also often wrong for thinner structures such as poles and branches due to the fact that the reconstruction at those regions fails often.

Final Edit:

1) and 2) have been partially addressed with fine-tuning two models. However, the authors claim that "fine-tuning helps little to the robustness to changing environments though average performance is improved because of reducing the domain gap, indicating that solely increasing the variability of training data cannot deal with the challenge of environmental changes" (line 239 - 242). Still, I am not convinced that this is true because clearly the AbsRel Variance of the finetuned method BTS [22] was reduced by half after supervised fine-tuning. Furthermore, it seems that (according to my understanding reading the rebuttal, comments, paper, and supplementary material) fine-tuning was done in a self-supervised fashion. The problem with the self-supervised fine-tuning of [28] is that it assumes a static environment to compute the loss. The increased variance is most likely due to dynamic objects that violate this assumption instead of any environmental factors. It could well be that supervised fine-tuning of this method would again substantially decrease the variance. In that light, I find that part of the paper unclear and potentially confusing.
correctness: 1) The claim that existing monocular depth estimation methods are not robust wrt do different environmental conditions cannot be made. This is mainly due to weakness (1) mentioned above (camera angle). Further evidence is provided in table 3 in the paper where the performance does not vary substantially within a method for different environments/sequences. Instead, it is likely that the different camera angle is a major challenge for these methods which were all trained/fine-tuned on KITTI on a forward-facing camera.
2) The claim that cross-seasonal monocular depth prediction is an open problem might not be the case if the right training data is available. Is this still the case if (some of) these methods are trained on the training split of the proposed dataset?

Final Edit:

1) is partially addressed with fine-tuning two models. This weakness does not apply to the supervised fine-tuned model [22].
2) See above in the weakness section.
clarity: The paper is mostly coherent and cohesive. Some sections require improved formatting (sec 4.1 with a lot of inline equations) or rephrasing (line 287 "sucked results").
relation to prior work: The relation to prior work is clear and summarized in table 1.
additional feedback: My current rating is dominated by the claims that are unwarranted due to camera angle (bias in dataset wrt KITTI for evaluation) and lack of training on the dataset for at least a subset of the methods. Overall, the dataset documentation is good and ready for use but the innovation from the dataset point of view is not outstanding. For example, there is no ground truth for moving objects and for night sequences both of which are challenging and relevant in practice.

Final Edit (apart from my previous feedback):

I am unsure whether this paper will have a significant impact on the NeurIPS community and find that there is a novelty in the dataset. However, the novelty is limited. For example, it is not larger, or higher quality than previous datasets nor does it include dramatic (one exception is the snow sequence) environmental changes. Indeed, the idea of seasonal changes for depth prediction is new but it might not be good or novel enough to convince researchers to use the dataset.
Finally, the authors took a great effort to improve the results and presentation of the paper such that I raise my rating to 5.
review_title: Ok dataset with uncertain impact

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces a new cross-season dataset for outdoor depth estimation. The evaluation reveals that there are some issues when trying to apply existing depth estimation approaches to different environments as described by the authors. The motivation is solid: the generalization of learning-based methods on different weather and illumination effects are still an open problem, so a high-quality dataset is indeed needed to ensure safe applications such as self-driving cars.
strengths: Strength of this paper:

1. This paper identifies the problem correctly and does make a step towards the goal by constructing a depth-prediction dataset with different season / weather conditions.
2. I like the exhaustive details supplied by the authors, which eases the reproduction for potential readers.
3. The extensive results do reveal some interesting facts on supervised, self-supervised, and domain adaptation approaches. 
weaknesses: I do have some significant concerns on this submission:

1. Dataset. The major concern is the complicated workflow of generating the ground truth depth. Given that there are only very limited number of views available, it is quite unreliable to use MVS to reconstruct depth, even if with careful filtering and manual editing. I agree that LIDAR output is sparse, but in this case it may have better accuracy since it's physically based. Another concern is that most (or even all) dynamic objects are removed. If I read the paper correctly, these two cameras are on a moving vehicle, then doesn't it mean all objects are in relative motion? Besides, from the provided images in the paper I can see some pedestrians/cars are removed but some other are kept. Is there a threshold for making this static vs dynamic decision? The authors are claiming the proposed dataset is mainly serving the autonomous driving and moving robots, and those dynamic objects are clearly the main prediction targets which are however ignored in this dataset.
2. Synthetic vs Real. The above opinions on the dataset make me think if a synthetic dataset is a better way to handle this case. Since the graphical rendering is becoming more and more realistic, and given the fact that you can control the lighting (and possibly seasons) very easily, seems to me it is a win-win choice since it also gives you the absolute correct depth from the renderer without needing any complicated ground-truth generation pipeline.
3. Contribution. Besides the dataset, this paper doesn't propose any solution (even a naive one) to the identified cross-season problem. So based on my understanding the problem may be just resolved by fine-tuning the network on the new data? If so, the contribution of this paper may not reach the standard of this conference track.
4. Evaluation and text. Despite the extensive experiments conducted by the authors, the current submission fails to deliver the clear insights of the finding. The majority of the text just simply describes those three big tables but doesn't summarize the identified patterns in a clear way. This makes me get lost many times when I went through the evaluation section. Some observations are not general to all the cases, like supervised vs self-supervised. I hope authors can summarize their findings in a more concise and concluded way. Moreover, currently most statements are based on the numeric metrics, however sometimes the qualitative zoom-in comparisons can help us readers better understand where, what, and why. 

I would say this submission has a large space for improvement before it is acceptable for publication.


correctness: Most of the claims on the dataset look good to me. The claims in the experiment section need to be revisited as some of them are not generalizable to other cases. 
clarity: The introduction and related work are ok. But the description of the method and evaluation are quite ad-hoc and lack of proper organization.
relation to prior work: I'm not completely sure but I think it differs in some extent.
additional feedback: Please refer to my weakness comments for details.

[Post rebuttal comments]
Score has been raised from 4 to 5 due to the revision. Please see my response for the new comments.
review_title: Good motivation for a new dataset but not very promising

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors proposed a new dataset called SeasonDepth (which is derived from a published dataset but with extra dense depth ground truth label provided) with multi-traverse environment for depth estimation problem and benchmarked multiple state-of-the-art depth estimation algorithm of different categories (supervised, self-supervised, synthetic domain adaptation). They used pre-trained models with weights trained on the same dataset (KITTI) without any fine-tuning for all the benchmarkings. They also provdied analysis to the benchmarked result and concluded that depth prediction has a long way to go.

The used a1 and absRel metric for base performance metric and followed-up with statistic metrics based upon these 2 like mean and variance to quantify the variability of these models performance across environments.
strengths: Although not as large as KITTI, this dataset (10G) contains a variety of environments for depth estimation for outdoor driving scene.  The process of getting groundtruth dense depth map is well-documented and the process of evaluation for benchmarking is also well written.

Significance of the contribution:
I rank the first contribution of making public a new depth estimation dataset being the most significant one. The 2nd (benchmarking SOTA algorithms) 3rd (analysis of benchmarking results) is much less significant due to concerns to be raised in "Weaknesses" sections. With the comparison to other existing depth estimation dataset shown in table 1, I believe this would fill the gap of real-world, outdoor, dense, multi-traverse, different object gap. The relative depth is a small drawback here though.

Relevance to the broader research community:
This is relevant to the broader research community and is of great interest in computer vision communities like small item detection, range variant detection and auto-dirving. With the help of accurate depth information, lots of down stream tasks can be conditioned on different depth and have better performances.

Accessibility:
I downloaded and inspected the dataset. This dataset is easily accessible and well-documented.

weaknesses: To me, my biggest concern of the paper lies in the "zero-shot benchmarking" process. 

The benchmark and the analysis following the major contribution of releasing a new depth estimation datset is formatted around a center question: Are current SOTA methods accurate and robust enough to different environment changes like season change? The author tried to answer this question by taking the pre-trained models trained on the KITTI dataset (which does not look quite similar to SeasonDepth) and directly applying it to the current unseen dataset. 

It would be fine to me if the authors are only comparing the model's robustness by simply comparing their performance (only 2 metrics, a1 and AbsRel) change when dealing with unseen domain (trained on KITTI, test on SeasonDepth), since there is one domain shift here due to "zero-shot".  To make it more specific, my acceptable argument here would be: "Here are the rank of the performance of SOTA methods on KITTI dataset (trained on KITTI, tested on unseen KITTI) on these metrics, here are the rank of the same peroformance metrics of same methods, trained on KITTI but tested on our SeasonDepth dataset. When testing on different environments, their performance downgrade differently compared to KITTI, hence they are different in robustness"

However, what the authors did (at least what I understood) was to use 6 auxiliary metrics to compare and illustrate the "robustness" in the testing domain, which is much less convincing due to there is already a large domain shift from the KITTI domain to the Pittsburg scenary.

To re-iterate, using pre-trained weights from KITTI (domain A) testing on Pittsburg (domain B) is shifting domain large enough that, vairance within domain B (lets say B1, B2 and B3 environments) is hardly captured by using the model trained on domain A data. Instead, to test the model performance robustness on different environmenet (vairance on domain B1, B2, B3), models should be trained on the held-out set of domain B.

Apart from this major concern that I have, I do not have other significant concerns or weaknesses identified.
correctness: The claims made in the submission is correct ot me. The dataset is constructed in a sound way and the benchmark is evaluated with no other concern then ones mentioned in the "Weaknesses" sections.
clarity: The paper is well written and clear. Maybe the abbreviations can be defined more clearly.
relation to prior work: The discussion of relationship to prior work is sufficient in this paper. Especially the part in section 2.1 the related algorithms. Table 1 captures the landscape of the depth estimation dataset. For the benchmarked models, as they are not the most relevant contribution, the discussion 
additional feedback: I only have that one major concern, without which I would vote for a good paper. Wish to hear what other reviewers and authors respond.

Updated comments upon reviewing rebuttal and other reviewers' comment: (Jul 19th)
Thank the authors for the extra information provided. The extra experiments are important and helpful in exclude the behavior and effect of the domain shift. However, I don't think the dataset itself, which is the major contribution here, is novel and large enough for me to raise a rating level.
review_title: Review for Benchmark under Multiple Environments

Meta-review: The paper received some critiques regarding the quality/novelty of the new dataset as well as regarding its analysis. The authors did a great job addressing all reviewers concerns. Overall, the data provided looks very useful to the community but the paper still needs some improvement to be ready for publication. Unfortunatelly it can not be accepted in its current for for round 1. We encourage authors to carefully improve current version of the paper and submit it to round 2.
Decision: Reject