Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - Introduces and makes available a long-term thermal dataset for investigating gradual concept drift. The dataset also includes additional metadata (timestamp and weather) that may be useful for a variety of associated tasks.
- Analysis of the causal relationship and correlations between the environment and performance.
- Performance of recent methods has been studied (Sec. 4.3) and analysis of performance compared with environmental variables (Sec. 5) is also analyzed.
strengths: - Makes a long-term public thermal dataset accessible for the community. Thermal imaging datasets enable personal data to be protected without requiring extensive post-processing, unlike RGB datasets. Hence, this dataset can comply with GDPR.
- Large dataset almost twice the size of the other available thermal datasets (Table 1).
- Metadata analysis is well-motivated in light of the application niche of the dataset.
- Recent deep methods have been used for the baseline experiments and the experiments are appropriately motivated.
- The drift analysis of the performance of methods compared with environmental factors is sufficient.

weaknesses: - Description of prior art seems to deliberately exclude Domain Adaptation works. DA, especially for applications in autonomous driving deal with season shifts and relevant 'concept' shifts such as time-of-day and day-to-night and widely different scenario shifts and methods, have performed well for detection and pixel-level labeling tasks. They also use real-world data such as [1] and [2]. Authors should address why and how these datasets are/are not relevant to the seasonal shifts discussed.
- Similarly, [1] and [3] also include additional metadata such as geographic location and scenes of the same location captured at various time/seasonal intervals. This enables datasets to be balanced, whereas capturing long-term videos with concept shifts at very few time intervals may lead to imbalanced datasets that may not generalize well to different geographical locations. These are relevant to be discussed here especially in light of why and what metadata is useful for concept shift identification.
  
References:  
[1] Sakaridis, C., Dai, D. and Van Gool, L., 2021. ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding. arXiv preprint arXiv:2104.13395.  
  
[2] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V. and Darrell, T., 2020. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2636-2645).  
  
[3] Sakaridis, C., Dai, D. and Gool, L.V., 2019. Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 7374-7383).

correctness: Yes, the submission is a dataset constructed in a manner useful for the machine learning community. The evaluation methods and experiments are appropriate. The dataset and the code to reproduce the experiments were shared.
clarity: Yes, the paper is clear and well written.
relation to prior work: Please refer to the Weaknesses section. Relation to prior work could include more discussions on datasets and methods that deal with similar niche outcomes albeit for a different use case (DA methods for example). 
additional feedback: L:106 imagining -> imaging
review_title: Useful thermal imaging dataset that can enable deployment of long-term solutions robust to concept drift

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a new thermal data set for video surveillance focusing on studying how ML algorithms can be affected by several environmental factors (aka as drift) when tested across multiple seasons (for about eight months). Building machine learning models resilient to drift is paramount for deploying and functioning persistently outside Petri-dishes use cases.
The authors evaluate auto-encoders, object detection, and anomaly detection algorithms for this dataset, highlighting drift effects.

strengths: Consistent dataset across 8 months along with the metadata. 
weaknesses: The literature review feels very short and incomplete as it does not account for many representative datasets in robotics that have been studied the drift concept for many years already: BDD 100k, Oxford Robot Car Seasons, CMU Seasons, DENSE, Ford Multi-AV Seasonal, FLIR dataset, etc.
More comments about how the data format is would be appreciated. For example: how does the information from the IR sensor compare to RGB? is it an 8-bit grayscale image with corresponding intensity values for temperatures?
No comments on how the algorithms used are adapted to IR images are given. It is briefly mentioned that the object detectors used have already been tested on thermal images. Is this the same case for the anomaly detection and the outlier detection methods? If they already have been used previously, please provide a brief description or intuition of what changes or adaptations were necessary for the algorithms to work.
Line 204-206: This needs to be reworded either to make it a hypotheses to be proven or as a conclusion from their result. 
line 218: 100 frames is incredibly small for such network models for any significant learning and the pretraining details are missing. 
Line 239-241: What about camera movement from wind? That would make the mean value of the difference ineffective as a measure of activity in the scene. 
Some analysis ensuring no overfitting of the models is occurring would help to strengthen the effectiveness. 
Last paragraph of the conclusion: It is stated that the proposed dataset can help researchers understand the causes of drift. But with the lack of a metric of how much drift is occurring this exercise will be half successful. Some indication of the quantity of drift will be helpful. 
Appex Dataset supp material: Section 1.5: This cannot be used for effective tracking due to the frequency of the data collected and long term nature of data. (2 min clips)

correctness: There is a consistent process with regards to timing, frequency of the data collected. That seems sound, but lacks on details as mentioned above in weaknesses to show the completeness in the process. The Kaggle page also lacked in terms of details. 


clarity: Yes, in terms of language. Minor edits here and there. 
relation to prior work: See weaknesses. 
additional feedback: N/A
review_title: Seasons in Drift: Intent to provide a thermal imaging dataset across 8 months

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors present a novel dataset to study the effect of data distribution drift on model performance. The dataset consists of thermal video taken across months, metadata about the weather, day/night cycle, etc. The authors benchmark many model types to understand the effect of time of year data drift on the model performance. Lastly, the authors study the correlation between model performance and metadata.

The paper's main contributions are a novel thermal imaging dataset taken across many months. The datasets come with metadata about the world state during the video capture. Lastly, the dataset comes with some bounding box labels for object detection tests.
strengths: -- The paper presents a novel thermal imaging dataset spanning many months. The dataset contains metadata that allows researchers to study the correlations between meta-features and model performance. 
-- The paper also has some bounding box labels allowing for some analysis of relationships between task diversity and concept drift.
-- The paper performance a few benchmarks on the correlation between concept drift and model performance. 
weaknesses: -- Due to the limited amounts of object detection data, this dataset will only enable model-centric methods to handle concept drift instead of data-centric methods.
-- The limited scene diversity also limits the dataset to studies of data distribution shifts caused by weather, day, and time of year variations.
correctness: The claims of the paper are mainly correct.
clarity: The paper is well-written and straightforward. However, some aspects of the presentation can improve. Specifically, the labeling and subset information can be moved out of the experiments section and added to the dataset section. 
relation to prior work: The references to prior work appear adequate.
additional feedback: none.
review_title: A dataset to study the role of seasonality, time of year and day distribution drift on model performance

Meta-review: All reviewers recommend acceptance. Congratulations to authors! I urge the authors to incorporate reviewers' suggestion in the final version of the paper.
Decision: Accept