Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors are presenting 2 spatio-temporal datasets taken using remote sensing satelites, The authors describe the methods for annotating change events, and a pipeline the authors used to automate the annotation of the dataset.  
strengths: The spatio-temporal dataset presented by the authors are novel, and has value in itself especially for researches working on object detection, anomaly detection, change detection, online learning, and remote sensing, among others. While I have issues with the writeup, I see value in the dataset given the lack of spatio-temporal datasets (see review section: discussion on previous work)
weaknesses: The fatal weakness to the paper is the description of the baselines in particular the "the baseline "Pretrained imagenet"".Without the architecture of the pretrained network (I am assuming that it is a neural network) that description is uselss. Also the note that "No Training involved" appears impossible; ImageNet is a 1000-class problem, and without fine tuning what exactly is the "pretrained network" trying to discriminate?

We need more clarification regarding this.

Also, the large difference between the recall and F1-scores seem to indicate that that the baseline methods have very poor precision and produces a large number of false positives. Can the authors provide more insights on the false positives and if they all fall into some natural grouping (seasonal change, shadows in the picture, occlusion, etc?) 

I would also have liked to see more detail on the self-supervised change detection network, namely some example images of that were considered "similar" 
correctness: The dataset appears to be constructed in a sound way, however I am concerned about the high number of false positives reported in the baselines. Without a breakdown of the nature of the false positives, it is hard to tell if there is a systemic problem in the dataset, or the false positives comes from the weakness in the baseline methods and the dataset reveals areas where improvements can be made to the baseline algorithms. 

I have issues with the benchmarks, elaborated in the previous section. 
clarity: Unfortunately, as noted in the previous sections, there are a number of details that are unclear in the paper, namely the imagenet Baseline method, the cloud processing, some discussions on the false positives, 
relation to prior work: The authors did not reference other spatio-temporal datasets. To be fair to the authors, to the best of my knowledge there are not many publicly available spatio-temporal datasets. Off the top of my head, the only ones i know of are Pheneo4D (temporal 3d scans of maize growing, https://www.ipb.uni-bonn.de/data/pheno4d/ , Soil-moisture of the Tibetan Plateau https://doi.org/10.6084/m9.figshare.7996448, and Global Spatiotemporal data for 2019-Novel Coronavirus Covid-19 Cases and deaths, https://data.humdata.org/dataset/2019-novel-coronavirus-cases, all of which are fundamentally different from the dataset presented. 

A more related dataset would be the Borneo forest logging dataset, however, to the best of my knowledge it is not publicly available to researchers. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0101654



additional feedback: While I like the dataset, and I do see the value the dataset would add to the machine learning community, the write-up in particular the benchmarking process requires more work in particular the description of the baselines. While I am inclined to accept the submission on the merits of the value of the dataset, I do think that more work is needed in the writeup as noted above
review_title: Good data set, paper needs work 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Overall. this paper presents interesting work + a dataset, but it is not described in a way that gives the reader a clear view of (1) what's being presented in terms of the dataset, 2) what portions of the methodology are novel, and (3) why certain decisions were made by the authors. 

Furthermore, the abstract and the conclusion need to be reworked, and the reporting of findings needs to be more quantitative. 

With additional work, this will be a nice submission for a future conference. 
strengths: The authors take a compelling approach to a very relevant problem: change detection is crucial in the field of EO, and self-supervision is increasingly being used to distill relevant information within large amounts of imagery.

The authors are obviously very comfortable with implementing novel computer vision algorithms. 

The presented dataset contains labels collected via hired enumerators, which would have been an expensive and time-consuming task. 

CaiRoad is a great name. 
weaknesses: No code is available for the submission. 

The abstract, conclusion, and reported results need to be more quantitative. 

There are not enough descriptions for why certain choices were made (see Additional Feedback for more specific comments). 

Overall, too much information is packed into too few pages. I understand the page limit is restricting, but the authors need to determine a way to communicate the paper's most crucial information more effectively, while cutting out unnecessary portions. 
correctness: As far as I can tell, everything in this submission is correct. The dataset is constructed in a sound way, and the models seem to be implemented correctly. 
clarity: The writing quality of this paper is okay. There are no major issues, but there is lots of repeated information and sentence structures. 
relation to prior work: This is a portion of the paper that can be improved. Self-supervision is being adopted for more and more EO tasks, but the authors do not discuss recent efforts to apply these computer vision-developed approaches to the realm of remote sensing. All discussion of self-supervision methods + papers pertain to non-EO tasks. 
additional feedback: The following comments are presented in rough chronological order.

1. Abstract: What is the dataset size? How many images are contained? What spatial locations are covered? How do benchmarks perform overall? All of this information is crucial to understanding the contributions of the paper, and needs to be specified. This section should be much more quantitative. 

2. Introduction: Italicizing and bolding is not necessary in the introduction. I would remove >95% of these font weights, except when you think absolutely necessary. 

3. When you described a new self-supervised approach for change detection, is it a completely new approach, or just applied to a new problem here?

4. What type of imagery are you using? Need to state in the abstract or introduction. 

5. Lines 87, 89, 91, e.g.: Don't use grouped citations. Split these references up so it's clear what the major contributions are. 

6. There are lots of papers on change detection in EO -- An overview of relevant, major papers, and how the field has evolved would be very useful to readers. 

7. Lines 103-108: Need more information on what's contained in these other datasets, i.e. the size, spatial extent, imagery, labels, resolution, uses. A table would be effective for this summary. 

8. Lines 109-116: See comment above under "relation to prior work. Use references for EO work instead. 

Figure 1, sentence 3: Typo/missing word. Caption does not fully describe the different portions of the figure. 

Figure 2: Similarly, the caption does not fully describe everything in the figure. Many variables are undefined. 

9. Line 120: Sec and all references to tables/figures/etc. must be capitalized. There are many other places where this applied, but I'm only listing the first. 

Section 3.1: It seems like this section could be distilled to a single sentence, defining change events as pixel-level reflectance changes. I also don't see how this term materially differs from a common understanding of "change". As a result I don't see how formally defining it is a major contribution of the paper. 

3.2: Needs to be condensed, or made substantially more descriptive. What imagery is being used?

10. Line 142: Here, you describe that feature representation of pixels is determined in an unsupervised manner. In sentence 2 of Figure 1's caption, it says that the feature representations are determined in a self-supervised manner. 

11. Line 167-168: Why do you need a feature space representation of each pixel instead of a representation for entire image? The same type of contrastive loss can be applied to a full image for unsupervised learning to learn appropriate embeddings. It's possible I'm missing something here, and if so, the case for pixel-level feature representations needs to be made explicit. Explicitly defining feature maps and feature vectors here will help. 

12. Line 196: Need to define voxels. 

Section 3.2.1: It is unclear what portion of this self-supervised approach is novel. Contrastive losses have been applied in the literature before, as you mention for the SimCLR. 

13. Line 188: Is there one threshold for all change detection? Or a separate one for each class?

14. Line 192: How do you prove that you're capturing the correct invariances?

15. Lines 201-2: How are these parameters determined? Need an explanation of how + why.

16. Line 208: How much S2 imagery is used? Need a description on the frequency/extent/etc. Also, why do you not use the near-infrared band? This is the band most common used in burned area detection approaches. 

17. Line 222: Similarly, why do you not use the shortwave infrared bands? These bands will be particularly useful for road detection. 

18. Lines 238-241: How is road change defined to enumerators? As state previously, change may be partial and may extend over multiple timesteps. 

Figure 3: There is not enough information in the Figure caption to understand what's being presented. I see the note about more information being contained in the Supplement, but you can't introduce a figure in the main text and not describe it in the Supplement. 

19. Line 261: Why is this for fairness? Explain. 

20. Line 255: OSCD is used for unsupervised learning and benchmarking. Why is all train + test data used for training? What is performance measured over then? 

21. Line 268: Need to explain why you're using an estimate here and how the estimate was calculated if you're going to use it as one of your main reported findings. 

22. Line 297: Need more explanation of how features are averaged across time. Won't this flatten the necessary time dimension for change detection? 

Conclusion: Like the abstract and introduction, needs to be more quantitative for both the dataset + benchmarks.
review_title: Review for Change Detection Dataset

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work introduces two datasets: Calfire and Cairoad that have been refined from about 6 years of publicly available Sentinel-2 RGB imagery. By developing an unsupervised feature learning technique the authors were able to segment regions of change over time, which are highlighted and provided as binary maps between consecutive images. Moreover, using the groundtruth of Calfire, OpenStreetmaps and crowdsourced human annotations, a semantic label has been assigned. The changing pixels together with the semantic label constitute a "change event" which the authors introduce in contrast to the more common pair-wise "change detection". The proposed methods for dataset generation are shown to be efficient and less time-consuming then similar approaches. Lastly, the value of the Calfire and Cairoad datasets as benchmark for the developing of semantic retrieval classification systems is demonstrated.
strengths: 1) Both datasets are indeed unique in the sense that not only pairwise changes but changes of largely varying spatio-temporal scales are available. This allows for a meaningful semantic separation of changes that occur on our planet and is valuable for the development of automated techniques.

2) The description of the datasets in datasheet format, the supplementary material and the webpage is done well, without any unnecessary baggage. The format of the data includes the change events and the full image files of Sentinel-2 along with identifying information that is ready to use and easy to grasp. If I was to develop some technique related to the monitoring of change, I believe this would be an excellent starting point to me.

3) The technique to generate the two datasets is of value and could be readily deployed to generate similar datasets at larger scale.

4) The selected events of road construction and forest fires especially are also of larger relevance (outside the development of change detection technologies) as they relate to important global issues, i.e. climate change.
weaknesses: 1) The authors highlight on several places (for example line 119) how their approach generates change events without any supervision. While it is true that feature learning and grouping of pixels is unsupervised, the assigning of the semantic label to finalize the "change event" is still left to humans.

2) The corresponding labels for the change events are also binary in nature: within each of the two datasets, only a binary decision, e.g. "forest fire? -> yes/no" has been performed and every event labeled "no" is not further described.

3) The pipeline for generating the dataset requires hyperparameter tuning, which the authors acknowledge as a limitation. This might offset the displayed good performance (in terms of quality, but also runtime) somewhat, as multiple trials and tinkering might be required, to achieve a decent result.

4) The fairness in comparison to competing methods is a little questionable (see "Correctness").
correctness: Since most (if not all) methods in comparison for the change event representation learning (Section 5.2) are not specifically designed to deal with the change events as formulated by the authors, they are necessarily at a disadvantage. It can not be expected that they can compete if temporal information is simply averaged out over features against a method that makes full use of it.

When it comes to the change detection itself (Section 5.1) the tuning and parameterization of the competing methods is not described so one should be critical about the reported runtimes and (albeit to a lesser extent) also about the reported quality metrics. 

While it is clear that the proposed method of the authors can be tuned to be competitive on the OSCD benchmark, it is not given that this performance will generalize to other sources of change/data. Investigating the supplementary material reveals that the authors also compared on the LEVIR-CD dataset, which increases a little the trustworthiness of their results.

More difficult to evaluate is the actual quality/correctness of the detected change events in Calfire and Cairoad, as no human groundtruth of the change segmentation is provided and it seems reasonable to assume that certain change events go undetected or being false positives. In particular puzzling is the statement that 1076 location of fire incidents were reported by Calfire (line 213) while only 204 fire events are positives in the constructed dataset? This is only 20%? I would be curious to learn more about this discrepancy.

From eye-balling some of the change events in the provided data, it seems the segmentations are on a decent enough level to be useful and not necessarily misleading. Given, that the authors also provide the original images along their refined change events, this allows also for evaluations on those quality questions in the future by other independent research groups. If better methods are developed, the revision of the change event portion of the data seems rather straightforward, so I don't consider it a major issue, just something that users of the dataset should be aware of.
clarity: The paper is well written and easy to understand. The supplementary material is helpful and elaborates on interesting details.
relation to prior work: The paper is set into the appropriate context when it comes to recent work in change detection.
additional feedback: Nitpicking time: in line 243 there is a "to" missing.
Figure 4 would benefit from a larger size but I acknowledge that NeuRIPS page limitations are an issue here...

Final recommendation: please make a small sample of a few change events from each dataset available for download, so one can get a first impression without downloading several GBs.
review_title: Useful datasets for studying spatiotemporal change in satellite imagery

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces the concept of "change event" and presents an unsupervised method for change event detection. Two benchmarks for road construction and forest fires are created and evaluated.
strengths: 1. The concept of "change event" is an interesting new idea in remote sensing change detection.

2. The change event detection pipeline (including the unsupervised change detection method) is clearly described.

3. The paper and dataset documentation are well written.
weaknesses: 1. The proposed self-supervised learning based unsupervised change detection method is not very novel.
2. The proposed datasets lack confidence analysis about both the event type label and the event pixel coverage. The event type can be rather accurate with human-like annotation, but the event pixels I am not sure. The results on OSCD dataset show an F1 score of only 0.3, which is not very good.
3. The code is an important component of this paper as it proposes a dataset generation pipeline, but it is still unavailable.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: I like the idea of grouping changing pixels into change events and the paper is very well written. However, If seeing it as a dataset or benchmark paper, only the event type is (able to be) benchmarked. The other very important component from my point of view, the changing pixels, are not confident and can not be evaluated as there's no ground truth label. My evaluation of this paper is not very certain, and I look forward to author responses about my concerns.
review_title: Interesting change event benchmarks and datasets

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The submission introduces the **novel concept of “change events”** in satellite images, together with a **scaleable (semi-)automatic pipeline to curate future change detection datasets**. To illustrate the feasibility of their approach, the authors curate **two new change detection datasets** (California wildfire & Cairo road construction). Besides, the developed pipeline makes use of some minor but handy methodological tricks, like learning photometrically invariant & geometrically equivariant representations to deal with domain- specific noise and applying SimCLR to feature maps. Key contributions are highlighted in **bold** font.

I judge this to be a strong submission and recommend for publication. The work convinces with a sound and interesting concept, supported by solid experiments. My review highlights a few points that would be beneficial to have discussed and raise awareness of, but they are relatively minor concerns that do not overly affect my positive view of the submission. I am familiar with the subject, and the conducted experiments as well as the provided results support the certainty of my assessment.
strengths: ### Significance:
* The work is introducing, formalizing and addressing the interesting new concept of “change events”, rather than solely providing incremental improvements over existing tasks or datasets.
* The work proposes a pipeline of generating novel datasets, rather than solely releasing new data. This is promising, as it allows the audience to curate their own datasets based upon the proposed approach. The authors demonstrate their pipeline by collecting and releasing two small Earth observation datasets for change detection themselves.
* To my best understanding, the proposed approach should be sufficiently generic and domain agnostic, such that it may extend beyond satellite imagery and also be applicable to e.g. co-registered repeated measures in biomedical imaging and other fields.

### Relevance:
* The presented generative approach eases the burden of manually labeling data. (Semi-) automating the annotation process in creating datasets, in particular with respect to pixel-wise annotations, is much welcome.
* Individual components of the overall pipeline are properly benchmarked, outperform existing baselines in terms of performance and scalability. The latter is important for any exploratory (semi-)automatic data curation process in order to traverse through large data bases.

###  Accessibility and Accountability:
The code will be released in a public git repository (not released at the time of this review), the two curated datasets are hosted on a university server for public access. For the proposed pipeline to prove beneficial to the community, it will be critical the released code (underlying the proposed dataset generation process, the submission’s main contribution) is be user-friendly and allowing for a flexible usage on diverse raw data.

###  Ethical and social implications:
The authors discussed potential concerns of eventually detrimental societal impact. I do agree with the authors’ assessment that their choice of medium-resolution satellite imagery over the selected areas minimizes potential violations of citizen privacy and do have no further concerns beyond this matter.
weaknesses: ###  Significance:
* The notion of “change events” and their self-supervised identification is primarily of interest during the process of dataset creation. Once dataset curation is accomplished, the inferred change maps are of their assigned semantics and can, in terms of their structure and ideally annotation quality, hardly be differentiated from conventionally obtained and manually annotated change maps.
* The two provided change detection datasets are mono-class. In theory and to the best of my understanding, the pipeline should be applicable to multi-class change events (i.e. mutiple event types within a single area). Fig. 4 in the supplementary material highlights a sample of different change type segments. However, no further experiments on multi-class change event detection are conducted in the submission, so I am not too certain about its effectiveness. Further comments by the authors on this matter may be insightful and welcome.

### Relevance:
* The two released datasets, curated for the sake of demonstrating the proposed change event discovery method, are very specific in terms of their locations rather than providing global coverage. While this may raise doubts about the representativeness and generalizibility of benchmarked models, it is of limited concern to me as the submission’s main contribution is in the dataset generator method.
* To my understanding, the proposed approach should be sufficiently generic and domain agnostic, such that it may extend beyond satellite imagery and also be applicable to e.g. co-registered repeated measures in biomedical imaging and other fields. While I consider the submission a valuable contribution for the Earth observation community, to go beyond and extend to other domains may have further increased the submission’s relevance. I would appreciate the authors comments about the feasibility of this, and potential limitations or obstacles when considering other domains.

### Accessibility and Accountability:
* My main concern is about the feasibility of hyperparameter tuning during change grouping, as this poses a risk to the general applicability of the proposed method to diverse and heterogeneous scenes or events. The authors are very honest and forthcoming about this limitation tho and acknowledge it in the main text. However, I’ll raise some questions that I consider important for the audience to assess the methods’ feasibility on a future user’s data of interest:

    a) How did you explore and choose the hyperparameters in equation (2)? Specifically, how were values such as the 20 voxels spatial thresholding, as denoted in section 3.2 of the supplementary material, chosen?

    b) How easy or sensitive are the hyperparameters to tune? The submission proposes a single hyperparametrization applied onto both sample datasets. But to which extent did you encounter different optima in between your two proposed datasets? 

    c) How is the seed pixel, as mentioned in [p.6, l.198] of the main text, set?

* The image data are provided as jpg / png. This is a rather unusual format for satellite imagery, which is geo-referenced and typically provided in raster format (e.g. [*.(geo)tiff](http://docs.opengeospatial.org/is/19-008r4/19-008r4.html)). This somewhat goes against conventions, making the provided datasets harder to handle by conventional and established manners. However, the authors do provide meta information (pertaining to acquisition dates and area observed), see [here](https://www.cs.cornell.edu/projects/satellite-change-events/static/data/Readme.txt). Nonetheless, I propose addressing the above issue by also releasing the datasets in raster format.

* It is surprising that Sentinel-2’s SWIR band is not utilized in generating the CalFire benchmark, given that infra-red bands are particularly beneficial detecting thermal activities. This raises some questions:  Is there any reason your proposed pipeline may not be applied on the entire spectrum of data? Did you attempt running your pipeline on all Sentinel-2 bands, and if so, did you observe any material difference in performances? Did the benchmarked baselines have access to the full multi-spectral information, and if not, is there any reason to suspect running them solely on the RGB data may put them at a disadvantage? What’s the differences in empirically measured computational run times, when including all bands?

### Ethical and social implications:
None.
correctness: Yes to all of the above assessment criteria. But please check to always make explicit that the dataset generation method is semi-automatic. This is not handled entirely consistently throughout the main text, see:

	“[...] we propose a semi-automatic approach: we detect change events automatically, and then ask human annotators to label them with semantic categories.” [p.2, l.40-21]


	“We present a novel method to automatically create a dataset of change events from spatio-temporal satellite images.” [p.2, l.68-69]
clarity: The paper is well written and clear.
relation to prior work: Relevant publications are cited. Closest related is probably prior work on open-set self-supervised change detection, as in [35, 40]. Related methods are referenced, compared and benchmarked against their corresponding counterparts in the proposed pipeline.
additional feedback: What follows is mostly questions (sorted descendingly by importance), who might be of potential interest for the audience:

* How would the two closely related (but contrary) change events of “crop growth” and “crop harvesting” (example from the supplementary material) be handled? Or alternatively, the change events of “receding water level” (example from the supplementary material) and “increasing water level”? Would each pair be summarized into a single change event cluster? If so, how could automatic clustering differentiate between their opposing directedness? Is there any potential way to account for such kinds of causality?

* It would be interesting to run the proposed pipeline on recently published change detection datasets (both released after or close to submission deadline) such as [1, 2], in order to compare change map agreements and evaluate how close the proposed method comes to human annotations.

* Section 2.4 "Preprocessing/cleaning/labeling" in the supplementary material specifies that cloud-covered pixels are filtered. How is smoke in the curated forest fire dataset handled?	

[i] Cha, M., Huang, K. W., Schmidt, M., Angelides, G., Hamilton, M., Goldberg, S., ... & Piou, J. (2022). MultiEarth 2022--Multimodal Learning for Earth and Environment Workshop and Challenge. arXiv preprint arXiv:2204.07649. https://sites.google.com/view/rainforest-challenge

[ii] Toker, A., Kondmann, L., Weber, M., Eisenberger, M., Camero, A., Hu, J., ... & Leal-Taixé, L. (2022). DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21158-21167).
review_title: A strong contribution, providing a generative approach do semi-automatically curate novel change detection datasets

Meta-review: metareview: This work presents a novel resource for detecting changes in aerial imagery. The paper is well written and contains significant amount of experiments and baselines for demonstrating its usefulness. The paper considerably improved upon reviewers feedback who are unanimously positive about this work. 
confidence: 5: The area chair is absolutely certain
Decision: Accept