Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: How is the 'mean average recall' (mAR) metric reported in table 4 defined? (I've never heard of this metric before, and it's not obvious to me what it would mean -- a direct analog of Average Precision would not be well-defined.)
review_title: mAR

final_score: rating: null
review_content: In the paper, you consider detections with a IoU of 0.1 or greater to be true positives. Could you discuss the choice of this threshold? How do your results change as this threshold is increased (i.e. when more accurate bounding boxes are required)? 
review_title: 0.1 IoU Threshold

final_score: rating: 6: Marginally above acceptance threshold
review_content: [EDIT: The thoughtful author responses addressed my major concerns. The github links for data and code will be really helpful for reproducing results (I haven't looked carefully, but this is great). The revision addressed many issues, including the additional results. As such I am upgrading my rating from a 5 to a 6 and recommend acceptance of the paper.]

The paper proposes to apply deep nets to perform detection and localization of extreme weather events in simulated weather data. The problem is related to object detection in computer vision in that the input is a 2D “image” (multichannel spatial weather data) or 3D “video” (temporal version of the data) and the output is a bounding box (spatial-temporal localization of a weather event) and class label (weather event type). It differs from standard object detection in that the input has multiple heterogenous channels and labeled data is scarce.

A simple but quite reasonable deep net is proposed for the task based on similar approaches in computer vision. While proposal based systems are most popular in vision currently (in particular Faster-RCNN) the proposed approach is simple and a fine starting point. There is little innovation on the part of the detection system, but as noted, it is a valid application of ideas from computer vision to the task at hand. 

The authors propose both a supervised approach (only ground truth bounding box location/label is used) and a semi-supervised approach that additionally incorporates the reconstruction loss as a regularization. In all cases the losses are fairly standard and again, reasonable. The only confusing bit is that the “semi-supervised” loss actually has all the labels used for the “supervised” loss and additionally incorporates the reconstruction loss. Hence, the “semi-supervised” loss is actually stronger, which makes the terminology a bit confusing.

The paper is easy to follow, but notation is sloppy. For example, above equation 5 it states that “the loss is a weighted combination of reconstruction error and bounding box regression loss”; actually it’s a combination of the supervised and unsupervised loss (Lsup and Lunsup), and Lrec is not defined (although I assume Lrec=Lunsup). The paper is fairly non-technical, but nevertheless these minor issues should be fixed. (E.g., see also reference to “figure 4 and 4”). 

The biggest concern w the paper though is experimental results. Only a single figure and table of results are shown (figure 4 and table 4). The metrics are not defined (what is mean average recall?). Only 2D versus 3D version of the model are shown, and supervised and semi-supervised. Moreover, numbers seem a bit all over the place, without consistent patterns (e.g., why is 2D supervised better than the seemingly much strong 3D semi-supervised?). One of the things that is unclear is how many events are actually in the training/testing data, and more importantly, how good are these results in absolute terms? Regardless, the experiments are fairly sparse and ablation studies and more discussion lacking. It is also unclear if future researchers will be able to reproduce the experimental setting (a commitment to open-source the data or a way to reproduce the experiments would be critical for future authors).

Minor nit: the authors use both a classification loss and an “objectness” loss. I’ve never seen both used together like this (normally objectness is used in two-stage object proposal systems where in the first stage class-agnostic proposals are given and in the second stage these are cropped and a class-specific classifier is applied). I strongly suspect removing the objectness loss would not impact results since the classification loss should provide strictly stronger supervisory signal. Regardless, this is a fairly non-standard choice and should be justified (experimentally).

Overall this is a borderline paper. I do believe that it is valuable to apply computer vision techniques to a domain that I’ve see little work on in our community. That being said, I have no expertise on this type of data -- it’s possible this deep learning techniques are now routinely used in the climate science literature (I suspect not, though). Overall, there is little novelty on the algorithmic side in this paper (the equations in section 3 are commonly used in the cv literature). The use of reconstruction loss to improve results in the data-sparse setting is interesting, but the experimental results are inconclusive. The experimental validation is generally insufficient. Reproducibility for future research is difficult unless the data is open-sourced. Overall, I think this paper is a good start, and with improved experiments and more careful writing I think could eventually make for a decent paper.
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper applies convnet-based object detection techniques to detection of weather events from 3D climate data, additionally exploring the effect of using an unsupervised autoencoder-style objective term.


Pros:

The application of object detection techniques to extreme weather event detection problem is unique, to my knowledge.

The paper is well-written and describes the method well, including a survey of the related work.

The best model makes use of 3D convolutions and unsupervised learning, both of which are relatively unexplored in the detection literature.  Both of these aspects are validated and shown to produce at least small performance improvements over a 2D and/or purely supervised approach.


Cons:

The benefits of the 3D convolutional architecture and unsupervised learning end up being a little underwhelming, with 52.92% mAP for the 3D+semi-sup result vs. 51.42% mAP for the 2D+sup result.

It’s a bit strange that 3D+sup and 2D+semi-sup are each worse than the 2D+sup base result;  I’d expect each aspect to give a slight improvement over the base result, given that using both together gives the best results -- perhaps there was not a thorough enough hyperparameter search for these cases.  The paper does acknowledge this and provide potential explanations in Sec. 4.3, however.

As other reviewers pointed out, the use of the 0.1 IoU criterion for true positives is very loose relative to the standard 0.5 criterion.  On the other hand, if the results visualized in Figure 3 are typical, a 0.1 overlap criterion could be reasonable for this domain as the detector does seem to localize events well enough that the system could be used to expedite human review of the climate images for extreme events.  Still, it would be useful to also report results at higher overlap thresholds.

Minor: eq 6 should (probably) be the squared L2 norm (i.e. the sum of squares) rather than the L2 norm itself.

Minor: table 4 -- shouldn’t the semi-supervised models have more parameters than the corresponding supervised ones due to the decoder layers?


Overall, this paper is well-written and applies some interesting underutilized techniques to a relatively unique domain.  The results aren't striking, but the model is ablated appropriately and shown to be beneficial.  For a final version, it would be nice to see results at higher overlap thresholds.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work presents a novel 3D CNN architecture for climate event detection that combines an unsupervised auto-encoder reconstruction loss with YOLO like bounding box prediction. The approach is trained and evaluated on a large-scale, simulated climate dataset labeled by a costly heuristic approach called TECA. 

For the most part, the paper is nicely written (minor comments below) and addresses an important and well motivated problem. The authors provide sufficient model details to allow reproduction (although public code would be preferred). I find the experiments a bit unconvincing (see below) but appreciate the attention to model capacity (via number of parameter) when comparing the 2D and 3D model variants.

I am concerned that the evaluation may be insufficient to assess the effectiveness of this method. An IoU threshold of 0.1 allows for many rather poor detections to count as true positives. If the goal of this work is to count the number of such instances this is defensible, but for localization this seems overly loose.Furthermore, the 3D CNN architecture (which is one of the the core novelties of this work compared to past efforts) does not seem capable of producing variable sized boxes (as noted in the last paragraph of page 7), which I imagine results in poorer performance at higher IoU thresholds (as many of the weather events appear small). 
The experiments also feel inconclusive about the effect of temporal modeling and semi-supervision. The temporal component does not seem to matter in the supervised settings (2D 51.45 mAP - 3D 51.00 mAP) but improves somewhat in the semi-supervised case (2D 51.11 mAP - 3D 52.92 mAP). Whereas the additional unlabeled data seems to hurt in the 2D case but improve results for the 3D model. Could the authors provide confidence intervals for these numbers? I would like to see further discussion of these trends especially with respect to the effect of the loss weights (alpha, beta, and gamma). 

I also note that it is not clear if both the 2D and 3D models were trained for equivalent time periods (seems like no from last paragraph of page 7). Could a plot of training and validation accuracy for each model be presented for comparison? 

Finally, is there any baseline approach the authors could report or compare too? Without one, it is difficult to evaluate the performance of the approach with respect to the difficulty of the problem.

Preliminary Rating:
I think this is an interesting paper that is well motivated but feel the experiments as presented do not seem adequate to support any conclusive trends. I would like to see the mAP trends across a wider range of IoU values and further discussion of training procedure, loss weight settings, and reasons for lack of bounding box variability in the 3D model (as stated above). 

Clarification:
In the paper you say "While climate models are run on a 3D grid, with the vertical dimension corresponding to 30 levels; we only consider surface quantities (i.e. 2D data) in this study." Could you elaborate on what the surface quantities correspond to? Is it the highest cloud level?

Minor notes:
	Please provide years for Prabhat et al. references rather than a and b.
	Footnote in 4.2 could be inline text with similar space.
	4.3 second paragraph the word table is not capitalized like elsewhere.
	4.3 4th paragraph the word section is not capitalized like elsewhere.

Edit: I appreciate the authors responding to my questions but still feel the relatively poor localization performance at stricter IoU thresholds fails to justify the complexity of the approach. I encourage the authors to continue pursuing this line of research.
review_title: Review

Meta-review: The paper addresses the little explored domain (at least from computer vision perspective) of analyzing weather data. All reviewers found the application interesting, yet felt that the technical side was somewhat standard. On the positive note, the authors try to use semi-supervised techniques due to limited labeled data, which is less explored in the object detection domain. The also released code, which is a plus. Overall, the application is nice, however, none of the reviewers was swayed by the paper. I suggest that the authors re-submit their work to a vision conference, where applications like this may be more enthusiastically received.
Decision: Reject