Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: =========== Update after rebuttal

Thanks for the clarifications and the update. I recommend acceptance of the paper and updated to 8.

Last comment: please still improve the appearance of Figure 4 by using a more diverse set of marker shapes as well as overlay and offset tricks -- see https://www.cs.ubc.ca/~schmidtm/Software/prettyPlot.html for an example. 

============

This paper introduces a new convolution operator (D-conv) specifically tailored to model point-cloud data evolving over time, i.e. a set of n points with features and localization-coordinates that can evolve over time. The main idea is to use the k-nearest neighbor structure for each point to get a fixed size k window to use in the convolution to determine the new location and feature values of a point (and a permutation-invariant operation). The D-Conv operator is included in a LSTM architecture (CloudLSTM) to enable the spatio-temporal modeling of point-cloud data, and can be combined in standard neural network architectures such as a Seq2Seq with attention. This is in contrast to previous approaches which modeled the data on a grid through preprocessing, or did not include the temporal component for cloud data. Experiments is conducted on 4 benchmark datasets, covering two point-cloud stream forecasting applications, showing how CloudLSTM give lower prediction error than numerous baselines and alternatives.

While the D-Conv idea seems fairly simple and natural, it is novel AFAIK and fairly appropriate to model point-cloud data streams. The approach is well situated in the literature, and the experiments are indicative that this method can improve on the current approaches. I am thus leaning towards accept.

The paper is fairly clear, though the notation is a bit confusing and somewhat sloppy (see detailed comment below).

Important clarification requested: the current notation suggests that each channel could have a different location for a point p_n, the K nearest points seem to be defined irrespective on the channel. So is the location fixed across channels; or does this paper allow the neighborhood structures to vary across channel?

== Other detailed comments ==

- p.3 Q_n^K -- it seems it would be more appropriate to define it as an ordered list of k points (rather than a set, as this would loose all information about the order); unless you append a new dimension to each point where you put the ordering information there for the purpose of defining the k points in Q_n^K.

- (2) the notation is a bit weird and overloaded for the summation (without being defined). Examples include "i in U1" (when U1 is an integer, not a set); "p_n^k in Q_n^K" when p_n^k does not appear in the summation (a clearer alternative would be using the notation v(p_n^k)_i^h for the h^th value of channel i of point p_n^k, e.g.; now p_n^k would indeed appear in the expressoin); "v_n^h in v_n" -> why not just summing over h as it is really doing? Etc.!

- (2) S_out^j: each p_n^' should be a *tuple* (not a set like currently written).

- Figure 4: the lines are really hard to distinguish just by the similar colors -- please use different markers for the different lines (and offset the marker so that they can be seen)

- Several neighborhood sizes are experimented with. Note though that smaller neighborhood sizes are just *special cases* of bigger neighborhood sizes (by using zero weight on the last few neighbors in the convolution). Wouldn't it make sense to use a big neighborhood size and regularize in some way the weights for the further neighbors?

- Table 2: for SSIM, there are two rows with 0.69 +/- 0.07 (minimal value) -- they could be both bolded.

- Appendix B, they claim that the complexity of finding the K nearest neighbors (in dimension L for n points) is close to O(K L log(n)) if using KD trees. I vaguely recall issues in high dimension though (in particular that the above complexity is only valid for specific distributions of points in low dimension). E.g. see https://en.wikipedia.org/wiki/K-d_tree#High-dimensional_data where it is mentioned that L << log(n) is normally needed to guarantee efficiency. The claim should properly be nuanced.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposed a new convolution operator, named dynamic post-cloud convention over spatiotemporal data, and the convolution operator can be embedded in different neural network architectures, like recurrent neural networks. In order to achieve the convolution over point-clouds by using both value features and the spatial-features, given a data point, the convolution is conducted over its k-nearest neighbors generated by CNN.  They compared the proposed convolution method by embedding it into RNN, GRN, and LSTM against a number of existing methods on two datasets, in terms of MAE, RMSE, PSNR, and SSIM. Overall, this paper is interesting but needs some clarifications on 

1. Given that the proposed convolution operator use KNN to choose the nearest neighbors. It would be good to empirically to study how K would affect the performance, does it data-dependent
2. Is it possible to study the time complexity for various models?
3. Table 1 and table 2 seem to show that the proposed convolution operator contributes to the performance in terms of the mean of each metric. It might be good to do further oblation test to study which mechanism actually contribute to the performance. The choice of RNN, attention, or the new operator? Furthermore, the std is quite large, which makes one wonder if the improvement is statistically significant.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Review Summary
--------------
Overall this is almost above the bar for me to accept, but I think there's enough concerns about the method and experiments that I'm hesitant. Strengths include the invariance to point cloud order and the relative simplicity of the architecture (compared to PointCNN). Weaknesses include a vulnerability to outliers, experiments that don't seem to think about practical effects like day-of-week in forecasting, and experiments that leave out baselines to help directly assess the impact of neighbors.


Paper Summary
-------------
The paper develops a new neural net architecture for processing data structured as spatial point clouds that vary over time (e.g. hourly traffic at several antennas spread throughout a city).

The core of the approach is a new neural net unit: the "D-Conv" operator (See Eq. 2). The output value at each point is obtained via a weighted combination of nearby coordinates and features, using only the K-nearest neighbors (stored in ranked order) to maintain invariance to the original order of points. This layer can be included in modern convolutional (CloudCNN) or recurrent (CloudLSTM) or attention-based architectures in a straightforward way.

Unlike many previous methods that require converting point clouds to quantized regular grids, the present approach directly consumes point cloud data. Unlike some existing methods like PointCNN, it avoids information loss (does not reduce dimension from input to output layer).

Two experimental evaluations are conducted: forecasting mobile app traffic across 2 European cities (given past 30 min, predict next 30 min), and air quality across several regions in China (given last 12 hrs, predict next 12 hrs). In both experiments, the locations of the sensors are fixed across time. Fig 4 further looks at traffic forecasting as a function of the lookahead time, from 0-3 hours ahead.


Novelty & Significance
-----------------------

The paper definitely tackles an important problem (point cloud forecasting). 

The present paper's new "D-Conv" operator appears new, though it looks really like a simplification of the PointCNN's "X-Conv" operator rather than a brand new operator. 

The most similar work seems to be the PointCNN (Li et al NeurIPS 2018). This work's contribution was a new "X-Conv" operator, which also consumes point clouds and produces learned representations. X-Conv, like the present paper's D-Conv, computes K-nearest neighbors of each point p, but performs first an embedding of each neighbor to a learned "local" feature space and then performs convolution on this embedding. Perhaps the biggest practical difference is that D-Conv has fewer parameters (does not perform the embedding) and does not reduce dimensionality from input to output. 

Technical Concerns
------------------

My biggest concerns are that the D-Conv has a strong reliance on nearest neighbors. This means the D-Conv has not much accomodation for "outlier" points that are far from others. The X-Conv operator has some nice properties in this regard (it changes coordinate systems so neighbor locations are centered around the current point), but I don't see this in the D-Conv operator, as in Eq. 2, where the coordinate locations are fed directly into the weighted sum after global rescaling to (0,1). I would imagine that data with outliers (whose values are unlike most others) would dramatically hurt performance, as the weights of D-Conv would need to be shared equally by outliers and inliers.


Experimental Concerns
---------------------

Is there a good reason to not try to compare on publicly available datasets like those used in the PointCNN paper (focusing only on the non-temporal versions of the model)? Using proprietary datasets makes following up on this work a bit hard, would be nice to have some reproducible experiment.

It's not clear to me that the experiments here consider realistic scenarios. Why would I predict mobile app traffic using only the past 30 minutes of data? Why predict air quality using only the last 12 hours? Certainly there are time-of-day, day-of-week, and seasonal effects that are all important. At a minimum, I'd think that for the mobile traffic case you could at least look at consuming the last 48 hr of data and predicting the next 30-90 minutes. I suspect that would make even simpler models do much better.  

Further, I think the experiments are missing some key simple baselines (or I misunderstand something). For example, rather than the complicated CNN/LSTM architectures, why not try to directly see how much value there is in "neighbors" in this 2d space? At each point, you can make predictions using only the K nearest neighbors' data, with K swept from 1 to 100 or something. I would expect with these features, using just a simple MLP or RNN would do quite well. I'd like to see a stronger qualitative case made for why we expect the complicated DConv weighting operator here to do better than this baselines.

Overall, the results tables appear promising (for app traffic forecasting in Table 1, the proposed CloudLSTM achieves 3.66 MAE compared to 4.95 for PointCNN and 4.8 for an MLP). However, it's not clear why and I'd like to understand why. Is it that the other approaches are overfitting? 


Minor Concerns
--------------
I would suggest avoiding calling the method "\mathcal{D}-Conv", and instead use just "DConv", since this is easier to type into search engines and easier to search for in a PDF document

Related: Point clouds could be represented as graphs, and then use graph embeddings as feature representations
review_title: Official Blind Review #1

Meta-review: The paper presents an approach to forecasting over temporal streams of permutation-invariant data such as point clouds. The approach is based on an operator (DConv) that is related to continuous convolution operators such as X-Conv and others. The reviews are split. After the authors' responses, concerns remain and two ratings remain "3". The AC agrees with the concerns and recommends against accepting the paper.
Decision: Reject