Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary of the reviews:
Pros:
•	A novel way to evaluate the quality of heterogeneous data sources.
•	An interesting way to put the data quality measurement as a regularization term in the objective function.
Cons:
•	Since the data quality is a function of local variation, it is unclear about the advantage of the proposed data quality regularization versus using a simple moving average regularization or local smoothness regularization.
•	It is unclear about the advantage of using the Multi-layer graph convolutional networks versus two naïve settings for graph construction + simple LSTM, see detailed comments D1

Detailed comments:
D1: Compared to the proposed approaches, there are two alternative naïve ways: 1) Instead of construct the similarity graph with Gaussian kernel and associated each vertex with different types of time-series, we can also construct one unified similarity graph that is a weighted combination of different types of data sources and then apply traditional LSTM; 2) During the GCN phases, one can apply type-aware random walk as an extension to the deep walk that can only handle a single source of data. It is unclear about the advantage of using the Multi-layer graph convolutional networks versus these two naïve settings. Either some discussions or empirical comparisons would be a plus.

review_title: This work proposed a new way to evaluate the quality of different data sources with the time-vary graph model, that is, the quality of data sources (e.g., time-series sequences) associated with each vertex is defined as a function of local variation at each vertex. In addition, the quality level is also used as a regularization term in the objective function for any general applications. The data quality evaluation is trained using the Multi-layer graph convolutional networks. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is an application of neural nets to data quality assessment. The authors introduce a new definition of data quality that relies on the notion of local variation defined in (Zhou and Schölkopf, 2004), and they extend it to multiple heterogenous data sources. The data quality function is learned using a GCN as defined in (Kipf and Welling, 2016).
 
1) How many neighbors are used in the experiments? Is this fixed or is defined purely by the Gaussian kernel weights as mentioned in 4.2? Setting all weights less than 0.9 to zero seems quite abrupt. Could you provide a reason for this? How many neighbors fit in this range?
2) How many data points? What is the temporal resolution of your data (every day/hour/minute/etc.)? What is the value of N, T? 
3) The bandwidth of the Gaussian kernel (\gamma) is quite different (0.2 and 0.6) for the two datasets from Weather Underground (WU) and Weather Bug (WB) (sect. 4.2). The kernel is computed on the features (e.g., latitude, longitude, vegetation fraction, etc.). Location, longitude, distance from coast, etc. are the same no matter the data source (WU or WB). Maybe the way they compute other features (e.g., vegetation fraction) vary slightly, but I would guess the \gamma should be (roughly) the same. 
4) Are the features (e.g., latitude, longitude, vegetation fraction, etc.) normalized in the kernel? If they are given equal weight (which is in itself questionable) they should be normalized, otherwise some of them will always dominate the distances. If they were indeed normalized, that should be made clear in the paper.
5) Why do you choose the summer months? How does the framework perform on other months and other meteorological signals except temperature? The application is very nice and complex, but I find that the experiments are a little bit too limited. 
6) The notations w and W are used for different things, and that is slightly confusing. Usually one is used as a matrix notation of the other.
7) I would tend to associate data quality with how noisy observations are at a certain node, and not heterogeneity. It would be good to add some discussion on noise in the paper. How do you define an anomaly in 5.2? Is it a spatial or temporal anomaly? Not sure I understand the difference between an anomaly and a bridge node. 
8) “A bridge node highly likely has a lower data quality level due to the heterogeneity”. I would think a bridge node is very relevant, and I don't necessarily see it as having a lower data quality. This approach seems to give more weight to very similar data, while discarding the transitions, which in a meteorological setting, could be the most relevant ?! 
9) Update the references, some papers have updated information (e.g., Bruna et al. - ICLR 2014, Kipf and Welling – ICLR 2017, etc.).

Quality – The experiments need more work and editing as mentioned in the comments.  
Clarity – The framework is fairly clearly presented, however the English needs significant improvement. 
Originality – The paper is a nice application of machine learning and neural nets to data quality assessment, and the forecasting application is relevant and challenging. However the paper mostly provides a framework that relies on existing work.
Significance – While the paper could be relevant to data quality applications by introducing advanced machine learning techniques, it has limited reach outside the field. Maybe publish it in a data quality conference/journal.
review_title: Nice idea and application, but needs significant work before publication.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update:

I have read the rebuttal and the revised manuscript. Paper reads better and comparison to Auto-regression was added. This work presents a novel way of utilizing GCN and I believe it would be interesting to the community. In this regard, I have updated my rating.

On the downside, I still remain uncertain about the practical impact of this work. Results in Table 1 show that proposed method is capable of forecasting next hour temperature with about 0.45C mean absolute error. As no reference to any state of the art temperature forecasting method is given (i.e. what is the MAE of a weather app on a modern smartphone), I can not judge whether 0.45C is good or bad. Additionally, it would be interesting to see how well proposed method can deal with next day temperature forecasting.

---------------------------------------------
In this paper authors develop a notion of data quality as the function of local variation of the graph nodes. The concept of local variation only utilizes the signals of the neighboring vertices and GCN is used to take into account broader neighborhoods of the nodes. Data quality then used to weight the loss terms for training of the LSTM network to forecast temperatures at weather stations.

I liked the idea of using local variations of the graph signals as quality of the signal. It was new to me, but I am not very familiar with some of the related literature. I have one methodological and few experimental questions.

Methodology:
Why did you decide to use GCN to capture the higher order neighborhoods? GCN does so intuitively, but it is not clear what exactly is happening due to non-linearities. What if you use graph polynomial filter instead [1] (i.e. linear combination of powers of the adjacency)? It can more evidently capture the K-hop neighborhood of a vertex.

Experiments:
- Could you please formalize the forecasting problem more rigorously. It is not easy to follow what information is used for training and testing. I'm not quite certain what "Temperature is used as a target measurement, i.e., output of LSTM, and others including Temperature are used as input signals." means. I would expect that forecasting of temperature tomorrow is solely performed based on today's and past information about temperature and other measurements.
- What are the measurement units in Table 1?
- I would like to see comparison to some classical time series forecasting techniques, e.g. Gaussian Process regression and Auto-regressive models. Also some references and comparisons are needed to state-of-the-art weather forecasting techniques. These comparisons are crucial to see if the method is indeed practical.

Please consider proofreading the draft. There are occasional typos and excessively long wordings.

[1] Aliaksei Sandryhaila and José MF Moura. Discrete signal processing on graphs. IEEE transactions
on signal processing, 61(7):1644–1656, 2013.
review_title: Interesting work. Good novelty component. Uncertain about practical impact.

Meta-review: With an 8-6-6 rating all reviewers agreed that this paper is past the threshold for acceptance.

The  quality of the paper appears to have increased during the review cycle due to interactions with the reviewers. The paper addresses issues related to the quality of heterogeneous data sources. The paper does this through the framework of graph convolutional networks (GCNs). The work proposes a data quality level concept defined at each vertex in a graph based on a local variation of the vertex. The quality level is used as a regularizer constant in the objective function. Experimental work shows that this formulation is important in the context of time-series prediction.

Experiments are performed on a dataset that is less prominent in the ML and ICLR community, from two commercial weather services Weather Underground and WeatherBug; however, experiments with reasonable baseline models using a "Forecasting mean absolute error (MAE)" metric seem to be well done.

The biggest weakness of this work was a lack of comparison with some more traditional time-series modelling approaches. However, the authors added an auto-regressive model into the baselines used for comparison. Some more details on this model would help.

I tend to agree with the author's assertion that: "there is limited work in ICLR  on data quality, but it is definitely one essential hurdle for any representation learning model to work in practice. ".

For these reasons I recommend a poster.


Decision: Accept (Poster)