Venue: KDD 2023 Workshop epiDAMIK

Reviews:

final_score: rating: 3: Marginally above acceptance threshold
review_content: ### Summary
This paper seeks to improve graph neural network driven epidemic forecasting by incorporating mobility data. They focus on forecasting the reversal of trends using this additional information. The effect of the mobility data is isolated by comparing the results with a similar GNN without mobility data. They emphasize the value that mobility data holds in this application compared to other sources of data. Although their results show promise for limited settings, they do not consider settings in which the trends changes in direction from downwards to upwards, which is arguably more relevant for disease forecasting.

### Strengths
- The authors test under real-world assumptions of data availability lags by using a 17-day gap between their training and testing sets.
- The contribution is simple and clearly stated, as well and the hypotheses they introduce concerning their results.

### Weaknesses
- Limited to a linear trend of incidence over 14 days. It is unclear whether this means the method only has the capability to detect changes in trend.
- They only evaluate an upwards to downwards trend, which may affect the viability of the method until the opposite is verified.

### Suggestions
- If possible, it would help to have a small GNN architecture diagram
- When you “design each graph sample to contain 15 temporal connections…”, does each “graph sample” represent a single time point?
- Elaborate more on the purpose of the cyclical sine/cosine encoding
review_title: Mobility data added to GNN architecture to improve outbreak forecasting

final_score: rating: 3: Marginally above acceptance threshold
review_content: Overall, the paper is well-written and provides valuable insights into the potential benefits of using mobility data and GNNs for COVID-19 forecasting. While the paper's quality is commendable, with a clear research objective and a well-defined methodology, the evaluation is lacking and somethings need better explaining (detailed in the list below). The paper's clarity is also good but needs improvement in a few areas. The significance of this work is evident in its potential to improve pandemic forecasting and inform decision-making but the originality of leveraging mobility data to enhance predictions is not something new.


Pros:

Model architecture and training setup: The paper mentions that the model architecture is well chosen and the training setup and evaluation metrics are appropriate. These aspects are positively reviewed.

Preprocessing: The paper's preprocessing steps of smoothing the 7-day incidence data using a center-aligned 7-day moving average (and normalizing where necessary) is considered a good approach to enhance the quality of the data and facilitate more accurate trend analysis.

Cons:
Accuracy of predicted trends: Using linear regression to predict the 14-day trends raises questions about the accuracy of these predicted trends themselves. It might be more appropriate to linearly approximate the ground truth data when evaluating the model's performance since that is the target being predicted.

Evaluation scenario: The evaluation scenario seems to be lacking since it only considers a one-month period. There is confusion about how the peak is determined using a 7-day moving window. If the window is limited to 7 days, there will always be a peak within that period since the paper mentions "We extract the date on which each county’s corresponding smoothed COVID-19 7-day incidence time series
has its maximum, i.e., its peak". If it is simply the peak of ALL 7day incidence time series for each county then it is more than reasonable, if not then further clarification is needed to understand the evaluation process.

Training data size: The paper does not clearly state the size of the training data. It mentions considering the 2nd half of the first omicron wave from November 10 to December 19, which implies two months of data. However, it is unclear when the summer period is included in the training data.

Oversampling: The necessity of oversampling is questioned, especially when predicting 14-day trends since low incidence should not have effect on trends. It is also hard to determine the negative effects of this since the training data's size is not mentioned. If ALL previous periods are used for training, there is a concern that discrete oversampling may not be suitable as different waves of variants can behave differently due to varying growth rates. While if only the brief period (1st half of the Omicron wave is used), it can potentially lead to overfitting due to a small amount of data, this is especially the case when the model is evaluated during a wave of a different variant. 

Testing on declining trends only: It is questioned why the evaluation is conducted only on a period where the trend is declining. The evaluation should consider periods of both increasing and decreasing trends to provide a comprehensive assessment of the model's performance.



In conclusion, while the paper presents some positive aspects, including the model architecture, training setup, and evaluation metrics, there are significant concerns and areas that require clarification and justification. The questionable use of predicted trends as target data, the unclear evaluation scenario and peak determination, and the potential issues with oversampling can raise some doubts about the methodology. It is also not clearly discussed how the GNN can utilize information from counties with leading changes for forecasting
counties that experience similar changes later. For future work, it would also be good to provide comparisons to other non GNN related models (even something as simple as the linear regression being done). Further elaboration and addressing of these concerns would greatly improve the quality and clarity of this work. The title of the paper can also potentially be changed since the forecasting being done is of COVID-19 trends rather than incidence.
review_title: A Comprehensive Review of a COVID-19 Forecasting Paper: Promising Approach with Room for Clarification and Improvement

final_score: rating: 3: Marginally above acceptance threshold
review_content: 
Quality
The paper is written in an understandable manner. The experiments are done on (or shown on) a limited set and the authors don’t compare with other baseline methods for case trend analysis of COVID-19

Clarity
It can be improved further to make contributions of the authors clearly delineated from the existing literature. The explanation of the graph can be better if it is shown visually, but I understand that there are space limitations for this submission

Originality
The idea is something that is already explored by other researchers to answer different or similar questions related to the pandemic forecasting

Significance
The work is significant to the workshop given the problem it is tackling. This is also interesting to society given the possibilities of other epidemics

Pros
Compare different error metrics and also perform a statistical test to show the significance
They show that the addition of mobility is helping in the forecasting; counties with earlier trends can also help predict those with later.

Cons
No baseline comparison apart from their own method without spatial information
Missing citations:
1) mobility network related - Mobility network models of COVID-19 explain inequities and inform reopening; this and follow-up works seem closely connected to what the authors explore
2) Trends are more important than actual numbers in terms of pandemic forecasts
3) Using information from one county to help the other 
I am aware of works from the CDC forecasting hub, XPRIZE pandemic challenge, etc that have discussed these issues. It would be relevant to cite those papers and show how this work is different 

review_title: Mobility data improve forecasting of COVID-19 incidence using Graph Neural Networks

Decision: Accept