Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a model for the causal structure of the misinformation influence on users. It proposes one of the first studies of this kind building on temporal point processes and modelling users' engagement and posting behaviour as events while misinformation is considered as treatment. A treatment effect evaluation is proposed which projects the intensity function to a vector to measure the individual treatment effect. This is then calculated more efficiently through reformulations and then estimated with a neural temporal point process model. In the experiments, synthetic data is used to show the accuracy of the model, whereas a real-world covid-vaccine Twitter dataset is discussed as a case study to show how misinformation influences users' behaviour differently compared to information, among other findings. 
strengths_and_weaknesses: Strengths:
- well-motivated paper, and interesting problem 
- paper is generally easy to follow and well structured

Weaknesses:
- presentation and writing needs improvement
- many key details are given only in the appendix and the paper is hard to understand as is 
- results are not convincing, and hard to interpret


Minor comments: 
"This is not applicable for social media post because most meaningful features of the posts, such as geographical information, sentiment score and subjective score, are naturally continuous. Forcibly discretizing them will lose meaningful information." =>  geographical information usually is at the city level, sentiment is often a classification. I am not sure what is the subjective score but overall the argument is weak.

What is S_{in} in line 122?

Many language/grammar errors, examples:

"Consequently, an effective and efficient automatic methodology to estimate the impact of the misinformation on user beliefs and activities."

"to study the relation between misinformation on people’s behaviours."

"Unless getting heavily revision, such previous work can not be simply transferred to our problem scenario."

"the treatment effect can be trivially computed by compare the difference of the outcomes from real world and counterfactual world"

"we are lack of well-established baselines for this specific task."
questions: Can you explain the main intuition of how this model works? 

What is the problem definition?

Can you explain why the subjectivity score is always zero in fig 3a? what is the accuracy of the misinformation labels used in 5.2. as well as the sentiment and subjectivity scores? have you done any evaluation on how reliable these are? what is the range for sentiment scores?

Can you explain how your data is different from the references cited? the data sizes do not match. 
limitations: Not discussed in the paper in depth. 

Checklist 4d -> using Twitter data is common in research but still, the question of consent is relevant and can be discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes a novel method of tackling the impact of misinformation in social media through the lens of causal analysis via a neural temporal point process model to estimate the effects using generated synthetic data and real-world data in the domain of healthcare (re: COVID-19 vaccines). The proposed method draws motivation from how misinformation is spread on social media sites and how related previous works have only focused on correlational studies and randomized trials without exploring causality-based studies. To properly evaluate the performance of the causal inference model, the authors used synthetic data. Results show that the proposed method slightly outperforms neural network models without causal modeling and autoencoder-based methods in accuracy and inference time. In the real-world data experiment, the authors draw summarized observations on how sentiments affect the behavior of users in spreading (mis)information and differentiation from the two using estimated ITE.
strengths_and_weaknesses: I appreciate the paper’s sufficient and balanced level of technicality on treatment effects, adversarial training, and the proposed causal structure model. I find the motivation for applying causal analysis in alleviating fake news on social media solid. I also commend the authors for structuring the paper in such a way that it’s very easy to follow, read, and understand. I have a few points, however, that can be considered and discussed to further improve the quality of the work:

1. The dataset used (both real-world and synthetic) should be at least described sufficiently in the paper including mentions of limitations of the domain used (health-related topics only), language specifications, and search parameters such as geolocation used to name a few. The same way goes for the evaluation metrics used: MatDis and LinCor. I believe these are essential information that should be present in the paper itself without being directed to the Appendix. In case of acceptance, I highly recommend including this information for completeness. 
2. Discussion on comparison between the proposed method vs. the CRN-VAE in Section 5.1 for causal effect inference and estimation residuals can be improved to add more depth. What is it with the sampling method of CRN-VAE that makes it relatively at par with the proposed method but lags behind inference time? Moreover, the note on heavily revising CRN-VAE confuses the reader. Aside from getting the sampled and approximated outputs, what other changes were made to the framework?
3. If there are no statistical tests done, I suggest switching the term significantly to substantially as the word is already overused in literature without support.

questions: 1. Is the BERT embeddings concatenated together with the subjectivity score and sentiment score? How were the latter two obtained? Is there a reliability rate for these scores?
2. In Section 5, there is a mention that previous models used synthetic datasets for evaluating proposals on causality analysis. Just out of plain curiosity, were the authors able to retrieve the generation parameters from previous works and apply the synthetic data generation close if not equal to the parameters used for potential comparison? Discussion on this is encouraged.
3. What do/does the author/s infer from the overlap of (mis)information based on ITE scores in Figure 3B?
4. In the experiment setup with both synthetic and realistic data, can the engagement actions (like, retweet, comment) be ranked or weighted according to how likely can trigger the user to spread (mis)information? For example, a user may have a higher chance of spreading a misinformed post after liking it *without* commenting on it as we don’t always assume that both actions are done together all the time. In this case, the *like* has more *weight* than the comment.
limitations: The paper poses no serious negative societal impact as long as the results, as mentioned by the authors, are only taken into consideration as additional reference and not an assumption of absolute truth in a decision-making scenario. In addition, as mentioned in the sections above, I suggest the authors include a Limitations section to cover discussion on the limitations of the study including the breadth of the domain used for measuring the impact and effect of (mis)information, data generation settings, and evaluation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to estimate the impact of misinformation on real-world social networks. It constructs a causal framework and leverages techniques such as neural temporal point process and gaussian mixture models for this task. Experiments were conducted on synthetic and real-world datasets to demonstrate the proposed approach's effectiveness.
strengths_and_weaknesses: Strengths:
- proposes a new and interesting direction of misinformation research
- the proposed approach is well motivated and clearly grounded yet improved from previous works
- superior experiment results on both synthetic and real-world datasets

Weaknesses:
- important details about datasets and experiments are missing
- experiments could be improved to better prove the author's claims
- underexplored implications for misinformation research and societal impact
questions: 1. What does ITE stand for? This might be domain-specific jargon and one could infer from line 156 that it denotes "individual treatment effect", while I feel that the authors could explain this at the first occurrence of ITE. This would help people like me to understand your work better :)

2. There are a lot of details regarding the synthetic dataset: "hidden vector $u_i$", "topic vector $v_{topic}(n)$", "influence vector $v_{in}(n)$", "a post of original contents". (line 265-268) Maybe I am missing something, but Section A.3 in the appendix does not seem to describe every single detail of the synthetic dataset. Since the authors only compared the proposed approach with existing methods on the synthetic dataset, these dataset details are crucial to validate that the synthetic dataset is a good indicator of model performance.

3. In checklist 3(c), the authors state that error bars are provided "in the appendix". In Section A.4, the authors do provide the error bars of inference time on the ITE estimation task. However, the error bars of performance metrics, such as Accuracy, RAE, and RRSE in Table 1 as well as MatDis and LinCor in Table 2, are perhaps more important, especially given that the proposed approach only slightly outperforms CRN-VAE (Sampling). I hope the authors could provide the error bars for these evaluation metrics.

4. For the real-world dataset, the authors simply point to [31, 25], while I would like to see more explanation. Why this dataset? Why not existing datasets on fake news detection such as PolitiFact or GossipCop? This real-world dataset seems to only focus on COVID, so maybe a multi-topic misinformation dataset would better support the evaluation?

5. I personally believe that Figure 3(b) could be stronger if the authors also plot this figure for other baselines. Yes, it seems that the estimated ITE vectors could differentiate between real and fake news, but perhaps a language model encoding of news text could achieve the same effect? What about baseline methods in Tables 1 and 2? I would hope for a quantitative comparison (using clustering metrics such as homogeneity score or V-measure) between the proposed approach of baseline methods in the format of Figure 3(b).

6. I am curious about the results in Figure 3(a). It seems that all pieces of news result in almost 0 change in user subjectivity. In light of this, should the subjectivity score remain a factor in the causal framework? I would like to hear about the authors' remarks on this issue.

7. I am excited about the potential of this approach to understanding the impact of misinformation on social media. However, I feel like this paper could better position itself in the context of misinformation research. Specifically, what are the implications of the predicted ITEs towards mitigating online information? How can other researchers use this model to advance misinformation research and address the infodemic?

8. In checklist 1(b), the authors describe this work's limitations as "certain baselines outperform us". I would like to see more discussion about the limitations of the proposed method itself. For example, the framework only models two user activities: post and engage, while there are other user features and there are different types of online engagement such as like and block, which signals vastly different user attitudes. Besides, the method is only evaluated on one real-world dataset that is COVID-related, while whether the approach would generalize to multi-topic real-world social media discourse is questionable.

9. In checklist 1(c) and Section 6, the discussion about potential negative societal impact could be enhanced. For example, operators of misinformation campaigns could use the proposed approach to evaluate how different pieces of misinfo perform in deceiving people and thus design highly misleading misinformation strategies. I would suggest a little more discussion about the societal impact.
limitations: Please refer to the questions section. I will definitely read the authors' follow-up comments and update my rating accordingly. :)
review_title: null

Meta-review: metareview: This work addresses the interesting problem of measuring the causal effect of exposure to misinformation from observational data culled from social media. Reviewers agreed on the importance and timeliness of this problem. The motivation for causal analysis here is also readily apparent (SxLq), and the proposed point process model seems novel (at least in its application here) and appropriate. 

The main weaknesses raised concerned presentation issues (5Zb3,vdd1) and some concerns about experimental details and setup (vdd1,SxLq). The former constitute relatively minor issues which can be readily addressed in revision, and the latter issues were mostly satisfactorily addressed during the response period. Reviewer 5Zb3 raises a concern about the sentiment model used as part of the evaluation, which the authors should discuss in future iterations of the work. 

Overall, while the evaluation suggests only modest empirical gains over baselines (which were themselves introduced in this work, for want of alternative existing methods), the work is novel in its investigation of causal methods for understanding the potential influence of misinformation on social media. 



confidence: Less certain
Decision: Accept