Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors tackle an important problem of identifying coordinated accounts on social media, which could help prevent misinformation propagation. Different from previous works, this paper focuses on incorporating prior knowledge into neural temporal point process model via variational inference framework. The authors show the improvements over the state-of-the-art method AMDN-HAGE on one dataset. 
main_review: The authors propose to incorporate prior knowledge into neural temporal point process model for coordination detection. However, the definition of prior knowledge, which is an important concept used in this paper, is not clearly introduced. What kind of knowledge is defined as prior knowledge?  

Second, in the abstract, the authors state that existing works that incorporated prior knowledge suffer from limited expressive power and poor performance. Such a statement is an important motivation of this paper, but the related works section does not include comparison with respect to such relevant work. Therefore, the current related works section cannot clearly depict the difference between the proposed way of using knowledge with that adopted by previous approaches. 

Third, the proposed method is closely related to the existing work AMDN-HAGE, which is used to initialize the embedding layer in this paper.  How is the performance of the proposed method without this initiation? This experiment would be needed to justify the contribution of the proposed techniques. In the current experiments, the proposed method utilizing prior knowledge is compared against other methods that do not incorporate prior knowledge. This comparison may only illustrate the important role of the prior knowledge but cannot justify the need of adopting the proposed model that incorporates knowledge in this specific way. Moreover, the authors only show the performance comparison on one dataset, which may not be enough to support the statement on the effectiveness of the model. In AMDN-HAGE, the paper includes the comparison in the supervised setting. Since supervised learning setting is very common and widely used, is there any specific reason not to show experimental results in this setting?
limitations_and_societal_impact: No, the authors did not discuss about how to handle failed cases. More discussions about potential social impact and failed modes may need to be added to the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provides an approach to detect coordinated groups of activity from social media. It defines a distribution over group assignments, and a function to measure how it matches the account embeddings and prior knowledge.  The main novelty is integrating prior into another recently proposed method for this task. The prior accounts for co-activity, interacting with the same tweet and time overlap of the activity. This information is integrated into the neural temporal point process.

main_review: 
The related work overlooks a large body of work from the Graph Mining community on dense block detection. This could be a good entry and a potential baseline:
Hooi, Bryan, et al. "Fraudar: Bounding graph fraud in the face of camouflage." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016.

The argument on graph-based detection ignores that social media data comes in naturally as graphs, i.e. the interactions between the users. It argues that the graph needs to be defined based on similarity or learned, whereas you can directly model the data as a temporal graph. K-means over (temporal) graph embedding methods, should have been another baseline.  Another baseline could be a tensor/matrix factorization method.

Overall, the paper is weak in terms of related works, and baseline and evaluations. There is a limited set of baseline used. Considering there available datasets are limited for benchmarking, there could be synthetic experiments, e.g. with SBM, to better test the method. And/or, more in-depth analysis of the real-world dataset studied. As is, the results are not convincing. Also, how does it compare with what 25 reported? Is the data the same? 


On a more high-level note, how would you differentiate between a group of users discussing the same topic heavily, say an election, and a coordinated group? Many of the detected hashtags in the covid example, are related to Canadian politics, cdnpoli, ableg, abpoli, onpoli. It seems to me, the method is detecting groups, which can be coordinated or not. There is also very limited information given on how the data is collected, timeframe, keywords, etc. for the experiment. 

On the novelty also, there could be a better separation from 25, as it is not clear which parts of the paper are novel in this submission. The datasets, experiments, and the model seem to be very similar. 

Figure 2a is not very readable.
Page 8, line 299: We use topic mining> please elaborate what method was used.
Page 9, line 309: we measure percentage of unreliable and conspiracy news sources shared in the tweets> please explain how this is measured
On Table 1, why is the reported performance for baseline lower than what is originally reported in 25, how is the setting different? Those results look more competitive.


limitations_and_societal_impact: potential negative societal impacts of your work are declared none. However, if applied in practice, this method can be used to label users as suspicious, coordinated, and get them banned from the platform. There could be more thought into how the method could be used and the implications.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes extensions to Temporal Point Processes and apply them to Coordination Detection on Social Media. The main contributions are the inclusion of a "prior knowledge based graph" and the corresponding Gibbs-sampling based EM Algorithm including an meanfield approximation. Empirical improvements are shown on two data sets, in one case also a short qualitative analysis is provided.
main_review: Without having deep expertise in this area, I got the impression that the overall approach and application is original. In general, there are original contribution in respect to the inclusion of a "prior knowledge based graph" the EM optimization algorithm and its meanfield approximation. Also the empirical evaluation is extensive with a clear dominance of the proposed approach. However, in detail, the contributions are rather incremental. The "prior knowledge based graph" promises less than expected. The "Cubic Function" and "Temporal Logic" remain very heuristic and specific. The general contribution of this steps remains unclear. More thorough appears to be the optimization procedure although all the combined techniques (EM, meanfield, Gibbs) are well known and typically combined in that way.

The quality and clarity of the presentation and formalization is very good. The paper is easy to follow and if provides an intuitive structuring. Some figures lack resolution and/or are too small (Fig 2a, 3). Some terminology used should be better related to existing work, like "knowledge based graph" vs "knowledge graph" or "temporal logic" vs formal logic.

My main concern is the significance. While the proposed approach shows a great performance, it lacks insights into why this is the case. Specifically, the effect of the inclusion of prior knowledge remains unclear. First, both, the "Cubic Function" and "Temporal Logic" are shallow. Why is the function cubic? Specifically the incorporation of "Temporal Logic" raises high expectations, but it seems this is basically a single and very simple inequation to rule out some account pairs. There are no Temporal Logic formulas or any sort of deeper integration. Also, the impact of this "prior knowledge based graph" remains unclear, since I couldn't find an no ablation study showing what is lost without it.
limitations_and_societal_impact: Coordinated misinformation campaigns have societal impact. The proposed method tries to mitigate it. Still the impact of such a method should be discussed, since there are also risks, like false positive classifications.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies the problem of detecting coordinated user activity in social media. 
A recent paper by Sharma et al., 2020 [(arXiv:2008.11308)](https://arxiv.org/abs/2008.11308) proposed AMDN-HAGE -- an approach for coordination detection based on neural TPPs.
Specifically, AMBN-HAGE trains a neural TPP model on traces (e.g., retween sequences) and treats each user/account as an individual mark in a marked TPPs.
Then, mark embeddings learned by the model are clustered with K-means to detect accounts exhibiting similar activity (cluster = group of coordinated accounts).

The submitted paper extends AMDN-HAGE by incorporating prior knowledge (e.g., the fact that coordinated accounts are active during the same time periods).
This is achieved by treating the cluster assignments  as latent variables in a Markov random field (MRF).
The MRF construction takes into account the mark embeddings of the neural TPP model and pairwise similarities between accounts derived based on domain knowledge.
Both parameter learning and cluster inference are done simultaneously by maximizing the ELBO. Since the naive formulation of the ELBO is computationally intractable, the authors derive a lower bound that enables an efficient approximation.
The experiments on real-world data show that the proposed approach provides a consistent improvement over the baselines, both in unsupervised and semi-supervised detection of coordinated accounts.
main_review: **Originality:** The proposed method is sufficiently different from AMDN-HAGE. The idea of clustering the marks and learning a TPP model simultaneously has not been explored in neural TPP literature, as far as I know. Though, I am not fully familiar with other literature on coordination detection, so I might not be aware of some related works.

**Quality:** The approach is theoretically sound. I wasn't able to find any mistakes in the derivation of the main theoretical result (except minor inconsistencies in notation that are listed below). The experimental evaluation is solid, and several ablation studies demonstrate that different model components indeed improve the detection quality. There are some baselines that I believe could be included in the evaluation to make the results even more convincing (see questions below).

**Clarity:** The proposed model is described clearly and the paper is easy to follow. The detailed description of the experimental setup should make the results easy to reproduce.

To summarize, this work proposes an interesting approach to a relevant task, and I haven't found serious flaws in the paper. However, I would like to see the following questions addressed during the discussion period.

**Questions:**
1. Have you compared to baselines that operate exclusively on the "graph constructed from prior knowledge" (Section 4.1)? 
Since the VigDet model combines two information sources (embeddings + graph), it would be interesting to see how a model that only considers each source separately performs. AMBN-HAGE covers the "embedding only" case, but the "graph only" case is missing. For example, one could perform mean-field VI on the MRF model that only includes the pairwise potentials (Equation 6). Other approaches, such as dense subblock/subgraph discovery could also be applied to the similarity matrices constructed in Section 4.1 (something similar to "M-Zoom: Fast Dense-Block Detection in Tensors with Quality Guarantees", ECML-PKDD 2016 and references therein, but for matrices instead of tensors).

2. How can we be sure that the model is not simply detecting communities of users in a social network? Have you considered any datasets where both coordinated groups as well as legitimate communities are present? For instance, machine learning researchers often like/retweet a similar set of tweets, but, intuitively, they shouldn't be considered as coordinated accounts.

3. How does the proposed approach address the power-law distribution challenge (Figure 1b)?


Minor suggestions:
- Lines 16 & 66: I find the usage of the phrase "theoretical guarantee" somewhat confusing. Potentially, something along the lines of "we derive a lower bound that allows us to efficiently approximate the ELBO" would be clearer.
- Theorem 1 (Equation 12)  
    - It would be helpful to specify that the constant terms are constant w.r.t. $\theta$ and $E$
    - Why include the $\sum_{1 \le m^\prime \le M} \exp (...)$ term (the denominator) inside the expectation? The term is independent of $Q$ and could simply be computed outside of the expected value, if I understand correctly.
- Line 106: In my experience, $dN(t)$ is usually referred to as the differential, not derivative. See e.g. these lecture notes http://courses.mpi-sws.org/hcml-ws18/lectures/TPP.pdf
- Line 125: The kernel function $\phi$ is usually periodic but not translation invariant, in general it doesn't hold that $\phi(x) = \phi(x + y)$ for all $x, y$.
- Line 171: `$Y$` instead of `Y`.

- Appendix:
    - Equation 3: Inconsistent usage of $y$ and $Y$
    - Equation 5: $Y^\prime$ instead of $Y$
limitations_and_societal_impact: A short discussion of limitations is provided. Potential negative societal impacts are not discussed.
review_title: null

Meta-review: The paper proposes a new approach to detect coordinated accounts in social media via neural TPPs. Reviewers highlighted that the paper proposes an interesting approach to this important problem and, furthermore, shows promising results. However, based on the current manuscript, reviewers raised concerns with regard to clarity (e.g., differentiation of coordination from normal user behavior/communities, modeling details), discussion with respect to related work, novelty, and experimental evaluation (baselines, datasets, ablations). After rebuttal and discussion, three knowledgeable reviewers indicated therefore to reject the current manuscript while one reviewer indicated weak support for acceptance. Although the paper proposes interesting ideas, I agree with the majority of reviewers that the manuscript is currently not ready for acceptance at NeurIPS due to the aforementioned concerns and would require an additional revision (e.g. with regard to clarity, experimental evaluation, as well as analyses that provide insights into coordination vs community). However, I encourage the authors to revise and resubmit their work based on the feedback from this reviewing round.
Decision: Accept (Poster)