Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a novel model MetaHG to automatically detect illicit drug traffickers on social media. It firstly builds a heterogeneous graph based on post content and relational structure information on social media to characterize the drug trafficking system. Then it integrates graph structure learning and relational-based GCN to learn robust node embedding on HG. Afterwards, it leverages meta-learning and knowledge distillation to optimize model parameters and further to detect drug traffickers on social media. Finally, it conducts extensive experiments to validate the effectiveness of the framework.
main_review: It is very interesting and meaningful to develop a novel model for drug trafficking detection on social media. I list the main strengths and weaknesses of the paper below.

Strengths:

+ This paper is well organized and easy to follow. It elaborates on background, related work, methodology, and experiments. 

+ To address the few labeled data issue, the proposed framework leverages meta-learning and knowledge distillation to optimize the model. In addition, to solve the problem of graph sparsity, it combines graph structure learning and relational GCN augmented by self-supervised learning to learn the robust node embedding on heterogeneous graph. The design of MetaHG framework is novel the technique quality of this paper is solid.

+ The paper conducts comprehensive experiments to validate the effectiveness of the framework. It compares with five groups of baseline models, especially previous methods of the same topic. To show the effectiveness and robustness of the framework, it conducts detailed ablation studies for analyzing the effectiveness of each module and it does comparison experiments on different tasks for analyzing the model robustness.

Weaknesses:

- The paper doesn’t explain why the heterogeneous graph is sparse. It is common that the relationships among users and posts on social media platform (e.g., Instagram) can be dense. Why the heterogeneous graph in this paper is sparse?

- The description of baseline models and their settings is short and some parts are confused. In Table 2 Group B2, it shows the results of other two papers with different shots, which is confused. The paper should introduce these two methods briefly.

- Minor issue: In Figure 1 and Figure 5, I think “63w” and “60w” should be how many weeks ago did the comment post instead of the number of like. Please double-check it.
limitations_and_societal_impact: This paper discusses the limitations and potential impact in the introduction and the supplement materials.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents a novel heterogeneous graph learning model for automatically detecting illicit drug trafﬁckers on Instagram. The proposed model addresses two challenges: 1) sparse graph structure, and 2) limited labeled samples for model training. The authors also collected a large social media dataset from Instagram and conducted a lot of experiments to evaluate the proposed model.
main_review: Overall, this work provides a complete study with sufficient motivation, novel techniques, and thorough experiments. More speciﬁcally, the authors consider the constraints of graph sparsity and limited labeled data for model design, which includes the three-fold contribution: 1) a novel framework named MetaHG to integrate both structural relations and unstructured content information for illicit drug trafﬁcker detection. 2) graph structure learning is used to refine HG structure and self-supervised R-GCN is employed to learn the node embeddings. 3) a meta-learning approach with knowledge distillation is adopted to optimize the model which could overcome the issue of insufficient labeled samples. I think it is an interesting work and has the ability for generalizing to other applications. 

On the other hand, the design of the method is reasonable and comprehensive. Two challenges of sparse graph connection and insufficient labeled samples are well addressed. Moreover, the presentation is good with straightforward and clear writing. 

There are some points where improvements could help further strengthen the paper:
In the methodology section, the motivation of modeling HG is not very clear. Why the heterogeneous nodes and relations can be useful for this task?

The paper uses eight types of relations to describe the relationships among different types of nodes. The authors should explain how to select these relations. Do they all benefit from this task?

The proposed self-supervised augmentation model is mainly based on the content information. I'm curious about the performance if it leverages some relations to implement the same model.

Do the authors evaluate the training and testing time for the proposed model?

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studied the task of detecting online drug trafficking using social media data. In particular, the authors proposed to use graph convolutional networks to model various types of entities in social media and the relations between them. Also, a meta-learning algorithm is used for the model optimization.
main_review: This work presented an interesting idea of using the GCN-based method to model the social media network for online drug trafficking detection. The detailed comments are listed below: 

1. The novelty of the proposed model is relatively limited. Meta-HG is mostly applying existing techniques, e.g., R-GCN and meta-learning, to the application of drug trafficking detection. The major contribution may come from the combination of FSG and original HG, and AS-SSL. 

2. HG construction is not clearly described. First, some details of the text feature vector generation are missing. For example, how is the username embedded in BERT? The authors said they concatenate the extracted features. However, if that is the case, then embedding of post/user may have different dimensions since the post length, and user information varies. How is this situation handled by the proposed graph convolutional network? Second, it is not clear how the authors use the image feature to represent post/user. One post may contain multiple images, and one user may post multiple posts.  How is this situation handled during the HG construction?

3. The proposed method models every entity in the social network in the graph, which raises the concern of the time/space complexity of the proposed model. It seems not practical to do this in a real-world situation, especially when the model tried to learn HG and FSG at the same time. 

4. How the HG and FSG are combined? By unioning the edge set of HG and FSG? 

5. The description of AS-SSL needs to be more clear about the terms such as node features, node attribute features, node attributes, node embeddings, node representations, and node pair features. 

6. What are the "node features" mentioned in line 228? Are they node features obtained in the HG construction? If this is the case, then it does not make sense to use node attribute features (which I assume is obtained in the HG construction) again for the classification task because this will make the AS-SSL part decoupled from the computation graph. If node features refer to the node embeddings learned generated while training on HG, it is better to highlight that the cosine similarity calculation is still a network layer, though no parameters are learned. Just to make sure that readers understand that the AS-SLL is still in the computation graph.  

7. Is AS-SSL applied only on the user nodes because, in Figure 2c, only the "vendor" icon is used there. If so, the authors should state it clearly in the paper. 

8. The dataset collected is kind of "ideal" for the task because 37% of the users are drug traffickers. In practice, when the users want to increase the "recall" of detection of drug traffickers detection, they will increase the list of keywords which will inevitably bring "noise" into the graph. So reporting the model's performance in a less ideal setting would be interesting.

limitations_and_societal_impact: The limitations and potential negative societal impact of this work are addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper propose a novel framework MetaHG based on R-GCN for illicit drug trafficker detection on social media. A series of methods including meta learning(MAML), self-supervised learning(AS-SSL) and knowledge distillation(KD) are used to MetaHG, and obtain better metrics than traditional models with few labeled samples. In addition, the paper also creates a new dataset from Instagram for the task.
main_review: This paper combines a series of methods including R-GCN, meta learning, self-supervised learning and knowledge distillation for illicit drug trafficker detection, and obtain better metrics than traditional models. A new dataset is also created for this task, which is a valuable contribution. Paper writing is overall very clear and persuasive with adequate experiments and high performance.

Some questions for the author:
1. The graph model seems to be undirected, but social network relationships are obviously directed.
2. In Figure 4(d), what’s the definition of stability? How does it calculate and reflect stability?
3. In case study, the homepage of “se***11” writes out several drug names straightforwardly. Why can’t existing models detect it? What improvements help MetaHG detect it?

Some small points and feedback:
1. In Figure 2(a), the word “vendor” is not mentioned which should be “user”.
2. I think that creating a new dataset is valuable, the authors would consider mentioning it in the abstract.

limitations_and_societal_impact: None.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Summary: 
The authors propose a model to detect drug-trafficker profiles on Instagram. The model consists of many steps and phases, but in short: they create a heterogeneous graph from users, keywords, and posts, and in addition to the regular cross-entropy loss they propose to add multiple regularizer terms extracted from several stages of their algorithm (i.e., a term to build the graph, a term to learn the node representations, and a term to distill the knowledge from an out-of-domain model).

To evaluate their model, the authors created a dataset and compared their model with many baselines and their combinations. They also reported several experiments, including user visualization and a case study.
main_review: Reasons to accept:

- The model works very well in the dataset. The improvement over the baselines is substantial.

- The authors release their code (not the dataset).


Reasons to reject:

- The model is enormous, it has many moving parts. This makes the model generalization to other datasets and scenarios very questionable. The sheer size of the model is not the main reason to reject, however this property cannot be evaluated with only one dataset (as the authors did). The astronomical performance of the model might be due to particular properties of this dataset—this is very likely, because the model is employing many aspects and areas.

- The model has many hyper-paremeters—I think five. None of them are discussed how they are tuned. This combined with the complexity of the model severely distorts the conclusions made in the paper. The last but not least is that the evaluations are made in the low-resource data regime. Practically such a gigantic model with many hyper-parameters is not tunable in semi-supervised settings.

- Even though the model has several steps, almost all of the employed techniques pre-exist. The actual contribution is very limited.

- Other minor issues:

  - The methodology section is not written well, it has several grammatical issues.

  - It is not described that where the numbers reported in Table 5 come from, what they represent, etc—in this table the main results are reported.

  - In Equation 4, are not X_i and X_j projected into the same space? Why do they have different weight vectors W_s1 and W_s2?

  - In Table 3, the ablation study shows that the meta-learning step is the most influential technique in the model. It is so effective that if we drop it from the model, the proposed model turns into a mediocre model, at best. Why is that? Do the baselines that employ meta-learning have the same module? This is not discussed in the paper.
limitations_and_societal_impact: They have addressed it.
review_title: null

Meta-review: The reviewers generally agree that the paper is interesting and like the new application with a novel dataset that will be published, thorough experiments that establish a baseline, and a clear presentation. The reviewers provided recommendations that should clearly be incorporated in the paper.

Decision: Accept (Poster)