Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper describes a method for using YouTube video comments to classify video content, with the aim to separate videos into those describing Covid-19 conspiracy theories (misinformation) and those not (factual). An annotated video corpus is created by searching for videos on social media, downloading them from YouTube and annotating them, resulting in a dataset of 113 misinformation and 67 factual examples.

These videos have a total of 151,567 attached comments, which are further annotated as agreeing and/or conspiratorial, with manual annotation for the first 10% followed by automated propagation using a BERT-based neural network model. Finally, different classifiers are trained using the video description and comment label proportions to classify the video itself, and a feature analysis is performed to detect tokens important for this classification.

The paper is well written and the overall approach is well designed. The results are presented clearly and their significance is analysed thoroughly. The authors show that using attached comments as additional information can improve video classification, although since both the dataset and the method are new, it's not possible to compare or contrast the performance improvement with previous work. However, the authors have already published their dataset so further research and validation of the results is possible. Licence information (preferably Creative Commons for the dataset) should be added to the GitHub repository.

The paper follows a well-established approach of using contextual text as a proxy for audiovisual content classification. The current work in Covid-19 misinformation detection is a special case of this larger research area, and while the authors briefly mention this in citing Momeni et. al. (2013), the current work should be more extensively connected to this related work. For example, there appear to be at least two well known publications already from 2010-2011 that describe textual proxies for video classification [1,2] and it is commonly known that e.g. Google image search uses contextual text for visual content classification (https://support.google.com/webmasters/answer/114016) and has been doing so for a long time.

As a technical issue there may be something odd about how the parameters for the various classifiers are optimized. For example, the authors mention that "For both SVM and random forest, we performed a grid search to obtain the best hyperparameters. In each run, we performed 10-fold cross-validation and report the mean accuracy in Table 2." This sounds like hyper-parameters were first optimized on the full data on which the final results were later generated, which sounds concerning, and should be clarified.

Finally, it is not clear how much of this classification approach is already used by YouTube itself, which most likely uses contextual text in its video classification algorithms. The authors mention that "In the case of Covid-19 misinformation, a large share of conspiratorial contents remain online on YouTube and other platforms, influencing the public, despite content moderation practices", but then go on to state that "We decided not to use YouTube’s search function as previous studies found few conspiratorial content on the top results". Doesn't this indicate that YouTube can already detect these conspiratorial videos, reflected by their low ranking in the search results, even if they have not been outright removed?

As a very minor issue, there appears to be a typo in "the percentage of conspiracy comments feature is based on _an_ aggregated _classifications_."

[1] Huang, C., Fu, T., & Chen, H. (2010). Text‐based video content classification for online video‐sharing sites. Journal of the American Society for Information Science and Technology, 61(5), 891-906.

[2] Filippova, K., & Hall, K. B. (2011, July). Improved video categorization from text metadata and user comments. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval (pp. 835-842).
review_title: A review of "NLP-based Feature Extraction for the Detection of COVID-19 Misinformation Videos on YouTube"

final_score: rating: 7: Good paper, accept
review_content: The study explores how user comments can be utilised to detect false information in Youtube videos related to COVID-19 origination. The authors did a great job reviewing and manually labelling a large number of videos and the corresponding comments. First, a model is created to detect user comments that convey agreement or amplify misinformation, referred to as conspiracy comments. Next, the percentage of conspiracy comments under each video is calculated and used along with video titles and raw comments to predict if a video contains conspiratorial content. This two-step approach allows the authors to avoid dealing with computationally-expensive video processing and efficiently detect COVID-19-related misinformation in Youtube videos. 

Pros: 
- A great effort in putting together the dataset of videos and comments;
- The paper is well-written and the ideas are clearly explained

Areas for improvement:
- I suggest clarifying in point 2 of the introductory section that other features (titles, raw comments) were used along with the percentage of conspiracy comments to detect COVID-19-related misinformation.
- There is a typo in “Pushshift Reddit API”
- It is said that conspiracy and agreement comments share 19.4% of their vocabulary and that only 1.95% of the vocabulary has more than four occurrences. Do you mean 1.95% of the shared vocabulary? Consider providing absolute numbers or restructuring the sentence to clarify the differences between the two collections. 
- Did you use cross-validation to report the training F1 scores (Table 1)? Did you change the threshold after analysing the precision-recall curves (Figure 1)?
- I suggest changing non-misinformation to factual in the legend for consistency (Figure 2).
- Please clarify how the data was split for the development of the second model, there should be a hold-out set to test the performance of the final model. It would be great if the authors could provide a confusion matrix for the final model and analyse the false-positives/-negatives. Given the dataset is slightly imbalanced, reporting the F1 score instead of the accuracy would also be more appropriate.
- In Table 2, it would be good to provide the results for title + comments for benchmarking.
- Please comment on why you chose exactly 10 most relevant words for Bayesian modelling. 
review_title: An elegant approach and a great effort in data labelling but the authors need to address several questions 

final_score: rating: 7: Good paper, accept
review_content: This work proposes a simple but effective method to detect COVID-19 misinformation YouTube videos by using the percentage of conspiracy comments. While the authors did not examine video contents at all, the classification accuracy reached 82.2 by considering video title and the percentage of conspiracy comments only.

pros:
- The approach is simple but works well.
- The complete YouTube comments dataset and the Colab notebook file are publicly available.
- Great efforts were made to construct COVID-19-related misinformation video datasets. 

cons:
- While I have some minor concerns/suggestions, none of them are serious flaws. 

comments:
- Details in the labeling process should be added: For video labeling, did two authors independently label videos, and were all the labels agreed? For comments labeling, more explanations on the actual labeling process are required. For example, how many labels are collected per comment? 
- Comparison with a bag-of-words model: As the authors already have all the text of comments, building a bag-of-words model based on the first 100 comments (videos with less than 100 comments were easy to predict) is not super hard but will make the author’s approach more concrete by adding another baseline. As there are several attempts to infer the political leaning of news articles based on their comments, bag-of-words models employing comments could be a solid baseline in this case as well. 
- Agree & Conspiracy labels in Table 1: drawing 2x2 table might be helpful to understand the characteristics of Agree and Conspiracy as the authors mentioned ‘their different linguistic properties.’ 
- Crowdtangle’s historical data of public Facebook posts (Section 3.1): citations should be added.

review_title: Simple but effective method to detect COVID-19-related videos containing misinformation

Meta-review: Thank you for your submission to the workshop.

Based on positive reviewer feedback, we are pleased to accept the paper for the workshop.

We look forward to your presentation on Thursday (5:30-9:30pm PDT) -- please plan on a 10-minute video presentation; pre-recorded is great.

Apologies for the late decision!
Decision: Accept