Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a method of extracting document class labels from the Wikipedia category hierarchy; the set of sub-areas on StackOverflow; and the list of sub-reddits on Reddit. These pairs are then used to train a BERT-based binary classifier that predicts whether a category matches a document. Since categories are named with natural language, the model can be applied to new classification tasks that have natural language labels.

When applied to a number of few-shot classification taskss, the model outperforms baselines that don't make use of large pre-trained language models. On average, it is beneficial to use all three sources of distant supervision, but this is not true for each individual task.

The results would be stronger with a comparison to a zero-shot approach that makes use of a large pre-trained LM but doesn't make use of the source of weak supervision. Currently, it is not clear where the accuracy improvements are coming from. An alterative way to show the importance of the NatCat data would be to report accuracy results for the BERT/RoBERTA models that have been trained on 1%/10%/50%/100% of the available dataset.

Strengths:
- Aggregation of three different sources of weak supervision is yielding nice results on zero-shot classification.
- Data may be useful resource for future work.

Weaknesses:
- Experiments don't properly untangle impact of NatCat weak supervision vs. adoption of large pre-trained LM.
- No comparison to recent work on zero-shot classification.

review_title: Interesting new source of weak supervision. Would be stronger with comparison to more recent work in zero-shot classification.

final_score: rating: 6: Marginally above acceptance threshold
review_content: # General comments
This paper presents a dataset and evaluation benchmark for weakly supervised text classification. The main resource, NatCat, contains a large number of diverse real-world documents and categories, making it appealing for general-purpose off-the-shelf text classification. The way of generating weakly supervised data is efficient, intuitive and easy to reproduce. The baseline models trained on this resource easily outperform previous weakly supervised approaches.

The main weakness of this work stems from the lack of support for its main hypothesis. This is a training and evaluation resource but instead of comparing against other resources, the main comparison (Tables 4, 5) is between multiple models trained on the same resource. If the motivation for this work is that NatCat is a better resource for text classification, the experiments need to demonstrate exactly that. 

Similarly for the CatEval benchmark - currently, the contribution is the consolidation of multiple existing evaluation datasets. There is nothing in the paper to test if CatEval is a better resource than existing benchmarks for this task or whether the datasets comprising CatEval are the best choices for the purpose.

Another practical consideration regarding any weakly supervised †raining resource is how much domain-specific fully supervised data does it take to get a better performance.


# Specific comments/questions
It's not clear what the example is Table 1 illustrates. Are the highest-scoring categories from NatCat a good fit? As there is no overlap with the AGNews categories (including the true one) wouldn't the score of the example be 0? Furthermore, if the categories from AGNews are part of NatCat (as is claimed in Section 3.1) why are they not appearing even further down the list?

None of the datasets used in CatEval has nearly as many labels as there are in NatCat (especially the Wikipedia part). Does this cause problems for the models? Would it be better for the number of categories in NatCat to be consolidated to match those found in CatEval?

Why are only the common categories (subreddits) chosen from Reddit, but not Wikipedia? What are some examples of subreddits that fell below the threshold?

The result of "superhuman" performance is a bit suspicious. To confirm this is semantic drift from the annotators' side, the same annotation should be performed with training sessions for each annotator. The alternative possibility is that the categories aren't applied consistently among the users of the websites comprising the dataset (which is plausible in community-generated content).

I disagree with the analysis in 5.1 claiming that Wikipedia is a better resource for training sentiment classifiers than Stack Exchange. Wikipedia's editorial style aims to be as objective as possible.


# Minor comments
- Incorrect capitalisation in the second paragraph of Section 1 ("of NatCat, We" -> "of NatCat, we")
- Inconsistent non-textual citation style applied to some references ([Yogatama et al., 2017] instead of Yogatama et al. [2017])
- Add the #words/doc statistic to Table 2.
- I don't think median was the statistic used in reporting the results over 5 runs (Section 4.1). Maybe the authors meant mean? Also, the standard deviation/error should be reported for completeness.
review_title: Good general-purpose text classification resource with lack of proper validation

final_score: rating: 7: Good paper, accept
review_content: **Summary:** The paper considers the task of distantly supervised text classification. The model cannot access the labels in the target tasks. Instead, weak labels constructed from Wikipedia, Stack Exchange, and Reddit are used to train the model. The main contributions are:

- Scraped and formatted weakly labeled texts from Wikipedia (labels = categories), Stack Exchange (labels = subarea names), and Reddit (labels = subreddit names).
- A compiled benchmark with 11 datasets. The datasets are diverse (topic, sentiment, and emotion labels; single-label and multi-labels).
- A recipe to train the model on weak labels (apply BERT/RoBERTa on the input concatenated with the label, then predict a score). The paper also shows results on other potential recipes (e.g., cosine similarity to the label embedding, or generative models such as GPT-3) in the appendix.

**Strengths:**

1. The experiments are very thorough.
    - Alternative solutions considered are stated and their results are included in the appendix.
    - The experiments were on a diverse collection of tasks with different label spaces and numbers of labels per input.
    - The ablation and error analysis are also informative. The comparison against human raters also show that the types of errors made are reasonable.
2. The resources provided would be very useful for other researchers.

**Weaknesses:**

1. If I am not mistaken, the main baseline is quite old (a pre-neural model from 2008).
    - That said, there is a comparison against Yin18 at the end of Section 4.3. I wish this is used as the main baseline. Is these results comparable to the setup in Table 4?
2. In practical scenarios, a weakly supervised model is often fine-tuned on at least a small amount of target data (so that it's familiar with the target label space). The paper does consider this semi-supervised setup in Appendix D (the "Half Seen" setting). The result in Table 8 shows that the model greatly degrades on the "unseen" subset after fine-tuning on the "seen" subset, which is undesirable.
    - However, this overfitting issue might be fixable via regularization, such as a distance-based penalty when the parameters move too far from the initial parameters.
    - The overfitting problem seems to also manifest in Section 5.2, where training on a bigger training dataset hurts.

**Other comments:**

1. Section 3.1: "The choice of label names can have a large impact on performance." --- It would be interesting to do an ablation on the choice of target label names.
2. Section 4.1: Are the negative labels chosen randomly? Wouldn't such negatives be too easy to solve by the model?
3. Section 4.1: For the "encode the document and category using pretrained models" --- are these pretrained on the NatCat task? If not, how well does the cosine-similarity model tuned on NatCat perform?
4. Section 4.3: Why is the human performance on Yahoo much higher than the models (including the supervised one)?
5. It is still strange that the model trained on Wikipedia labels is so good at sentiment (Table 5). Are there Wikipedia categories that resemble sentiment or emotion labels? The Reddit results is more understandable due to sentiment-like subreddit names (like oddlysatisfying or mildlyinfuriating).

**Edit suggestions:**
- Section 1: "StackOverflow" --> "Stack Exchange"?
- Section 4.3: Yin 18 has an incorrect citation. Should be "Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach"
- Section 6: "Yin et al. [2018] and" --- wrong Yin18 and a missing second reference.
review_title: Useful resources + nice distantly supervised results; Old baseline + overfitted semi-supervised results

Meta-review: The reviewers agree that the resource presented is likely to be of interest to the community. However, they also point out that the paper has methodological issues. e.g. the resource is not compared as such to any other resources to demonstrated its value. The models used in the comparisons are out of date, and some of the conclusions drawn are over-stated.  We strongly urge the authors to incorporate reviewers' comments and address these issues in the camera-ready version.
Decision: Accept