Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The manuscript focuses on a trending topic of applying a Bidirectional Encoder Representations from Transformers (BERT)-based prediction model to a new domain. More precisely, it addresses classifying Electronic Recruitment Records (ERRs) with respect to job skills. Its contributions include, but are not limited to, (i) releasing a related de-identified ERR dataset to the public domain, (ii) introducing a BERT-based embedding model, called SkillBERT, to group skills present in this ERR dataset into as competency clusters, and (iii) giving experimental evidence of the obtained modelling gains. 

However, I am not convinced that these experiments are sufficient to support accepting the manuscript for the following five main reasons:

First, the compared models and methods are somewhat old and elementary (e.g., word2vec and TFxIDF), thereby failing to capture the current advances. 

Second, in my opinion, the data annotation process calls for clarifications. For example, what was the expertise of the expert annotators, how were research ethics and informed consenting assured in this process involving human annotators as study participants, and do the authors obtain the rights to release the dataset and/or annotated dataset?

Third, the overall experimental setting does not seem to be adequately captured and justified, and I am unable to find a description of the performed statistical significance testing.

Fourth, the manuscript demonstrates only a reasonable understanding of related work in applications to skill/competency demand and existing studies in relevant computational methods/models/data (see, e.g., https://www.aclweb.org/anthology/search/?q=%22job%22 for recent relevant papers from the ACL Anthology that are largely missing from the reference list).

Fifth, the manuscript should be edited more carefully by clarifying both its contributions and limitations in relation to related work; describing and justifying its methodology and experiments; moving the in-text citations from the abstract to the body text of the later sections; and enhancing the image readability.

In conclusion, the study is valuable but needs further work.
review_title: Review of the submission called SkillBERT: “Skilling” the BERT to classify skills!

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a model for job application screening. Since there is no job-related dataset available, the authors manually assigned labels to a large job application dataset. A skill set (e.g., Apache Hadoop, Apache Pig, HTML, Javascript) is firstly extracted from the job dataset.  Then a competency group is constructed (e.g., big data, front-end) as the labels. The problem is then formulated as a multi-label classification problem. That is, given a skill (which may belong to multiple competency groups), the model has to predict its competency groups. The authors proposed to use BERT as the main model. Moreover, the authors use additional features like similarity-based and cluster-based features. The experimental results are good. We think it can help recruiters find a suitable applicant.

However, this paper is straightforward. Using BERT as a main model for text classification is a well-known technique, and many papers already applied BERTs in other domains like biomedicine and law. So we think the technical contribution of this paper is limited. Furthermore, some parts of this paper are not clearly explained. For example, the authors mentioned that they also use some features like frequency-based and group-based features, but did not find detailed descriptions of these two features.

One positive side of this paper is that the authors release a publicly available job application dataset.
Establishing a dataset is time-consuming, and requires a lot of human efforts. The dataset consists of 700,000 job requisitions, which is large enough. It is good that the authors are willing to share this dataset.

To sum up, this paper proposes a skill classification model for job screening, which is useful, but we think that the methodology and its technical contribution are not strong enough. It might not be qualified as a regular paper for ICLR.

SkillBERT: "Skilling" the BERT to Classify Skills
review_title: SkillBERT: "Skilling" the BERT to Classify Skills, novelty low, an application case study.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studied the problem of classifying skills into competency groups. The authors proposed SkillBERT, a BERT-based model, to extract the embeddings of skills and use that for the classification task. 

Strengths:
1. The authors presented the details of feature engineering and experiment design. They also conducted extensive and comprehensive experiments which compare a lot of classification/feature engineering methods. It is a very good practical guide for doing related tasks.

2. The authors released the code and dataset, which largely improve the reproducibility of this work.

3. The paper is well written and easy to follow.

Weakness:
1. This paper studied a very specific problem which might only be interested to a very small group of researchers. It seems more like an industry track paper. 

2. If I am correct, the problem size is relatively small. The total number of skills is 2997, and each of them will be classified into one or more competency group out of 40 competency group. Do we really a BERT model for this problem? Maybe the authors could provide some cost-performance tradeoff analysis here.

Questions for authors:
1. In Table 3, I only see the precision/recall/F-1 score for class 0 and class 1. Where is the number for class 2? Did I misunderstand this experiment?

Overall comments:

This is a good paper and I personally support it to be accepted. However, I do think that it seems more like an industry track paper. This is not for me to decide, maybe Meta-reviewer could share some opinions about whether this paper is suitable to ICLR.
review_title: Official Blind Review #3

Meta-review: The authors propose an approach for the task of categorizing competencies in terms of worker skillsets. This is a potentially useful (if somewhat niche) task, and one strength here is a resource to support further research on the topic. However, the contribution here is limited: The methods considered are not new, and while the problem has some practical importance it does not seem likely to be of particular interest to the broader ICLR community. 
Decision: Reject