Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper “MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts” details the construction of a manually annotated dataset covering biomedical concepts. The novelty of this resource is its size in terms of abstracts and linked mentions as well as the size of the ontology applied (UMLS). 
The manuscript is clearly written and easy to follow. Although other resources of this type already exist, the authors create a larger dataset covered by a larger ontology. Thus, allowing for the recognition of multiple medical entities at a greater scale than previously created datasets (e.g. CRAFT).
Despite the clarity, this manuscript can improve the following:
Section 2.3 – How many annotators were used?
Section 2.4, point 2 - The process used to determine biomedical relevance is not detailed. Section 4.1 - No reason is given for the choice of TaggerOne. In addition, other datasets could have been tested with TaggerOne for comparison with the MedMentions ST21pv results.
Misspelling and errors in section 2.3: “Rreviewers”, “IN MEDMENTIONS”
Overall, this paper fits the conference topics and provides a good contribution in the form of a large annotated biomedical resource.
review_title: MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts

final_score: rating: 7: Good paper, accept
review_content: In this paper the authors introduce MedMentions, a new dataset of biomedical abstracts (PubMed) labeled with biomedical concepts/entities. The concepts some from the broad-coverage UMLS ontology, which contains ~3 million concepts. They also annotate a subset of the data with a filtered version of UMLS more suitable for document retrieval. The authors present data splits and results using an out-of-the-box baseline model (semi-Markov model TaggerOne (Leaman and Lu, 2016)) for end-to-end biomedical entity/concept recognition and linking using MedMentions.

The paper describes the data and its curation in great detail. The explicit comparison to related corpora is great. This dataset is substantially larger (hundreds of thousands of annotated mentions vs. ones of thousands) and covers a broader range of concepts (previous works are each limited to a subset of biomedical concepts) than previous manually annotated data resources. MedMentions seems like a high-quality dataset that will accelerate important research in biomedical document retrieval and information extraction.

Since one of the contributions is annotation that is supposed to help retrieval, it would be nice to include a baseline model that uses the data to do retrieval. Also, it looks like the baseline evaluation is only on the retrieval subset of the data. Why only evaluate on the subset and not the full dataset, if not doing retrieval?

This dataset appears to have been already been used in previous work (Murty et al., ACL 2018), but that work is not cited in this paper. That's fine -- I think the dataset deserves its own description paper, and the fact that the data have already been used in an ACL publication is a testament to the potential impact. But it seems like there should be some mention of that previous publication to resolve any confusion about whether it is indeed the same data.

Style/writing comments:
- Would be helpful to include more details in the introduction, in particular about your proposed model/metrics. I'd like to know by the end of the introduction, at a high level, what type of model and metrics you're proposing.
- replace "~" with a word (approximately, about, ...) in text
- Section 2.3: capitalization typo "IN MEDMENTIONS"
- Section 2.4, 3: "Table" should be capitalized in "Table 6"
- Use "and" rather than "/" in text
- Section 4: maybe just say "training" and "development" rather than "Training" and "Dev"
- 4.1: Markov should be capitalized: semi-Markov
- 4.1: reconsider use of scare quotes -- are they necessary? 'lexicons', 'Training', "dev', 'holdout'
- 4.1: replace "aka" with "i.e." or something else more formal. In general this section could use cleanup.
- 4.1: last paragraph (describing metrics, mention-level vs. document-level) is very confusing, please clarify, especially since you claim that a contribution of the paper is to propose these metrics. Is it the case that mention-level F1 is essentially entity recognition and document-level is entity linking? An example could possibly help here.
review_title: Solid biomedical entity extraction/linking dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes a new resource "Med Mentions" for entity linking of Pubmed abstracts, where entities are concepts in the UMLS type hierarchy -- for example "Medical Device".

The annotations were manually verified. (I assume the intention is to use this as a benchmark, but the paper does not say)

The paper is very rigorous in describing which concepts were considered, and which were pruned. Authors suggest to combine it with "TaggerOne" to obtain end-to-end entity recognition and linking system.

It is a little bit unclear what the main contribution of this paper is. Is it a benchmark for method development and evaluation (the paper mentions the train/dev/test split twice)? or do the authors propose a new system based on this benchmark?, or was the intent to test a range of baselines on this corpus (and what is the purpose?) -- I believe this lack of clarity could be easily addressed with a change in structure of headings. (Headings are currently not helping the reader, a more traditional paper outline would be helpful.)

I appreciate that the paper lists a range of related benchmarks. However, I am missing a discussion of: where the advantage of MedMentions is in contrast to these benchmarks? What is MedMentions offering that none of the other benchmarks couldn't?


It is indisputable that a new resource provides value to the community, and therefore should be disseminated. However, the paper quality is more reminiscent of a technical report. A lot of space is dedicated to supplemental information (e.g. page 6) which would be better spent on a clear argumentation and motivation of the steps taken.


review_title: Useful resource, but claims could be better supported, and the uniqueness of the resource better argued

Meta-review: metareview: The paper provides a valuable new resource to the community, a data set of 350,000 mentions from 4000 abstracts, all linked to UMLS concepts.  MedMentions has some advantages over existing datasets that are either smaller in size, narrower in coverage of concepts, or only provide weakly supervised labels of the mentions (i.e., concepts are associated with an abstract, but not explicitly identified as mentions therein). The reviewers all agree that MedMentions would be a valuable resource for the community.  The main criticism of the paper is that the motivation and contribution were not initially clear; however, the authors have addressed this criticism in the responses and have already updated the introduction to make the motivation and contribution more explicit.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)