Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a discussion of vagueness of noun phrases along with a categorization of types of vagueness, and a corpus study that measures their prevalance in English Wikipedia. There ais a nice related work section that discusses efforts to identify and reason with vagueness in NLP, with a focus on formal reasoning systems. The authors suggest that their corpus could e a useful resource in creating detectors of vague NPs in a system that then has special reasoning capabiilties designed to deal with this vagueness.

I enjoyed reading the paper and I think that discussions of vagueness, underspecification, and ambiguity are becoming increasingly important as the reasoning capabilities of our NLP systems are reaching the point where these phenomena are important. However, I also think that the current paper would be stronger with a more formal definition of the process by which the two annotators identified vague NPs, in particular with relation to the effect of context. I'd also like to see a discussion of the pragmatic effect of vague NPs on a listener --- i.e. does it actually affect the listener's interpretation or is it mostly just an issue for set-theoretic modeling approaches.

The authors do mention the importance of context, but do not tel us how much context the annotators consider. If the annotators are considering anything less than the full document context, I would like to know to what extent the different forms of vagueness are reducible through the consideration of more context. Similarly, it would be nice to know to what extent the vagueness can admit signicantly different interpretations to a reasonable reader.

Strengths:
- Interesting discussion of vagueness, which is distinct from other types of ambiguity often discussed in the field.
- Nice overview of related work from adjacent fields.
- Enjoyable to read.

Weakensses:
- Definition of annotation process could be more formal. In particular w.r.t. amount of context considered and importance of pragmatics.
- Corpus is small and likely to be of limited utility.

review_title: Fun discussion and study of vagueness in NPs. Corpus limited in size and could be better defined.

final_score: rating: 6: Marginally above acceptance threshold
review_content: # General comments
This paper presents a fascinating attempt to define "vagueness" as it applies to noun phrases, and to annotate it and model it in natural language text.  The authors attempt to arrive at a working definition of NP vagueness and then annotate noun phrases using the simple schema they develop.  I have two main concerns: the first is that I found some of the definitional parts of the paper somewhat lacking in rigor, and the second is that I am not certain what the main position of the authors is, other than "vagueness is rampant in natural language" and "researchers have not explored it enough".  I *suspect* the position is that we need to do a better job of modeling vagueness so that our knowledge representations and our ability to answer questions about knowledge expressed in natural language form can be improved, but these points were things I could only surmise but were not exactly made explicit: they were raised in the introduction but not really fleshed out in the "Vision" or conclusion sections.

# Specific comments
* It is a little too strong to say that the use of any vague noun phrases makes a sentence "inaccessible to current knowledge extraction methods".  Perhaps the authors are thinking only about KB's/KG's that consist of named entities and facts related to those entities?
* We can and should differentiate with NP's that are vague in their current sentence, but which could be made more precise, either by understanding the greater discourse or social context, and/or by further research.  In the example sentence "An anti-tobacco sentiment grew in many nations from the middle of the 19th century", we can imagine that we could reasonably ask "in which nations did anti-tobacco sentiment rise after 1850?"
* The definition of ambiguity seems overly rigid.  A lot of work in lexical semantics and word sense disambiguation (yes, that was an intentionally vague phrase) has looked at resolving the ambiguity of even closely related definitions of words; see, e.g., the definitions of the verb "cut" in WordNet.
* It is somewhat odd to group "subjectivity" into "vagueness".  Subjectivity just means that the truth value is dependent upon the observer, but not that the concept is lacking in precision for whatever reason.  It certainly does mean that any "knowledge" that we might want to extract should probably be attached to that observer, as the authors note near the end of Section 6.
* Similar to the above, it is somewhat odd to conclude that "beauty" is "clearly vague".  I think it is that beauty is both a subjective concept *and* it is complex and/or hard to define.
* Some questions:
  * How much of the vagueness observed with noun phrases was resolvable when reading the entire document?  Natural language derives much of its expressive power and compactness due to its pervasive use of ambiguity and vagueness, coupled with our ability to come up with helpful interpretations based on many different types of context to resolve ambiguities and vague expressions.
  * You mention how a phrase can exhibit multiple forms of vagueness.  How often does that happen, and with what types?
  * In your "Vision", you mentioned that we should have a better, end-to-end treatment of vagueness.  Have you considered extending a question-answering dataset so that it includes questions where the answers—or support for the answers—lie in sentences with vague noun phrases?
review_title: fascinating exploration of vagueness in noun phrases, with many open questions and some gaps in the conceptual framework

final_score: rating: 6: Marginally above acceptance threshold
review_content: # General comments
In this paper the authors discuss the problem of vagueness in NLP, present a new categorisation scheme for vague noun phrases (NPs) and demonstrate through a case study annotation of Wikipedia passages that their categorisation is intuitive enough to lead to high inter-annotator agreement. 

The paper is a well-crafted resource for reviewing the literature in the field and the case study (and subsequent analysis) is well designed and potentially useful as a resource in itself. My only concern is that if we consider this exclusively as a position paper, other than "vagueness is still vague" it's not clear what the position is. I would have liked to see some more concrete suggestions in the final section on how the recommended categorisation scheme advances the research in the field, especially when it comes to practically representing vagueness in knowledge graphs and other NLP applications.


# Specific comments/questions
In the Definition paragraph of Section 2, it's not clear why "head" is not vague under the vagueness doctrine of American constitutional law. If this a hypothetical example, I would imagine a similar argument could be made for "heap" (where a court ruling or a law would define "heap" in terms of e.g. weight or countable items). If it is a concrete legal example, I would like to see some more details as to why "head" was able to defined but a term like "heap" wasn't.

The example of "beauty" is used to justify the rejection of previous categorisations but the reasons for that rejections are themselves vague. It's not clear why neither the "referent unclear" or the "overlapping of disjoint but overlapping concepts" definitions fit the type of vagueness described by "beauty". It would be helpful to see what types of disagreements or coverage gaps appeared during the annotation effort when using one of the existing categorisations. 

The terms "meaning" and "interpretation" used in the paragraph "Practical Considerations" should be defined.

Is the proposed categorisation system adhering to either definition given in Section 2? In particular the vagueness doctrine seems to provide a very useful functional definition, even for the example used in the first section. I can imagine for instance a word like "sentiment" being operationalised for a specific legal case. In addition, it would be an interesting exercise to annotate patents or other legal documents with the same annotation scheme applied to Wikipedia and compare the prevalence of vague words in the two fields.

A few questions on the Wikipedia annotation:
- Are named entities (proper nouns) included in the set of NPs considered for annotation? If not, how were they identified and removed?
- What tool was used for identifying the different PoS tags used the NP defintion?
- How many semantic classes were used in total?
- Were annotators allowed to choose more than one type of vagueness?

The result of the baseline classifier are already higher than the reported IAA (0.78 vs 0.744). This implies that the dataset is already "solved", unless there were efforts to ensure higher agreement post-hoc and the adjudication of disagreements was itself consistent.

The work by Kees van Deemter "Not Exactly: In praise of vagueness" is a great reference that needs to be included, not only for its coverage of different definitions and categorisations, but also because of the discussion of the practical aspects of dealing with vagueness in AI.


# Minor comments
"Weasel word" is used (Section 3) before being formally defined (Section 4)
review_title: A well-crafted and thought-provoking study of vagueness with limited outcomes.

Meta-review: This paper presents a corpus of noun phrases annotated with "vagueness", with "vagueness" itself split into multiple types (e.g., subjective versus quantitative). The paper is particularly relevant to AKBC because KBs often assume categorical, unambiguous entries, which vague NPs can thwart. The reviewers' overall sentiment was positive. The paper contributes an interesting dataset and its positioning in the literature should inspire good discussion. We also urge the authors to incorporate the reviewer's comments into the camera-ready version.
Decision: Accept