Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper summary:  This paper presents a method of learning word embeddings for the purpose of representing hypernym relations.  The learning objective is the sum of (a) a measure of the “distributional inclusion” difference vector magnitude and (b) the GloVE objective.  Experiments on four benchmark datasets are mostly (but not entirely positive) versus some other methods.

The introduction emphasizes the need for a representation that "able to encode not only the direct hypernymy relations between the hypernym and hyponym words, but also the indirect and the full hierarchical hypernym path.”  There has been significant interest in recent work on representations aiming for exactly this goal, including Poincare Embeddings [Nikel and Kiela], Order Embeddings [Vendrov et al], Probabilistic Order Embeddings [Lai and Hockenmaier], Box embeddings [Vilnis et al].  It seems that there should be empirical comparisons to these methods.

I found the order of presentation awkward, and sometimes hard to follow.  For example, I would have liked to see a clear explanation of test-time inference before the learning objective was presented, and I’m still left wondering why there is not a closer correspondence between the multiple inference methods described (in Table 3) and the learning objective.

I would also have liked to see a clear motivation for why the GloVE embedding is compatible with and beneficial for the hypernym task.  “Relatedness” is different than “hypernymy.”
review_title: Recent relevant work not adequately discussed or compared.  

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a joint learning method of hypernym from both raw text and supervised taxonomy data.  The innovation is that the model is not only modeling hypernym pairs but also the whole taxonomy. The experiments demonstrate better or similar performance on hypernym pair detection task and much better performance on a new task called "hierarchical path completion". The method is good motivated and intuitive. Lots of analysis on the results are done which I liked a lot. But I have some questions for the authors.

1) One major question I have is for the taxonomy evaluation part, I think there are works trying to do taxonomy evaluation by using node-level and edge-level evaluation. 'A Short Survey on Taxonomy Learning from Text Corpora:
Issues, Resources, and Recent Advances' from NAACL 2017 did a nice summarization for this. Is there any reason why this evaluation is not applicable here?

2) At the end of section 4.2, the author mentioned Retrofit, JointReps and HyperVec are using the original author prepared wordnet data. Then the supervised training data is different for different methods? Is there a more controlled experiment where all experiments are using the same training data?

3) In section 4.4, there are three prediction methods are introduced including ADD, SUM, and DH. The score is calculated using cosine similarity. But the loss function used in the model is by minimizing the L2 distance between word embeddings? Is there any reason why not use L2 but cosine similarity in this setting? Also, I'm assuming SUM and DH are using cosine similarity as well? It might be useful to add that bit of information.

4) The motivation for this paper is to using taxonomy instead of just hypernym pairs? Another line of research trying to encode the taxonomy structure into the geometry space such that the taxonomy will be automatically captured due to the self-organized geometry space. Some papers including but not restricted 'Order-Embeddings of Images and Language', 
'Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures'. Probably this line of work is not directly comparable, but it might be useful to add to the related work session.

A few minor points: 
1) In equation four of section 3, t_max appears for the first time. This equation maybe part of the GLOVE objective, but a one-sentence explanation of t_max might be needed here.
2) at the end of section 3, the calculation of gradients for different parameters are given, but the optimization is actually performed by AdaGrad. Maybe it would be good to move these equations to the appendix.
3) In section 4.1 experiment set up, the wordnet training data is generated by performing transitive closure I assume? How does the wordnet synsets get mapped to its surface form in order to do further training and evaluation?

review_title: Well motivated approach but some concerns

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method to jointly learn word embeddings using co-occurrence statistics as well as by incorporating hierarchical information from semantic networks like WordNet. 

In terms of novelty, this work only provides a simple extension to earlier papers [1,2] by changing the objective function to instead make the word embeddings of a hypernym pair similar but with a scaling factor that depends on the distance of the words in the hierarchy.

While the method seems to learn some amount of semantic properties, most of the baselines reported seem either outdated or ill fitted to the task and do not serve well to evaluate the value of the proposed method for the given task. 
For example the JointRep baseline is based on a semantic similarity task which primarily learns word embeddings based on synonym relations and seems to not be an appropriate baseline to compare the current approach to.
Further, there are two primary methods of incorporating semantic knowledge into word embeddings - by incorporating them during the training procedure or by post processing the vectors to include this knowledge. While I understand that this method falls into the first category, it is still important and essential to compare to both types of strategies of word vector specialization. In this regard [3]  has been shown to beat HyperVec and other methods on hypernym detection and directionality benchmarks and should be included in the results. It would be also interesting to see how the current approach fares on graded hypernym benchmarks such as Hyperlex. 

Minor comments : Section 4.2 there is a word extending out of the column boundaries. 


[1] Alsuhaibani, Mohammed, et al. "Jointly learning word embeddings using a corpus and a knowledge base." PloS one (2018)
[2] Bollegala, Danushka, et al. "Joint Word Representation Learning Using a Corpus and a Semantic Lexicon." AAAI. 2016.
[3] Vulić, Ivan, and Nikola Mrkšić. "Specialising Word Vectors for Lexical Entailment." NAACL-HLT 2018.
review_title: Needs better evaluation 

Meta-review: metareview: All reviewers voiced concerns regarding the comparison to recent related work. However, in my view, the authors addressed these concerns well in their revision, comparing directly against Poincaré embeddings and LEAR. While the comparison reveals mixed results with respect to LEAR, I believe this work is well executed and of interest to the AKBC community. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)