Imposing Category Trees Onto Word-Embeddings Using A Geometric Construction

Abstract: We present a novel method to precisely impose tree-structured category information onto word-embeddings, resulting in ball embeddings in higher dimensional spaces (N-balls for short). Inclusion relations among N-balls implicitly encode subordinate relations among categories. The similarity measurement in terms of the cosine function is enriched by category information. Using a geometric construction method instead of back-propagation, we create large N-ball embeddings that satisfy two conditions: (1) category trees are precisely imposed onto word embeddings at zero energy cost; (2) pre-trained word embeddings are well preserved. A new benchmark data set is created for validating the category of unknown words. Experiments show that N-ball embeddings, carrying category information, significantly outperform word embeddings in the test of nearest neighborhoods, and demonstrate surprisingly good performance in validating categories of unknown words. Source codes and data-sets are free for public access \url{https://github.com/gnodisnait/nball4tree.git} and \url{https://github.com/gnodisnait/bp94nball.git}. 

Reviews and author responses:

This paper proposes N-ball embedding for taxonomic data. An N-ball is a pair of a centroid vector and the radius from the center, which represents a word.

Major comments:

- The weakness of this paper is lack of experimental comparisons with other prominent studies. The Poincare embedding and the Lorentz model are recently proposed and show a good predictive performance in hypernymy embedding.
- WordNet concepts are actually structed in DAG. Recent studies on structure embedding can hadle DAG data. It is not clear how to extend N-ball embedding for handling DAT structures. 

- Related work is not sufficiently described.

- It is not clear why N-ball embedding is suitable for hierarchical structures.


Thanks for the comments. Unfortunately, three reviewers misunderstood or neglected the main point of this work.  

Our work is targeted at the possibility of strictly imposing structures into  deep-learning systems. Such imposing must achieve global minimum (zero energy cost).  We imposed external tree structures into word-vectors produced by neural networks, achieving zero energy cost (this has not been achieved in the literature of representational learning). Our embedding also leads to perfect experimental results – no deep-learning systems in the literature has achieved 100% precision, as their training processes are only targeted at local minimums.  We would not use models, which only achieving local minimum, to compare with our model. This is not our aim.

Our aim is to perfectly impose external structures into vectors produced by deep-learning models, while keeps these vectors unchanged as much as possible, which can be reviewed as an optimization for representation learning (one of the topics of iclr-19).



 







thanks for the detailed comments, though not acceptable. 

the anonymization issue is carefully taken cared of.  the github account is temporarily created for this projects. readers/reviews cannot figure out who we are.

on the other hand, the iclr submission policy allows papers that have appeared on non-peered reviewed websites (like arXiv).  therefore, anonymization  is not an issue.



Attention!!! This submission contains Github and Google Drive links to author-related accounts (see e.g. the abstract). I do not think this is permitted or standard. I leave the decision regarding "automatic rejection" of the submission to meta-reviewers of the paper.
------------------------------------------------
The paper presents a method for tweaking existing vector embeddings of categorical objects (such as words), to convert them to ball embeddings that follow hierarchies. Each category is represented as a Eucldiean norm ball in high dimensional space, with center and radii adaptable to data. Next, inclusion and exclusion constraints on each pair of  balls are imposed based on the hierarchical structure. These constraints are imposed via an algorithmic approach.  The empirical study includes investigating the consistency of the representation with the hierarchy and demonstrating nearest neighbors for a set of words.

On the positive side, the paper addresses an important problem. It is readable and well organized. The related work could be improved by adding a number of representative related works such as [3,4].  

The major concern about the paper is the originality of the method. Encoding hierarchies with high dimensional balls and encoding inclusion and exclusion as constraints on those balls is a neat and powerful idea from modeling perspective. However, it is not novel, since the approach is already established for example in [1, 2 Chapter 5]. 
The next major concern is regarding the evaluation of the quality of embeddings.
The empirical evaluation does not sufficiently evaluate the quality of tweaked embeddings. In contrast, the quantitative evaluation is more concerned with if the embeddings being consistent with the given hierarchy. In particular, not enough quantitative evidence that the proposed embeddings are actually effective in capturing semantics or in prediction tasks is provided. It should be noted that, the first aspect, ie consistency of the feasible solutions with hierarchy, can be theoretically established (see e.g. [1]).  The first paragraph of 3.2 seems unclear or wrong. See for example [2] for a gradient based solution for the problem.
Finally, using an algorithmic approach as opposed to learning method for constructing embeddings, makes the method not directly related to the topic of ICLR conference.

Overall, due to the above reasons, I vote the paper to be rejected. (The poor anonymization makes it a strong case for a reject.)

[1] Mirzazadeh, F., Ravanbakhsh S., Ding N., Schuurmans D.,  "Embedding inference for structured multilabel prediction", NIPS 2015.
[2] Mirzazadeh, F."Solving Association Problems with Convex Co-embedding", PhD thesis, 2017. (Chapter 5)
[3] Vilnis, Luke, and Andrew McCallum. "Word representations via gaussian embedding.", ICLR 2015.
[4] Vendrov, I., Kiros, R., Fidler, S., Urtasun, R. "Order-embeddings of images and language." ICLR 2016.

This paper focuses on adjusting the pretrained word embeddings so that they respect the hypernymy/hyponymy relationship by appropriate n-ball encapsulation. They propose to do so by augmenting the word embeddings with information from a resource like Wordnet and applying 3 kinds of geometric transformations to enforce the encapsulation.

The motivation of doing this is not very clear and experimental results are mainly qualitative (and subjective) showing that hypernymy relation can be predicted and preserved by their adjustment. Since, this work relies on Wordnet, the coverage of vocabulary is severely limited and as the authors discus in the results with the section titled "Experiment 3: Method 2", they had to remove many words in the standard semantic similarity datasets which casts shadow on the usefulness of the proposed approach. It is unclear what the main contribution of such an approach.

Apart from this, the paper is diffcult to read and some parts (especially those pertaining to Figure 3) encode a simple concept that has been expressed in a very complicated manner.

Overall, I give a score of 4 because of the limited coverage of the approach because of reliance on Wordnet and inadequate empirical evidence of usefulness of this approach.

Meta-review: The authors provide an interesting method to infuse hierarchical information into existing word vectors. This could help with a variety of tasks that require both knowledge base information and textual co-occurrence counts.
Despite some of the shortcomings that the reviewers point out, I believe this could be one missing puzzle piece of connecting symbolic information/sets/logic/KBs with neural nets and hence I recommend acceptance of this paper.

Decision: accepted-poster-papers