Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors suggest using a variational autoencoder to infer binary relationships between medical entities. The model is quite simple and intuitive and the authors demonstrate that it can generate meaningful relationships between pairs of entities that were not observed before.  
While the paper is very well-written I have certain concerns regarding the motivation, model, and evaluation methodology followed:

1) A stronger motivation for this model is required. Having a generative model for causal relationships between symptoms and diseases is "intriguing" yet I am really struggling with the motivation of getting such a model from word co-occurences in a medical corpus. I can totally buy the use of the proposed model as means to generate additional training data for a discriminative model used for information extraction but the authors need to do a better job at explaining the downstream applications of their model. 

2) The word embeddings used seem to be sufficient to capture the "knowledge" included in the corpus. An ablation study of the impact of word embeddings on this model is required. 

3) The authors do not describe how the data from xywy.com were annotated. Were they annotated by experts in the medical domain or random users?

4) The metric of quality is particularly ad-hoc. Meaningful relationships in a medical domain and evaluation using random amazon mechanical turk workers do not seem to go well together. 

5) How does the proposed methods compare against a simple trained extractor? For instance one can automatically extract several linguistic features of the sentences two known related entities appeared with and learn how to extract data. The authors need to compare against such baselines or justify why they cannot be used.

review_title: An interesting application

final_score: rating: 4: Ok but not good enough - rejection
review_content: SUMMARY.

The paper presents a variational autoencoder for generating entity pairs given a relation in a medical setting.
The model strictly follows the standard VAE architecture with an encoder that takes as input an entity pair and a relation between the entities.
The encoder maps the input to a probabilistic latent space.
The latent variables plus a one-hot-encoding representation of the relation is used to reconstruct the input entities.
Finally, a generator is used to generate entity pairs give a relation.

----------

OVERALL JUDGMENT
The paper presents a clever use of VAEs for generating entity pairs conditioning on relations.
My main concern about the paper is that it seems that the authors have tuned the hyperparameters and tested on the same validation set.
If this is the case, all the analysis and results obtained are almost meaningless.
I suggest the authors make clear if they used the split training, validation, test.
Until then it is not possible to draw any conclusion from this work.

Assuming the experimental setting is correct, it is not clear to me the reason of having the representation of r (one-hot-vector of the relation) also in the decoding/generation part.
The hidden representation obtained by the encoder should already capture information about the relation.
Is there a specific reason for doing so?


review_title: doubts on experimental setting

final_score: rating: 2: Strong rejection
review_content: In the medical context, this paper describes the classic problem of "knowledge base completion" from structured data only (no text).  The authors argue for the advantages of a generative VAE approach (but without being convincing).  They do not cite the extensive literature on KB completion.  They present experimental results on their own data set, evaluating only against simpler baselines of their own VAE approach, not the pre-existing KB methods.

The authors seem unaware of a large literature on "knowledge base completion."  E.g. [Bordes, Weston, Collobert, Bengio, AAAI, 2011],  [Socher et al 2013 NIPS], [Wang, Wang, Guo 2015 IJCAI], [Gardner, Mitchell 2015 EMNLP], [Lin, Liu, Sun, Liu, Zhu AAAI 2015], [Neelakantan, Roth, McCallum 2015], 

The paper claims that operating on pre-structured data only (without using text) is an advantage.  I don't find the argument convincing.  There are many methods that can operate on pre-structured data only, but also have the ability to incorporate text data when available, e.g. "universal schema" [Riedel et al, 2014].

The paper claims that "discriminative approaches" need to iterate over all possible entity pairs to make predictions.  In their generative approach they say they find outputs by "nearest neighbor search."  But the same efficient search is possible in many of the classic "discriminatively-trained" KB completion models also.

It is admirable that the authors use an interesting (and to my knowledge novel) data set.  But the method should also be evaluated on multiple now-standard data sets, such as FB15K-237 or NELL-995.  The method is evaluated only against their own VAE-based alternatives.  It should be evaluated against multiple other standard KB completion methods from the literature, such as Jason Weston's Trans-E, Richard Socher's Tensor Neural Nets, and Neelakantan's RNNs.

review_title: Lack of advantages over (or evaluations against) pre-existing work

Meta-review: The authors seem to miss important related literature for their comparison.
They also tuned hyperparameters and tested on the same validation set.
They should split between train/validation/test.

Reviews are just too low across the board to accept.
Decision: Reject