Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work aims to address the problem of answering visual-relational queries in knowledge graphs where the entities are associated with web-extracted images.

The paper introduces a newly constructed large scale visual-relational knowledge graph built by scraping the web. Going beyond previous data sets like VisualGenome having annotations within the image, the ImageGraph data set that this work proposes allows for queries over relations between multiple images and will be useful to the community for future work. Some additional details about the dataset would have been useful such as the criteria used to decide "low quality images" that were omitted from the web crawl as well as  the reason for omitting 15 relations and 81 entities from FB15k. 

While existing relational-learning models on knowledge graphs employ an embedding matrix to learn a representation for the entity, this paper proposes to use deep neural networks to extract a representation for the images. By employing deep representations of images associated with previously unseen entities, their method is also able to answer questions by generalizing to novel visual concepts, providing the ability to zero-shot answer questions about these unseen entities.

The baselines reported by the paper are weak especially the VGG+DistMult baseline with very low classifier score leading to its uncompetitive result. It would be worth at this point to try and build a better classifier that allows for more reasonable comparison with the proposed method. (Accuracy 0.082 is really below par) As for the probabilistic baseline, it only serves to provide insights into the prior biases of the data and is also not a strong enough baseline to make the results convincing.

Well written paper covering relevant background work, but would be much stronger with better baselines.
review_title: Interesting approach to a new task + new data set

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper introduces several novel tasks for visual reasoning resembling knowledge base completion tasks but involving images linked to entities: finding relations between entities represented by images and finding images given an image and a relation. The task is accompanied with a new dataset, which links images crawled from the web to FreeBase entities. The authors propose and evaluate the first approach on this dataset.

The paper is well written and clearly positions the novelty of the contributions with respect to the related work.

Questions:
* What are the types of errors of the proposed approach? The error analysis is missing. A brief summary or a table based on the sample from the test set can provide insights of the limitations and future directions.
* Is this task feasible? In some cases information contained in the image can be insufficient to answer the query. Error analysis and human baseline would help to determine the expected upper-bound for this task.
* Which queries involving images and KG are not addressed in this work? The list of questions in 4.1. can be better structured, e.g. in a table/matrix: Target (relation/entity/image), Data (relation/entity/image)
review_title: Compelling new task and dataset

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes to extend knowledge base completion benchmarks with visual data to explore novel query types that allow searching and completing knowledge bases by images. Experiments were conducted on standard KB completion tasks using images as entity representations instead of one-hot vectors, as well as a zero shot tasks using unseen images of unknown entities.

Overall I think that enriching KBs with visual data is appealing and important. Using images to query knowledge bases can be a practical tool for several applications. However, the overall experimental setup suffers from several problems. The results are overall very low. In the non-zero shot experiments I would like to see a comparison to using entity embeddings, and maybe even using a combination of both, as this is the more interesting setup. For instance, I would like to see whether using images as additional information can help building better entity representations. The explored link prediction models are all known, so apart from using images instead of entities there is very limited novelty. The authors find that concatenation followed by dot-product with relation-vector works best. This is very unfortunate because it means that there is no interaction between h and t at all, i.e.: s(h, r, t) = [h;t] * r = h * r_1 + t * r_2. This means that finding t given h,r only depends on r and not on h at all. Finally, this shows that the proposed image embeddings derived from the pretrained VGG16 model are not very useful for establishing relations.

Given the mentioned problems I can unfortunately not recommend this paper for acceptence.

Other comments: 
- I wouldn't consider a combination of pretrained image embeddings bsaed on CNNs with KB embeddings a "novel machine learning approach", but rather a standard technique   
- redefine operators when describing LP models: \odot is typically used for element-wise multiplication, for concatenation use  [h; t] for instance
- (head, relation, tail) is quite unusual -->  better: (subject, predicate, object)
- baselines are super weak. Concatenation should be the baseline as it connects h and t with r indepedent of each other. What is the probabilistic baseline?
review_title: REVIEW

Meta-review: metareview: This paper introduces a useful new dataset called ImageGraph that allows for the assessment of tasks on the combination of images and knowledge graphs. The paper presents a number of tasks over that datasets and architectures to address those tasks. The reviewers agree that the baselines could be improved upon and there is a question as to whether the architectures are promising or not. 

I think the paper should be accepted because the dataset is fundamentally useful and the authors establish good baselines for the considered tasks. Additionally, using KGs with images together is really promising. However, joint image + kg embeddings have already been investigated elsewhere see [1]. I would recommend that the authors soften their claims of novelty. The work is useful and points to a number of good directions for future work. 

[1] Towards Holistic Concept Representations: Embedding Relational Knowledge, Visual Attributes, and Distributional Word Semantics. S Thoma, A Rettinger, F Both. International Semantic Web Conference, 694-710
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)