Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Title: Good work, requires some edits.

1. Summarize:

This paper proposes a new problem setting in visual relation detection which is called “Predicate Zero-shot Learning (PZSL)”. They provide a clear motivation and description of this setting. They propose a solution to this problem which leverages linguistic priors and knowledge bases. Furthermore they propose an unbalanced sampled-softmax to tackle the long tail distribution of predicates.

2. Clearly state your decision. One or two key reasons for this choice.

I will go for a weak accept for the paper at this stage. (+) I think the proposed problem setting is well-motivated and useful. Also, (+) the proposed initial solution to this problem is interesting. However, (-) they propose a “fast graph convolution network” which seems to be precisely equivalent to a PinSage.  Also, (-) the paper requires to be polished as it lacks clarity.

3. Main discussion

My first argument is: I’m not sure why the authors have changed the name of PinSage and just mentioned that “their” “Fast Graph Convolution Network” is “inspired” from PinSage. To me it looks exactly the same. If there are any differences, it should be stated clearly. In fact, I would not be against using PinSage as a part of their approach. However, trying to rename it without clear reasons is not a good idea.

My second argument is that the paper lacks clarity in writing (for detailed suggestions please refer to comments and feedbacks). Specially the evaluation section lacks details and clarity: a) In the beginning of this section (page 7), the authors talk about “generalized” and “traditional” settings without properly defining them. b) The descriptions for Table 1 and Table 2 fail to provide enough details to help understand the difference between the results in these two tables (one of them states “Accuracy of unseen predicate recognition” and the other one “Accuracy of recognition of triplets with unseen predicates”). 

4. Comments and feedback.

Introduction: 

Paragraph one in the: 
1. The relationship recognition methods are mainly supervised “that” → “to”.
2. last line: …. and do not study “on generalizing” → “the generalization of”.

Paragraph two:
1. no manual annotations or “real samples” → “image samples”. (a real sample is ill-defined)
2. For example, no instance of chew → For example “given” no instance of chew.

Paragraph three:
1. … is difficult since predicates are often abstract not as specific →   is difficult since predicates are often abstract “and” not as specific.
2. Furthermore, unlike many object ZSL methods … → This line to the end is very complicated and hard to understand.

Related Works:

1. Visual Relationships: I would cite “Graph R-CNN for scene graph generation” since it is the most relevant work regarding the similarity of pipeline (using GCNs).
2. External Knowledge bases  (KB): I would cite “Improving Visual Relationship Detection using
Semantic Modeling of Scene Descriptions” since it is one of the most relevant works using knowledge graph modelings to improve visual relation detection.

Problem Setup:

Do you plan to provide the proposed dataset splits so others can work on this setting? I consider this very important given your paper’s contribution. Maybe it is better if it is also mentioned in the paper.

Pipeline:

1. Paragraph 2: … the output of which is fused with …: Given Figure 2, it does not seem like $V_p$ is being created by fusing $V_s$ and $V_o$. It looks more like it is extracted directly from the image (union of bounding boxes).

2. In Figure 2: In the representation of Pipeline (A), the graph is colored by dark blue for objects and light blue for predicates. The represented graphs show Object to Object and Predicate to Predicate connections which I’m not sure if it is correct. Shouldn’t we always have a light blue between every pair of dark blue connections?

Evaluation:

1. Please consider the mentioned points in the Main Discussions.
2. In Table 1, I suggest re-naming “embedding” to “initial embedding”.
3. In Table 1, Hit@k should be Hits@k.
4. Please define the metrics clearly (Hits@k).

Extra: I have a question regarding the ablation studies with GloVe, Normal and InferSent initialization. The question is whether this initialization is necessary? It seems like in the setting “W/O KG”, even though the embeddings are initialized with GloVe, there is no gain (all of the Hits@k values are 0.0). So GloVe embedding without KG bring no external semantic knowledge? Then why use them? Regarding that, I can see that a GCN, initialized with normally distributed embeddings (row 14 in Table 1) has given 0.0 accuracies, but I find this very counter intuitive, as graph convolution layers already have trainable weights capable of compensating for the lack of ‘proper’ initialized embedding and getting 0.0 does not make sense to me.

Conclusions and Future Work:

1. two lines before the last: please use “\citep”.


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper considers the task of predicting visual predicates (e.g., eat, bite, take) between pairs of entities. In particular, the paper focuses on the zero-shot setting where the test predicates are unseen during training. The model uses linguistic prior from a knowledge graph (WordNet):
with graph embedding (fast GCN), unseen predicates are embedded based on the information propagated from seen predicates. The model is trained so that the visual feature vector and the correct predicate embedding are nearby in the joint embedding space. The method was evaluated on a zero-shot split of the Visual Genome dataset.

Overall, as a task and dataset paper, the paper should have sold the task more by highlighting its special properties. While the task of predicting unseen predicates is interesting, the setting and the technique are similar to previous work on predicting other types of unseen labels (e.g., unseen objects, as referenced in the paper). The new task could still be interesting if it presents different challenges (e.g., maybe predicates are more ambiguous than objects, or visual predicates are harder to embed). But from the description in the paper, most of the challenges seem to also exist in other zero-shot settings (long-tail distribution; large space of labels). The paper could benefit from providing examples or statistics that demonstrate the challenges of the task.

The proposed method looks correct but is a rather direct application of existing methods. The experiment setup looks OK. Based on the error analysis, the labels look very noisy and subjective, but this seems to be a common problem in the visual predicate prediction task (hence the recall-based evaluation).

Additional questions:

- The provided examples in the error analysis look pretty tricky; e.g., "swing" and "slug" are judged as different. How well would a human do on this task?

- How much would Hit@k be if the test label is seen during training (not zero-shot)?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper creates a new task for zero-shot learning of predicates (specifically in cases where the individual predicate components have never been seen in the training, rather than the more traditional setting where the full s-v-o relationship is unseen but each component is).  They create a new subset of the Visual Genome dataset specifically targeted towards predicting unseen predicates.  They also provide results using a knowledge graph (in this case WordNet), to integrate linguistic and visual features for prediction.  Interestingly, their pipeline introduces a new softmax variant, the unbalanced sampled softmax, which addresses the problem of over-predicting common predicates.

I generally tend towards accepting this paper.  The reasons being that this paper has a few strong contributions: (1) they design a new task set-up with data they selected and cleaned from VG, (2) new modelling pipeline with empirical analysis backing the modeling choices, and (3) new softmax variant.  

A few comments about things that could be strengthened or addressed further: 
- There could be more meaningful comparison to other zero-shot learning algorithms.  Even if they are not fully comparable because they were originally meant for a slightly different zsl set-up, it would be nice to have more baselines from external work.
- Why was the unbalanced sampled softmax was being used for only predicate prediction and not entity prediction? 
- It wasn’t totally clear to me whether all of the verbs/entities were in WordNet and/or Glove.  If not, can the authors clarify what the overlap was and how this might be affecting performance?
- As noted by the authors, there are some cases in Figure 3 where humans would consider the answer to be confusing or might actually prefer the machine response.  Can human performance on this task could be measured?  Perhaps humans could possibly evaluate a subset of the machine vs. gold answers?

Minor Edits:
- Related work: WordNet is misspelled in the last line
- Figure 3: please consider using colors other than red/green, this is not readable for color-blind readers
- Section 5.2: “The case c is confusing that even” →  “The case c is so confusing that even”
- Section 5.2: “and output a more appropriate” --> “and outputs a more appropriate”
- In the references: the Devise paper by [Frome et al 2013] is listed twice
- In the references: the first author of ConceptNet5 should be Robyn Speer
review_title: Official Blind Review #1

Meta-review: The paper proposes a new problem setting of predicate zero-shot learning for visual relation recognition for the setting when some of the predicates are missing, and a model that is able to address it.

All reviewers agreed that the problem setting is interesting and important, but had reservations about the proposed model. In particular, the reviewers were concerned that it is too simple of a step from existing methods. One reviewer also pointed towards potential comparisons with other zero-shot methods.

Following that discussion, I recommend rejection at this time but highly encourage the authors to take the feedback into account and resubmit to another venue.
Decision: Reject