Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper explores the problem of generalizing to novel combinations of verbs and nouns in a task for captioning video stills from videos about cooking. The paper introduces a new dataset based off of EPIC-Kitchens (Damen et al. 2018) which masks out verbs and nouns and splits the evaluation data into seen combinations of verb/noun pairs and unseen combinations of verb/noun pairs, challenging a model to generate captions for pairs which were not seen during training. 

The paper also proposed a model, ARTNet, to address this task. The model works by first finding nearest neighbor examples in the training data that are similar in (masked) caption and image, then embedding these similar examples into an "analogical embedding", which is used to condition a network that fills in the masked verb and noun in the target example.

*************

Reason for score: 

The proposed dataset is interesting, and the proposed model works well on it. However, I have some concerns about the dataset's construction and suggestions for additional experiments.

*************

Strengths:

1. The proposed model performs well on the proposed task.
1. The baselines seem solid.
1. Includes experiments on different amounts of training data.

*************

Weaknesses:

1. Some missing citations for compositional reasoning on novel images/visual contexts, including datasets like VQA-CP (Agrawal et al. 2017), CLEVR CoGent (Johnson et al. 2017), Room to Room (Anderson et al. 2018), and ALFRED (Shridhar et al. 2020).
1. Would be good to see experiments with a differing number of images in the random pool -- how does this affect performance to have a larger or smaller pool? There seems there would be a tradeoff between accuracy and computing resources necessary to run inference, but what exactly is that tradeoff? Similarly, would be good to see a similar analysis for the number of top candidate examples that are passed to the ARN module from the AMM module.
1. Many terms in the model section should be defined more formally, e.g., "candidate analogy composition", "ordered constituents in a composition", "pars(ing) each sample into candidate analogy compositions", "multimodal resulting set of pairs / analogy pairs", "linguistic clues".

*************

Questions:

1. How does the split for EPIC-Kitchens handle synonyms of verbs? It seems to split based off of verb/noun combinations, but what about things that are (at least partially) synonyms such as "cut", "slice", "chop"? This also seems to be relevant for evaluation -- is it strict for synonyms? (I am looking at the case study, which proposes "slice" squash, "chop" squash", and "cut" squash, all of which seem reasonable to me.)

review_title: Recommendation: accept

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary

The paper describes a model to predict unseen verb-noun compositions from a fixed set of verbs and nouns, given a training set of seen compositions. For an image with incomplete sentence, the model learns to select a set of relevant examples from the train set to serve as reference pairs of image-sentence mapping, and then uses these pairs to draw distribution over potentially unseen verb-noun pairs.


Strengths
- The problem is well-motivated
- The model is shown to perform well both quantitatively and qualitatively, along with different experimental setups such as validity, low-data


Weaknesses and concerns

- Hanging notations and unexplained variables:  Unfortunately this is a bottleneck for me to understand the paper thoroughly
    - Equation 2
        - What is $a$?
            - $a$ is used as a function within the softmax expression - $\exp a()$, but $a$ is also used as a subscript $c_a$, and since I cannot find any reference for $a$, I’m unclear as to what it actually is
        - The leftmost expression marginalizes out $j$ to yield $c_a^m$, but where does $i$ go? Are $a$ and $i$ related?
        - What is $k$? Is $k$ the cardinality of the set of all analogy pairs?
    - Equation 8 - Triplet loss
        - What is $v_i$?
- Results:
    - Potential non-determinism
        - You mentioned that for each test sample, you randomly pick 200 examples and then select top-K examples from them (ref: Section 2.2). Now if I understood this correctly, this introduces non-determinism in your results. Is it fair to say that your results should report standard deviations along with the numbers currently stated? Are the improvements statistically significant then?
    - Lack of ablation
        - Frequency bias in predictions? What’s the accuracy when you pick the most common verb, noun, and verb-noun pair?
        - Choice of K in Top-K
            - The Appendix (Implementation details) mentions that you pick K as 3. Why? What happens when you increase or decrease the value. Is there any insight to be gained from here?
        - Accuracy numbers for nouns and verbs separately
            - Again, this is an attempt to uncover biases in your predictions. You mark a prediction correct if both verb-noun pair gets correctly classified, but which one of them gets more wrong? Is there something to be learned here?
    - Trends in Table 1
        - BERT pre-trained v/s scratch: Why is there a reverse trend in top-1 and top-5 accuracy? Is there a conclusion to be drawn here?

- Misc
    - Number of objects (visual tokens) and words (lingual tokens) in an image?
        - What is the distribution of number of object and words in the dataset for an image-sentence pair?
    - Figure 3 - Attention over visual
        - What is “ref1_full_img”, “ref2_full_img”? Are complete images also included in analogy pairs along with objects?
    - Section 3.4
        - “We annotate the validity accuracy for 8400 Top-1 verb-noun compositions” How many people were involved? What does the distribution look like?
    - Reasoning module:
        - (e.g. “washing carrot” = “washing apple’ + “cutting carrot” - “cutting apple”)
            - Is this arithmetic observed in predictions?


Minor concerns (suggestions, typos, etc.)
- Section 3.4: grammatical errors


Preliminary Rating and its justification

The current version of the paper seems to have a couple of loose ends in terms of clarity and completeness of experiments. However, I’m open to changing my rating if the above concerns are addressed to some extent.

review_title: Clarity and ablation

final_score: rating: 3: Clear rejection
review_content: This paper presents a model that takes in a keyframe from a video and emits the noun and verb best matching what is being done in the frame.
At inference time, the noun and verb have never been seen in combination with one another, but have each been seen paired with other nouns/verbs at training time.
The paper presents a complex, three part model (ArtNet) to tackle this challenge, as well as unimodal linguistic baselines.
Notably, the evaluation does not include vision-only baselines or pretrained model toplines, making it difficult to assess exactly what ArtNet is learning and where its advantage lies.

Questions:

- There are multiple references to "creation", e.g., "create novel compositions" which makes the method sound like it's performing generation. From my understanding, though, ArtNet is purely discriminative, taking in a keyframe and predicting a noun and verb. However, there is one line in the paper that says "We also learn visual reconstruction via a regression task.", which makes it sound like there's a formulation of ArtNet that maybe takes in a noun and verb and produces a keyframe image (using a GAN, maybe? Or a nearest neighbor lookup?), and so does "create novel compositions". If that's the case, it isn't described, and this image reconstruction task is never mentioned again in the paper or described in any equations.

- Were pretrained ViLBERT/UNITER run alone as a topline? Establishing how much is lost in performance due to lack of pretraining + how this method addresses that with sparse data would make a much stronger argument, I feel. In particular, we would want to see pretrained ViLBERT/UNITER and then pretrained + ArtNET to give a sense of how performance will change as models have seen huge amounts of aligned data.
- [Related] In Table 1 what's the intuition for language pretrained mBERT/UNITER/ViLBERT falling behind from scratch? Why use language pretraining and not vision pretraining (e.g., topline)?


- In Eq (1), why use cosine similarity for visual embeddings but then back off to surface forms for words? Was cosine similarity for word representations computed by UNITER tried? What is the intuition for this not working, if it did not?

- Not sure in Eq (4) what the sequence input to the LSTM is; does c range over some sequence? Doesn't the sum already collapse that?

- "To ensure the focus is on new compositions, rather than new words, we removed new compositions that contain new words not seen in the train set." All new words or just nouns/verbs? It's a big advantage/relaxation on the test set to have no OOV tokens.

- Why no vision-only baseline? Strip word contexts at training time except noun/verb, then predict only noun/verb at test. A lot of this could be basically object recognition followed by activity recognition or strong priors on p(activity | object) (e.g., always "open" or "close" for cabinets).

- "outperforms Multimodal BERT/BERT with 1.23%/4.42% improvements, which is significant"; what statistical significance test was used? How many random initializations were tried to establish the average performance numbers for comparison between performance populations?

Suggestions for Improvement:

- "We call attention to a challenging problem, compositional generalization, in the context of machine language acquisition, which has seldom been studied." This is poorly worded, since compositional generalization is well and commonly studied, to the point that even in this paper there is a section in the related work about it. Major workshops also list compositionality as a topic of interest, so I don't think it's fair at all to say that this has "seldom been studied" [ https://sites.google.com/view/repl4nlp2020/home ]. In the context of language acquisition specifically, emergent communication work focuses heavily on composition [ https://sites.google.com/view/emecom2019/home ].

- ViLBERT in intro, UNITER in description of method, "Multimodal BERT" (mBERT?) in Table 1. What was used? Needs to be consistent in presentation.

- "We discard the object labels due to strict constraints in the language acquisition scenario." Because Faster RCNN is trained on ImageNet, which is based on WordNet, the object categories still exist in the form of supervision. The model has a linguistically-motivated notion of what an "object" is that can be traced to the WordNet. This should be acknowledged; you can't actually "get rid" of linguistic information inherent in a pretrained Faster RCNN.

- "But there are few works addressing this emerging and valuable challenge for language acquisition in a multimodal reasoning view." This paper does not really tackle language acquisition, though? There's a restriction so that the test set has no OOV words, even. I think the claims and presentation of the paper need to be carefully re-scoped.

- There is a lot of focus on "learned arithmetic operations" but no analysis as to what exactly this component ends up doing or learning.

Nits:
- Typo Introduction "a language model that generate" S/V agreement.
- Figure 1 doesn't feel like it communicates anything about the method, and does not seem tied to the caption. Are boxes (1, 2, 3) meant to represent the association, reasoning, and inference steps? What's happening in each?
- "The results show that ARTNET achieves significant performance improvements in terms of new composition accuracy, over a large-scale
video dataset." strange wording makes it sound like ARTNet is outperforming a dataset, not a method.
- "We train the model to acquire words by directly predicting them." this sounds like the model will be predicting words unseen at training time, which is not so. In particular, "acquire" here sounds like the model will be exposed to the word at most once (at inference time) and then be able to memorize that exposure in sequence.
- Typo? In 3.1 "by running faster R-CNN too" what is the "too" pointing to? Do you run Faster-RCNN somewhere?
- Typo 3.3 "stringest baseline" strictest?
- Figures 4 and 5 are so close together their captions bleed together and are really difficult to disentangle.
- Typo "than our baselines 86.5" makes it sound like the baselines achieved 86.5.
- Typo 4 "the goal of learned model" missing "the"
review_title: A straightforward task with a complex architectural approach, unclear contribution scope, and missing baselines.

Meta-review: The authors propose a new dataset and compositional task based on the EPIC Kitchens dataset.  The goal is to test novel compositions and to build a transformer based network specifically for this inference (by analogy). Specifically, the analogy here references the use of nearest neighbors in the dataset.  There are a lot of concerns raised by reviewers which require a large number of changes to the presentation of the manuscript and they are not at present convinced by the current setup or experiments. Explicitly motivating which pretraining methods do or do not violate which aspects of composition and what role other factors like synonymy play in generalization is necessary. Several aspects of the claims made in the paper and in the discussion are big claims that require substantial discussion and analysis (e.g. the surprising weakness of pretrained models) which the reviewers do not feel can be so easily explained away (e.g. by domain shift).
Decision: Reject