Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, authors reformulate zero-shot action recognition such that does not rely on nearest neighbor
classification, but rather consists of a pairwise scoring function. Given a video and a set of action
classes, the method predicts a set of probabilities for each class, allowing for semantically distinct
classes to be predicted with high confidence. The proposed method improves on previous state-
of-the-art zero-shot action detection models and is tested on UCF101, HMDB, AVA datasets among others. 
main_review: The proposed method is tackling the problem of Zero shot action recognition in videos. The core idea is to learn both video representations and semantic text embeddings in and end to end manner with and merge the representations with a cross entropy loss to handle multi label prediction. The authors did a good job describing the literature, writing a clean paper (a bunch of places with grammar issues but not major), doing significant experimental analysis of the method on various small and large scale datasets and also have ablation studies to backup the utility of the proposed method. 

The only limitations if any would be that the proposed method is a simple extension of existing methods and for NeurIPS standards might not have the deep technical contributions. Nevertheless, i believe the work is valuable and is tackling an important problem of zero shot learning in videos.
limitations_and_societal_impact: Limitations - The main limitation of the proposed work is it is a simple extension of backpropogating through two different models instead of freezing one model's output (text embedding) and using it with a cross entropy loss.  The video encoding, text encoding, jointly training the models and the final cross entropy loss are all very well studied modules and simpler extensions of existing works. So, the technical novelty of the proposed approach is a bit limited even though the results are good. The simplicity of the method though could be a positive in-spite of these limitations.

On Societal impact - N/A. Authors didn't touch on it in anyway but the subject area and the models built seem fairly safe from this angle that authors didn't have to think deeply about the societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a strategy for zero-shot action recognition. One of the contribution is to design the model such as it directly outputs a score given an unseen class name. This is different from past work that instead rely on nearest neighbor in a joint text-video space. The authors explore various ways to combine the visual and semantic class name in the architecture. They report results on multiple datasets UCF-101, HMDB, RareAct, AVA and MEVA where they show improvements over previous work trained on the same dataset [4].
main_review: ## Strengths

- Zero-shot action classification is an important problem to tackle
- The paper is clearly written
- Compelling numbers on HMDB51 and UCF101 compared to previous work using the same setting
- The idea of outputting a score directly for a given class allows to explore more complex interactions between the visual signal and the text contained in the class name than using simple nearest neighbors in a vector space. In particular, according to Table 6, it seems that the multiplicative interaction is the important architecture design of the paper.

## Weaknesses and possible improvements

- **More complex interactions**: Right now only concatenation, sum and multiplicative interactions are explored. As it seems to be quite important to improve the performance, have you explored other architecture for interactions? One could for example use attention mechanisms to combine the visual and the text features (e.g. similar to what is used in works such as [a] VilBERT). 

[a] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks, Lu et al.

- **What happens on the regular classification accuracy?** Even if not the main focus of the work, I believe it would be worthwhile to provide the accuracy numbers obtained on Kinetics700 when training with that technique (the one illustrated in Figure 2) compared to standard softmax loss. Can we expect improved numbers on standard action recognition as well thanks to the information contained in word2vec and the proposed interaction architecture? Even if the result is not compelling I believe that would be an interesting datapoint to obtain.

- **Comparing/discussing recent zero-shot work**: I think it would be worth discussing a recent trend in zero-shot recognition which consists in training on large corpora of webly supervised text+image/video datasets (see [b,c,d] below) and discuss how the two methods could potentially be merged together. In particular even if not comparable because the training datasets are very different, [c] obtains 80.3 on UCF101 zero shot even though it was only trained on still images (compared to 40.1 in the paper). Similarly [b] was trained on noisy instructional videos (HowTo100M dataset) and still perform quite well on the RareAct dataset (see next point).

- **RareAct correct evaluation metric**: Please provide the metric of RareAct used in the paper (wAP). The authors provide an implementation of this score in [here](https://github.com/antoine77340/RareAct/blob/master/compute_score.py) and this should be compatible with your model as you only need to provide a matrix score containing the score for each video and for all classes. That way you will be able to compare to [b] and [c] which would give a point of reference in the future.

- [4] further improves its results using SUN augmentation. Have you considered using this as well in your case to verify the improvements still hold?

- Missing related work:
  - On the multiplicative interactions, there are a few work that would be worth mentioning:
    - FiLM: Visual Reasoning with a General Conditioning Layer, Perez et al.
    - Squeeze-and-Excitation Networks, Hu et al.
    - Learnable pooling with context gating for video classification, Miech et al.
  - Some recent work that can perform zero-shot (mentioned above) and that would be worth discussing:
    - [b] End-to-end learning of visual representations from uncurated instructional videos, CVPR2020 (for comparison on RareAct)
    - [c] Learning Transferable Visual Models From Natural Language Supervision, Radford et al. (provide comparison on UCF101 and RareAct).
    - [d] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision, Jia et al (ICML21). 

## Overall assessment

The paper proposes an interesting idea and the results seem to suggest that the proposed changes are effective as it improves over a recent baseline [4]. For this reason I am leaning towards acceptance however there are a few things that could improve the paper further and I will wait on the rebuttal to make my final decision.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work has two main contributions: (1) reformulation of the zero shot action recognition task for the multi-label case, and (2) a new model motivated by the authors idea that nearest neighbor is sub-optimal on a static semantic space. A pairwise scoring function is proposed in this work which first maps both visual and semantic embeddings into a joint space to predict match/no match probabilities of a 3D-CNN based visual representation and a Word2Vec/Sent2Vec based text representation. Their approach is evaluated not only on multi-label action recognition (where the zero shot performance is evaluated for the first time) dataset but also on single-label action recognition dataset, achieving state-of-the-art results. Evaluation protocols for single-label zero-shot action recognition datasets, UCF-101, HMDB, RareAct is proposed and another two evaluation protocols for multi-label zero-shot action recognition based on MEVA and EVA datasets are proposed in this work. 
main_review: Several highlights and shortcomings are found during review.
Highlights:
1)	The authors reformulate zero-shot action recognition for a multi-label setting, which in my opinion is an important task.
2)	Interesting although a relatively simple idea for a new approach:  learning to infer semantic similarities given text label and a visual representation as input. The authors argued that the most leveraged nearest neighbor-based approach in zero-shot recognition is sub-optimal and proposed a scoring function-based approach to achieve one-stage zero-shot multi-label prediction.
3)	For the dataset contributions, one evaluation protocol based on UCF-101 and HMDB leveraged random seed, and another one based on full RareAct dataset for single-label zero-shot action recognition are given. Besides these, two evaluation protocols for multi-label zero-shot recognition based on EVA and MEVA datasets are also given in this work.  
4)	Extensive evaluation results: Ablations for different refining modules of text encoder, such as no refining, single layer refining and full model refining, for text encoder and video encoder feature fusion modalities, such as concatenation, addition and multiplication and for different textual encoders such as Word2Vec, Sent2Vec and BERT are illustrated in their work, indicating clearly which modality setting is better for label text feature extraction and fusion between video embedding and text embedding.
5)	State-of-the-art results are achieved for both: the newly proposed multi-label and the existing single-label zero shot action recognition. 
Shortcomings:
1)	Since the authors argue that for semantic similarity nearest neighbor approach is sub-optimal, for single-label and multi-label zero-shot recognition there is no comparison experiments between nearest neighbor-based approach and score function-based approach proposed by authors leveraging both semantic representation and video representation. 
2)	The parameterized function which maps an input video to the semantic space illustrated in line 109 has the same representation with the scoring function mentioned in line 115, which may cause confusion.
3)	There is a small wrong typing in line 248. 
Question:
1)	There is a question for formular (5) under line 142, does N forward passes be leveraged or only one forward pass for confident score prediction of different classes? Do the authors reinitialize the fc layer for different classes’ confidence score prediction?
After the analysis among highlights, shortcomings and question, this is a good work, which provided interesting multi-label and single-label zero-shot recognition evaluation protocols on the well-known datasets and proposed a score function-based approach to take place with the common leveraged nearest neighbor method for zero-shot action recognition considering the similarity between video embedding and semantic embedding, while there are still several problems found in this work. The efficacy verification of their score function has not been illustrated in experiments by comparing the zero-shot recognition performance between nearest neighbor and score function proposed in their work while using same feature embedding setting. And also the forward pass of the score function is not illustrated clearly as the aforementioned question part. According to the above several analyses, this paper is recommend scored as marginal accept and we would kindly ask for clarification on the posed question.


We have read the other reviews and the author rebuttal and will stay with our initial assessment. 

limitations_and_societal_impact: Since this work focuses on zero-shot recognition it makes a good step towards solving the data efficiency problem of the deep CNNs. The models are evaluated on public datasets and we do not see any ethical concerns beyond the conventional issues linked to visual recognition.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work investigates zero-shot action recognition from a multi-label perspective. Rather than a nearest neighbour action selection, this work opts for a binary cross-entropy optimization. Experiments on common datasets as well as AVA and MEVA show the effectiveness of the approach.
main_review: Strengths:

In zero-shot action recognition, the commonly used datasets are multi-class, with one action relevant per video. This work goes against this common assumption. Zero-shot multi-label recognition has been investigated multiple times before in the image domain, but hardly in the video domain. It is good to see additional effort in this direction.

Moreover, the approach is evaluated on five datasets, both multi-class and multi-label. The results are already good in the multi-class setting, which actually warrants further discussion as the approach is geared towards the multi-label setting. The main components are also studied individually to get a better sense of the most important elements in the overall architecture.

Weaknesses:

The paper is low on novelty. While multi-label zero-shot action recognition has not received a lot of attention, it has been investigated before as indicated by the authors. For the method itself, the notion of binary pairing at the video-level, instead of the class level is interesting, the rest of the approach follows standard binary cross-entropy optimization. The experiments and especially the evaluations on AVA and MEVA are new.

The method itself is not tied to actions. What makes the method interesting for actions? And how does it perform on other multi-label zero-shot problems? More importantly, how does the method compare to multi-label approaches from the image domain on actions? I.e. what happens when the approaches of e.g. [25-27] are used on the same (pre-)training data? Unfortunately, evaluations such as on AVA are lacking these baselines. It is therefore not possible to assess the effectiveness of the proposed approach.

The paper also misses a number of relevant zero-shot action papers for discussion and comparison. The paper focuses mostly on Zhu et al. [2] and Brattoli et al. [4] in the discussion and as baselines, but a number of relevant papers are missed in the process, e.g.:

[a] Mishra, Ashish, et al. "A generative approach to zero-shot and few-shot action recognition." 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.

[b] Jain, Mihir, et al. "Objects2action: Classifying and localizing actions without any video example." Proceedings of the IEEE international conference on computer vision. 2015.

[c] Tian, Yi, et al. "Aligned Dynamic-Preserving Embedding for Zero-Shot Action Recognition." IEEE Transactions on Circuits and Systems for Video Technology 30.6 (2019): 1597-1612.

[d] Mettes, Pascal, William Thong, and Cees GM Snoek. "Object priors for classifying and localizing unseen actions." International Journal of Computer Vision (2021): 1-18.

Especially [b] and [d] are relevant as they do not transfer from seen to unseen actions, but from objects to unseen actions. A score is computed for each action separately. As these works focus on the multi-class setting, the argmax is taken, but it is equally possible to use a threshold as done in this paper. On UCF-101, [d] scores close to Brattoli et al., making it a viable candidate for comparison on AVA.

Conclusion:

This paper shins a brighter light on the multi-label recognition of actions in the zero-shot action, which is indeed needed. An approach that reasons on video-label pairs is proposed to make this possible. Experiments one five datasets show that the method is effective overall. The novelty of the paper is however low, as the task itself is not new, the method is not really new and especially not specific to actions, while the experimental comparisons are lacking. Moreover, relevant related work is not cited and discussed. Please address these issues in the rebuttal.
limitations_and_societal_impact: The authors mention that limitations are discussed, but this can not directly be found back in the paper. A more explicit discussion is beneficial. Societal impact is not listed and discussed explicitly, which could be relevant due to the topic of action recognition.
review_title: null

Meta-review: There was active engagement between the authors and reviewers following the rebuttal. 

It is important that the authors update the paper with the new results requested by the reviewers, as well as update the related work section with (i) missing citations for video ZSL, and (ii) related work on image ZSL, that should also be discussed. 

One other issue that could be more thoroughly discussed is the significance of the type of interaction, where multiplicative gives a subtantial performance improvement over the other choices.
Decision: Accept (Poster)