Reformulating Zero-shot Action Recognition for Multi-label Actions
Authors: Alec Kerrigan
Kevin Duarte
Yogesh S Rawat
Mubarak Shah
Venue: NeurIPS 2021 Poster (2021)
Keywords: zero-shot
action recognition
computer vision

Abstract: The goal of zero-shot action recognition (ZSAR) is to classify action classes which were not previously seen during training. Traditionally, this is achieved by training a network to map, or regress, visual inputs to a semantic space where a nearest neighbor classifier is used to select the closest target class. We argue that this approach is sub-optimal due to the use of nearest neighbor on static semantic space and is ineffective when faced with multi-label videos - where two semantically distinct co-occurring action categories cannot be predicted with high confidence. To overcome these limitations, we propose a ZSAR framework which does not rely on nearest neighbor classification, but rather consists of a pairwise scoring function. Given a video and a set of action classes, our method predicts a set of confidence scores for each class independently. This allows for the prediction of several semantically distinct classes within one video input. Our evaluations show that our method not only achieves strong performance on three single-label action classification datasets (UCF-101, HMDB, and RareAct), but also outperforms previous ZSAR approaches on a challenging multi-label dataset (AVA) and a real-world surprise activity detection dataset (MEVA).

Reformulating Zero-shot Action Recognition for Multi-label Actions
Alec Kerrigan Kevin Duarte Yogesh S Rawat Mubarak Shah
Center for Research in Computer Vision, University of Central Florida, Orlando, FL 32816,
faleckerrigan, kevin hererrraannakiinlaaarreeeeeiiii ucf edu, t Seac) ucf edu
Abstract
The goal of zero-shot action recognition (ZSAR) is to classify action classes which were not previously seen during training. Traditionally, this is achieved by training network map, regress, visual inputs to semantic space where nearest neighbor classifier used to select the closest target class. We argue that this approach is sub-optimal due to the of nearest neighbor on static semantic space and is ineffective when faced with multi-label videos where two semantically distinct co-oceurring action categories cannot be predicted with high confidence. To overcome these limitations, we propose ZSAR framework which does not rely on nearest neighbor classification, but rather consists pairwise scoring function. Given video and set of action classes, our method predicts a set of confidence scores for each class independently. This allows for the prediction of several semantically distinct classes within one video input. Our evaluations show that our method not only achieves strong performance on three single-label action classification datasets (UCF-101, HMDB, and RareAct), but also outperforms previous 7SAR approaches on challenging multi-label dataset (AVA) and a real-world surprise activity detection dataset (MEVA).
Introduction
Current image and video classification models require large labeled training datasets but they perform really well on previously seen classes. However, if novel classes are presented to these models, as is the case in many real-world applications, they tend to fail. The system has to be retrained with additional samples for this class to correctly predict these novel classes, which requires additional training time well as computational resources. Therefore, it would be beneficial if we train single system on a fixed dataset which can be applied to new, previously unseen classes. To this end, zero-shot learning (ZSL) approaches [1] have been proposed which leverage semantic information, e.g. textual descriptions or class names, and relate them to new unseen class categories.
In this work, we focus zero-shot action recognition (ZSAR), where the goal is classify videos of unseen action categories. There has been a great progress focusing on ZSAR 22 however, most of these existing methods utilize similar fundamental approach. These approaches project video representation to a fixed semantic space (c.g. a text embedding space generated from pre-trained Ver (5) model) and perform classification using nearest neighbor operation. We argue that such a solution is sub-optimal because the class selection is performed on static text-based semantic space. This formulation forces classes that are classification boundaries, even they are visually dissimilar; conversely, actions that are visually similar, appear further apart in the semantic space. For example, the action "Tennis Swing" is closer in semantic space to "Swing" (i.e. a child on a swing) than "Table Tennis Shot" even though more visually similar the latter action.
similar in the semantic space have closer
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Traditional ZSAR
Neighbor
Top-5 Predictions
Video Model
Tolking
Aampinng
1] Walking
2] Running
Running
Aumping
Louphing
Wolking
Talking
5 Loughing
Vieual Space
Video
Joint Space
Top-S Predictions
Model
lolking
1] Walking
Walling
2] Talking
Semantic Space
Not Presess
Rwcning
Junspaing
4) Running
humping
5) Laughing
Tolking
Tumping
Loughing
Pairwise Scoring ZSAR
mughting
Wathing
Figure I: C onventional nearest neighbor ZSAR methods (top) cannot predict semantically dissimilar actions "Walking" and "Talking" without actions "Running" and "Jumping" also being predicted. Our proposed pairwise scoring approach (bottom), can predict two dissimilar classes by mapping the video and semantic embeddings joint embedding space which can be linearly separated.
In addition, this is detrimental in the multi-label case where multiple semantically distinct actions can occur within sample. This is illustrated in Figure [ (top). In conventional ZSAR approaches, the video network can only generate single representation in the semantic space so the actions "Walking" and "Talking" can not both be predicted without many incorrect action categories also being predicted. This issue could be solved by fine-tuning the textual encoder which generates the semantic space (i.e. moving the representations of "Walking" and "Talking" closer together). However, previous approaches train the video models by regressing to the action class's ' semantic feature vector. If the vector not fixed (i.e. the text encoder is fine-tuned), this would collapse to a trivial solution: it can map all classes to zero vectors which are easily regressed to by the video model. Several constraints could prevent this collapse, however such solutions to the best of our knowledge, have not been proposed in the ZSAR literature.
We propose novel solution this problem where a model finds agreement between the features from pair of inputs (i.e. the video and a semantic/textual feature vector) and outputs a match vs. no-match probability score. Our method consists of a pairwise scoring function which has two main benefits. First, the model can refine the input text features through additional parameterized layers, without collapsing to a trivial solution. Second, the method outputs individual action probability scores used for classification instead of relying on nearest neighbor. Figure [ behaviour. The Pairwise Scoring 7SAR (PS-7SAR) approach able to map the merged visual and semantic representations into joint space, on which decision boundary can be created to successfully predict both "Walking" and "Talking" without predicting irrelevant actions.
(bottom) depicts this
PS-ZSAR is trained end-to-end using standard classification losses. We evaluate our proposed network on five video datasets. On three single-label action recognition datasets UCF-101, HMDB, and RareAct it achieves strong performance when compared to previous approaches. We report, to the best of our knowledge, the first results for zero-shot multi Mabli-labell action recognition on the AVA dataset, which we believe will be challenging baseline for future ZSAR methods. Lastly, we use PS AS-ZSAR the NIST ActEV challenge "Surprise Activity" task which demonstrates the method's ability to scale to difficult real-world scenarios.
Related Work
There vast literature pertaining the zero-shot action recognition (ZSAR) []. There are bodies of
work on Iransductive 7SAR 6 7 g 9. where test data (i.e. videos) are available during training but classes are not, and generalized 7SAR (GZSAR) [10 performance both seen and unseen classes. However, in this work, concentrate inductive
15] which evaluates method

---

ZSAR in which the test data and classes are unknown during training and the method is evaluated only on unseen classes.
2.1 Zero-shot Action Recognition
Typical zero-shot video classification systems [16 17
18 19 extract visual features from frames or 3D video clips using pre-trained image or video networks, resepetively (e.g. ResNet [20 or C3D 21 models), and then train a separate model that maps the visual features to semantic embedding space. For ZSAR methods, if video networks generalize well to the semantic space, the model can perform well on semantic embeddings for classes previously unseen during training. At test time, these methods predict the class which has semantic embedding that is the nearest neighbor of the model's output. The semantic space usually takes the form text embeddings from a Word2Vec (5) the semantic space consists of attributes manually selected by an expert; however, such methods are difficult to apply to real-world scenarios.
model. In other methods 12] 23 24
There have been variety of methods to learn models which can map video representations to semantic space. Zhu al. 2 treat ZSAR as generalized multi-instance learning problem and learn a kernelized representation that can be directly compared with unseen action prototypes. Action2Vec uses a combination of supervised classification loss and a ranking loss to map video features to textual space. Recently, Brattoli et al. 4 showed that large performance increases can be obtained by learning the video model rather than using pre-extracted video features. We find that this idea can be extended to the text model: by learning improved text representations, i.e. refining the static Word2 Vec embeddings, we achieve improved generalization on unseen classes.
2.2 Multi-label Zero Shot Learning
Multi-label zero shot learning has been explored in limited capacity, primarily for images. Lee 22 utilized a graph structure to learn semantic dependencies between words in class name to learn relevant textual features. Huynh and Elhamifar [26) used an attention framework to map important segments of image to join visual-label embedding space for classification. Huang (27 used similar approach while also proposing transferring learned embedding spaces from systems trained on ImageNet. However, multi-label zero shot learning for action recognition is far less explored space. To the best of our knowledge, Wang and Chen (28] propose the first work to address this problem. Their work focuses learning temporal relationships between frames in a video, then using a ranking loss to learn alongside a semantic encoder. In this work, we simplify the multi-label ZSAR task by proposing a pairwise scoring network that can be trained end-to-end with entsopy loss. Furthermore, we evaluate on two multi-label datasets, AVA and MEVA, and show that our approach can scale to these challenging benchmarks.
3 Method
3.1 Problem Formulation
Zero-shot action recognition aims classify unseen actions, 21, by transferring knowledge learned from seen training classes, S. The training set, Ds = ((x, y) jar E X, E S). consists of labeled videos in video space X' and label y. The test set is defined as Dy {(x, v) jæ E X, y E UJ. The action classes in both datasets must be disjoint, i.e. S Su = 0. The goal of ZSAR is to predict label for a videos of unseen class (activities), i.e. -U.
For each class, a semantic embedding vector & (v) E R. of length is obtained. These vectors traditionally obtained by averaging word2vec (5] embeddings for each word in the class name,
however different text encoders (c.g. sent2vec 29 BERT B0]) could also be used. Conventional ZSAR methods learn a parameterized model fo which maps an input video to the semantic space, i.e. fo X + R.D Then, for a given video, .r, a classification is performed by nearest neighbor in the set of test class embeddings, as defined by:
F (a) argmin d (fo (z). (y)),
(1)
yeu
where d is a distance metric (c.g. cosine distance).
3

---

Cross-Entrope
9x(+)
Backprop
Input
Embedeing
(a] Video Encoder
4
P(y())
Predicted
Play
(d) Merge Modalities
(c) Classifier
Instrument"
Pretrained Encoder
Semantic Emtedding
MIP
Embedding
tafied
Thanuble Layers:
Foad Model:
(b)
Refining
Figure 2: Overview of our proposed network architecture. Our method consists of four main parts: (a) The Video Encoder which extracts video features from given video clip, (b) the Text Encoder which generates the semantic embedding from the action class name or natural language description, (c) Text Refining module which fine-tunes the semantic embeddings during training, and (d) the Scoring Function which generates the probability that the pair of inputs match. The network trained end-to-end using cross-entropy loss.
We argue that this stage approach metric learning on the semantic space and nearest neighbor classification is ineffective. Instead, we reformulate the ZSAR problem to learn a scoring function he which predicts match vs. no-match probability when given a video and semantic embedding pair, i.c. hg X' x Rd + [0, 1]. Once the network is trained. the final action classification can be obtained by selecting the class which achieves the highest probability:
(x) argmax hg (2r, (y)).
yeul
This formulation allows for multi-label action recognition by selecting all classes with probabilities over the threshold t, i.e.
F (z) = {y E he (z, v (v)) > T
(3)
whereas nearest neighbor classification can not predict multiple semantically dissimilar actions to occur within given video.
3.2 Proposed ZSAR Framework
Our proposed network architecture is illustrated in Figure Here, describe the different compo- nents of our architecture.
Video and Text Encoders Given a video clip x E Tx of frames with height and width of Hl and T respectively, we learn 3D-CNN 31, Jo. to extract video feature representation. Go (x) E RR, Also, we obtain the semantic embedding (text encoding), e (y). by passing the action class name through fixed pre-trained text encoder. Unless otherwise stated, Word2 embedding created for each word and the embeddings over words in the action name are averaged to obtain 300 dimensional text feature representation, vé (y) 30000 We also present results with the more recent Sent2Vec Vec text encoder in Section 4.4
Refining Text Embedding Since text encoders are trained on large text corpora, they tend to lack information about visual attributes of actions. For example, "running water" and "person running" are visually distinct, but embedding for the word "running" is the same for both. To remedy this, propose module, 9t. which refines, or fine-tunes, the text encodings to learn vision-specific representations and generates a new text representation with dimension D,: gt (v (y)) e Pi.
Merging Features and Scoring With the extracted visual features and refined textual features, we need to determine whether or not the inputs match. We first merge the features to obtain a single representation for the pair of inputs, m (g. (x), Se (xb (y))) E PR,,, This merging operation can take many forms (concatenation, addition, multiplication, etc.), which are explored in Section 4.4

---

Once merged, the joint feature representation is passed through an MLP, Jets. to obtain classification logits:
flngits (I, v/ (y)) Octs (m (ge (x) ge (+) (v)))).
Then, final probability score is obtained by using a softmax operation across all classes:
exp (Jigits (x,, 1o (y)))
P (vlar) = ho (x, 10 (v))
(5)
Σ
exp (Fogiss (I, 1 (y))))
y'ES
The final probability scores are then be used for either single-label or multi-label classification following equations and respectively.
Training Procedure We train the network an end-to-end manner using cross-entropy loss. When training a single-label dataset, we compute the loss
1 B
==
log (y)(z;))
(6)
on a mini-batch samples B { V11 .... (I yk)) When training on multi-label data, which is the case for experiments the MEVA dataset, the softmax activation is replaced with sigmoid so that each probability is independent and binary-cross entropy loss used.
4 Experimental Evaluation
Implementation Details For our experiments, the video encoder is the PyTorch [32 implemen- tation of the R(2+1)D-18 31 network. This network outputs visual embedding dimension of we average predictions over 25 clips per video at test time. Unless stated otherwise, the text encoder is a pretrained Word2Vec [5] model with max-pooll max-pooling over words, resulting in 300-dimensional text embeddings. We also present results using the Sent2Vec 22 model, which pre-trained on twitter unigrams and results in dinenimnni embedding for given sequence of words. Our Text Refining module consists of learned 3-layer MLP with hidden dimensions of 1024 and ReLU activations, and a final output dimension of D2 600. The output is passed through a sigmoid activation to obtain an embedding in the range [0. 1]. The loss of model is minimized using the Adam optimizer [34] with starting learning rate of le-3 and batch size of 114. The model is trained for 50 epochs with learning rate decrease by a factor of 10 epoch 30. All experiments are performed on two of Nvidia Tesla V100 GPUs.
D. = 512 for each 16-frame video clip. As with previous, work 4 33
Training Dataset We train our models on the Kinetics 700 35 action recognition dataset. Follow- ing [4), we ensure that these experiments remain true zero-shot tasks by filtering out classes from Kinetics that are not sufficiently different than our testing classes. This results in 545,317 training videos consisting of 662 action classeg
4.1 Single-label Action Recognition
Evaluation Datasets We evaluate on the UCF-101 [36,, HMDB (3). and RareAct [38 dataests. UCF101 has 101 action classes across 13320 videos taken from YouTube. HMDB has 51 human focused actions based around sports and everyday activities across 6767 videos sourced from com- mercial areas as well as YouTube. RareAct is video dataset containing unusual actions such as "blend phone", "cut keyboard", and "microwave shoes". It contains 905 videos with 122 different actions created by combining rarely co-occuring action verb pairs.
Evaluation Protocols
To compare with previous 7SAR methods,
evaluation protocols
on UCF-101 and HMDB. The first, proposed in 4. involves testing on all videos from each dataset. The second involves randomly selecting and evaluating on half of the actions from a dataset (50
It should be noted that 4 trains on 664 classes, but due to differences in dataset versions (L.e. changed action names) we have 2 fewer classes for training.
5

---

The Atomic Visual Actions (AVA) dataset 41 annotates 80 atomic visual Table 1: Evaluation on all classes of UCF-101 and HMDB datasets.
UCF-101
HMDB
RareAct
Method
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
URL
34.2
Brattoli et al. 4
37.6
62.5
26.9
49.8
9.4
24.8
PS-7SAR (ours)
40.1
66.3
27.3
55.7
11.5
32.8
Table Evaluation on UCF-101 and HMDB-51 datasets. Accuracies are the average 10 random splits each with 50% of the classes.
UCF101
HMDB
DataAug
18.3
19.7
InfDem 339
17.8
21.3
BiDrrectional
21.4
18.9
FairZSI. 40
23.1
TARN [66
19.0
23.5
Action2Vec
23.5
Brattoli el al.
48.0
32.7
PS-ZSAR (ours)
49.2
33.8
for UCF-101 and 25 for HMDB). This is repeated 10 times and the averaged results are reported Since RareAct is composed of "positive", "negative", and "hard negative" samples, we consider only positive clips; follow the first evaluation protocol and evaluate on all 1765 positive video samples.
Comparison with State-of-thee -frte-rr- Methods For the first evaluation protocol, compare to Zhu et al. 12 and Bratolli et al. [4. The results are presented in Table Our approach outperforms the previous state-of-the-art methods both datasets, with large improvement in Top-5 accuracy (3.8% and 5.9% improvements on UCF-101 and HMDB, respectively). We also observe this performance improvement when compared to other previous ZSAR methods using the second evaluation protocol in Table 2] The RareAct dataset presents a challenging benchmark for zero-shot action classification, because the actions are drastically different than those used during training. Again, we find that our approach outperforms the strongest baseline with a 2.1 1% improvement on Top-1 accuracy and 8% improvement in Top-5 accuracy.
4.2 Multi-label Action Recognition on AVA
Evaluation Dataset
actions 340 15-minute video clips. All people in each video are exhaustively annotated at rate of one frame-per-second, and each person can perform multiple actions at once. We present results on the validation set which contains 64 videos split into 54k one-second clips.
Evaluation Protocol Since action localization (i.e. predicting bounding-box localizations for all actors in a video) out of scope for this work, we evaluate only our the action recognition capabilities the AVA dataset. To this end, we extract video clips centered each actor's ground- truth bounding-box, and pass these through the ZSAR model to obtain class predictions. For fair comparison with (4) we generate multiple predictions using several strategies which are outlined in the Supplemental Material. Ultimately, selecting all classes using a threshold on the distance from predictions the text embedding leads best scores, which are reported in this work. The resulting predictions are then used to compute the f-mAP metric as well as the standard multi-label classification metric Fl-score.
As previous methods were not designed with multi-label action recognition, these systems must be modified allow for comparison; specifically, to compute the mAP metric they must predict
"This evaluation protocol is greatly influenced by the random seed used to generate the splits. We discuss this further in the Supplementary Materials.

---

Table 3: Multi-label action recognition evaluation the AVA Dataset. w2v indicates Word2 is used as the text encoder; s2v indicates Vec is used.
Table 4: Results from the MEVA Se- questered Data Leaderboard. For both met- rics, lower score is better.
Text Enc.
mAP
Fl-score
Pmiss000 TFA
P Pmiveo.o TFA
Brattoli et al. 4
w2v
6.4
10.0
IBM-Purdue CMU-DIVA
0.8898
0.8176
0.8795
0.8044
PS-ZSAR (ours)
w2v
11.4
Purdue
0,8537
0.7617
PS-ZSAR (ours)
s2v
7.0
12.3
UMCMU
0.8724
0.8009
Supervised (L3D)
23.4
PS-ZSAR
0.7752
0.6686
confidence scores for their final classifications. We apply the method in [4] and obtain a confidence score for each class by computing the softmax the inverse cosine distances between the video encoding and the classes' textual encoding. As most resultant distances are relatively close, we scale them by factor of 10 prior to in the softmax. For final class predictions, follow equation which we find leads to the best results for the baseline
Results We present results the AVA dataset in Table 3. We find that, when using the same Verd2Vec text encoder, our model outperforms previous approaches when modified for multi-label action classification in terms of F1-score. This improvement is noticable when using Sent2 Vec as the text encoder- we achieve a 0.6% improvement terms of mAP and a 2.3% improvement in Fl-score.
4.3 Real-world Action Recognition on MEVA
Dataset So far, the previous evaluation datasets have been curated contain specific action classes. Also, unseen classes are identified by class names, rather than natural language sentences, which is not ideal for real-world applications. To demonstrate our approach's 's ability to deal with real-workh scenario, we train and evaluate on the Multiview Extended Video with Activities (MEVA) dataset 42.. This dataset consists 9,300 hours of untrimmed video collected from multiple viewpoints and scenes. For 144 hours of video, bounding box annotations are available for actors performing 37 possible activities. Each actor can perform multiple actions at once and each activity has detailed natural language descriptions, which is beneficial for learning ZSAR models. The data is split into 22 hours for training and 122 hours are sequestered for the NIST Activity in Extended Video (ActEV) challenge. This challenge has a "Surprise Activity" task, which involves classifying previously unseen classes (i.e. classes not within the annotated 37 activities) from natural language descriptions. As the test data is sequestered, the number of unseen action categories well as the number of instances are unknown.
Training and Evaluation Protocol On this dataset, PS-ZSAR is trained using the natural language descriptions of the activities (rather than their names). Since these descriptions consist of full-length Sent2Vec our text encoder. Evaluation is performed by submitting system to the ActEV challenge website. Performance evaluated based on the probability of missed detection fixed time- based false alarm per minute (Pmiss @ @Tx). The time-based false alarm per minute is set to 0.02 and 0.04 (i.e. Pmisseuu FA and P misss TrA). We refer to MEVA SDL 43 for detailed explanations of the evaluations well evaluation code.
sentences,
Results Our method achieves strong performance when compared with other submitted systems, as seen in Table We find that our approach greatly outperform all other systems the P @Trx metric. These experiments highlight our method's ability to scale to large amounts real-world video data well its ability to leverage natural language activity descriptions for the ZSAR task.
We include various methods for obtaining confidence scores and final class predictions in the Supplementary Materials
7

---

Table 5: Ablation on Text Refining Module.
Table 6: Ablation on how modalities are merged.
Dataset
UCF-101
HMDB
Dataset
UCF-101
HMDB
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
No Refining
32.0
58.5
18.9
45.6
Concat.
34.4
60.6
21.3
47.7
Single layer
36.3
64.2
23.7
50.1
Add.
35.1
62.6
24.0
51.6
Full Model
40.1
66.3
27.3
55.7
Mult.
40.1
66.3
27.3
55.7
Table 7: Ablation on different textual encoders.
Dataset
UCF-101
HMDB
RareAct
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Werd2
40.1
66.3
27.3
55.7
11.5
32.8
Vent2
39.7
65.4
23.9
51.2
7.3
22.7
BERT
31.8
58.9
22.2
45.6
8.3
22.9
4.4 Ablations
We perform several ablations on the UCF-101 and HMDB datasets to evaluate various components of our network.
Text Refining Module We evaluate the importance fine-tuning the textual features during training. We first remove the text refining module, and merge the semantic embedding, vhyy). with the visual features. This results in Top-1 accuracy of 32.0% and 18.9% UCF-101 and HMDB, respectively. Next, we use a single dense layer of size 1024 before merging features;his improves performance full 3-layer refining module which
by 4.3% UCF-101 and 5.7% on HMDB. Finally, outputs new textual encoding. We find that this leads to the best results with accuracies of 40. % on CCF-101 and 27.3% on HMDB. This suggests that single linear transformation (i.e. single dense layer) is insufficient to learn the visual semantics for given action classes, and a deeper refining module is required.
Modality Merging To determine the most effective technique for combining video and textual features, we use three common methods: multiplication, addition, and concatenation. For these experiments, all other components of the network remain unchanged while varying the merging method. We find that addition and concatenation perform similarly, but multiplication substantially outperforms them with 5% improvement on UCF-101 and 3.3% improvement on HMDB.
Importance of the Textual Encoder Traditionally, ZSAR methods trained using Word2Vec encoder with max-pooling over the words of an action name. Pooling, however, can lead to loss of vital information. Newer text encoding methods like Sent2Vec BERT (30] do not pool feature representation, but rather generate a single feature vector when given a sequence of words. We report scores on the UCF-101, HMDB, and RareAct datasets for various text encoders in Table 7 We find that for these datasets, Word2 Vec tends to outperform these sentence based encoders: this can be attributed the fact that the class names used in these datasets consist of few words (the majority of classes are fewer than 2 words) so the max max-liooiigg operation used with Word2Ve Vec has minimal negative effect. This is supported by our experiments on the MEVA dataset, where Sent2 Vec outperforms Word2 Vec+Poling since action classes are described with natural language descriptions.
4.5 Analysis
Improved Textual Embeddings We first analyse the text representations generated by our Text Refining module. We present t-SNE 455 visualization the Word2Vec encodings and the refined encodings in Figure 3 We find that the representations are changed based on the visual semantics of the given action class. For example, the original Word2Vec embedding for the action "Tennis Swing" is closer to the action "Swing" (i.e. child on a swing) than the action "Table Tennis Shot" which is visually more similar. Our Text Refining module, however, learns to bring "Tennis Swing" and "Table Tennis Shot" closer together in the feature space.
8

---

WordZVec Embeddings
PS-ZSAR Refined Embeddings
TableTennisShee
Table TableTennisshot
TannisSwing Swing
TannisSwing
50
Swing
20
-25
-40
-60
-75
-60 -20
-40 -20
Figure 3: comparison of the texual embeddings of UCF-101 classes of the previous method and PS-ZAR, reduced to 2-dimensions via t-SNE. "Swing" and "Tennis Swing" both contain the word "Swing", and therefore averaging the more visually similar "Table Tennis Shot" class is further away. In PS-ZAR, the more visually similar classes are grouped closer together in the embedding space.
Word2Vec embeddings falsely associates the two, while
Ground-rruhh
Seettall at et
PS-ZZAR
Ground-truth
Brnttll
P52SAR
10:00
Camy anioieed
21 Stand
Carry diigedt
Touch en ubieet
Uoten apesso
Wilte
Litter to person
Walls
Bend at the weist
aeewn
patron
Giound truth
Bratwii ot
PS 26AR
Ground-auth
Seettel et
PS-24A8
1 Bmoke
Talk
Liten pemor)
2) Uiten person]
2] parson)
to person
2)
Talc
fall Down
3| Lirien
Caryy object
Open
Camy Object
Figure 4: Qualitative results on multi-label samples from the AVA dataset for PS-ZSAR and the state-of-the-art baseline [] Our proposed approach is able predict semantically distinct classes, which leads improved multi-labe accuracy.
Predicting Multiple Actions We argue that previous nearest-neighbor based ZSAR approaches are ineffective when dealing with multi-label data, since semantically distinct actions cannot be predicted together. We show some qualitative results this effect Figure for given video we compare the top-5 predictions for the method [4] and our approach. We observe that our method successfully predicts semantically dissimilar classes while the previous approach can only predict classes which are similar.
5 Conclusion
In this work, we reformulate zero-shot action recognition such that it does not rely on nearest neighbor classification, but rather consists of pairwise scoring function. Given video and set of action classes, our method predicts a set of probabilities for each class, allowing for semantically distinct classes to be predicted with high confidence. The proposed method improves previous state-of-the-ar zero-shot action detection models. We explore the use of feature merging as opposed simply learning on a static textual embedding space, as well as multiple textual encoding schemes that preserve semantic relationships. Our results highlight the need for zero-shot learning models learn both visual and textual spaces to properly represent unseen classes. The PS-ZSAR method achieves state-of-the-arr performance on 5 zero-shot action classification benchmarks including in the challenging multi-label dataset AVA. Finally. we demonstrate our approach's ability deal with real-world surprise activity detection by providing results on the MEVA dataset.
Acknowledgments This research is based upon work supported by the Office of the Director of National IntelligenceeODNN)) Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. D17PC00345. The views conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Govermment is authorized to reproduce and distribute reprints for Govermmental purposes not withstanding any copyright annotation thereon.

---

References
[1] Valter Estevam, Helio Pedrini, and David Menotti. Zero-shot action recognition in videos: survey. Neurocomputing. 439:159-175, 2021.
[2] Yi Zhu, Yang Long, Yu Guan, Shawn Newsam, and Ling Shao. Towards universal representation for unseen action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9436 9445, 2018.
[3] Mccra Hahn, Andrew Silva, and James M Rehg. Action2vec: A crossmodal embedding approach to action learning. arXiv preprint arXiv: 1901.00484 2019.
[4] Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero- shot video classification: End-to-end training for realistic applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4623, 2020.
[5] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representations in vector space, arXiv preprint arXiv:1301.3780 2013.
[6] loannis Alexiou. Tao Xiang, and Shaogang Gong, Exploring synonyms as context in zero-shot action recognition. In 2016 IEEE International Conference on Image Processing (ICIP), pages 41 4190-4194, IEEF, 2016.
[7] Xun Xu, Timothy M Hospedales, and Shaogang Gong, Multi-task zero-shot action recognition with prioritised data augmentation. In European Conference on Computer Vision, pages 343-359. Springer, 2016.
[8] Xun Xu, Timothy Hospedales, and Shaogang Gong. Transductive zero-shot action recognition by word- vector embedding. International Journal of Computer Vision, 123(3):309-3))) 2017.
19] Qian Wang and Ke Chen. Zero-shot visual recognition via bidirectional latent embedding. International Journal of Computer Vision, 124(3):356-388 383, 2017.
[10] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 4582-4591, 2017.
[11] Devraj Mandal, Sanath Narayan, Sai Kumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shahbaz Khan, and Ling Shao. Dut-of-distribution detection for generalized zero-shot action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9985 99993,99,,,4 2019.
[12] Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. A boundary based oft-tt-disttiuution classifier for generalized zero-shot learning. In European Conference on Computer Vision, pages 572-588. Springer, 2020.
[13] William Thong and Cees GM Snoek. Bias-awareness for zero-shot learning the seen and unseen. arXiv preprint arXiv:2008. // 185, 2020.
[14] Jiamin Wu, Tianzhu Zhang, Zueng-Jun Zha, Jicho Luo, Yongdong Zhang, and Feng Wu. Self-supervised domain-aware generative network for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12767-12776. 2020.
[15] Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang. Zheng-Jue Jun Zha, and Yongdong Zhang. Domain- aware visual bias eliminating for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22664-12677 2020.
[16] Mina Bishay, Georgios Zoumpourlis, and Ioannis Patras. Tarn: Temporal attentive relation network for few-shot and zero-shot action recognition. arXiv preprint 1x7iv0190..00021, 2019.
[17] Chuang Gan, Ming Lin, Yi Yang. Gerard Melo, and Alexander G Hauptmann. Concepts not alone: Exploring pairwise relationships for zero-shot video activity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[18] AJ Piergiovanni and Michael Ryoo. Learning shared multimodal embeddings with unpaired data, CoRR, 2018.
[19] Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In Proceedings of the European Conference Computer Vision (ECCV), pages 374-390, 2018.
10

---

[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778. 2016.
[21] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489-4497, 2015.
[22] Qiang Qiu, Zhuolin Jiang, and Rama Chellappa. Sparse dictionary- representation and recognition of action attributes. In 2011 International Conference on Computer Vision, pages 707-714. IEEE, 2011.
[23] Jingen Liu, Benjamin Kuipers, and Silvio Savarese. Recognizing human actions by attributes. In CVPR 2011, pages 3337-3344, IEEE, 2011.
[24] Chuang Gan, Ming Lin, Yi Yang. Yucting Zhuang, and Alexander G Hauptmann. Exploring semantic inter-class relationships (sir) for zero-shot action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[25] Chung- Chung-Weil Lee, Wei Fang. Chih-Kuan Kuan Yeh, and Yu-Chiang Frank Wang. Multi-label zero-shot learning with structured knowledge graphs. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1576-1585, 2018.
[26] Dat Huynh and Ehsan Elhamifar. A shared multi-attention framework for multi-label zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8776-8786, 2020.
[27] He Huang, Yuanwei Chen, Wei Tang, Wenhao Zheng, Qing-Guo Chen, Yao Hu, and Philip Yu. Multi-label zero-shot classification by learning transfer from external knowledge. arXiv preprint wxXv:2007..16100 2020.
[28] Qian Wang and Ke Chen. Multi-lahel zero-shot human action recognition joint latent ranking emhedding. Neural Networks, 122:1-23, 2020.
[29] Mahdi Naser Moghadasi and Zhuang. Sent2vec: new sentence embedding representation with sentimental semantic. In 2020 IEEE International Conference on Big Data (Big Data), pages 4672-4680. IEEE, 2020.
[30] Nils Reimers and Tryna Gurevych. Sentence-hert: Sentence embeddings using siamese hert-networks. arXiv preprint arXiv: /908. 10084, 2019.
[31] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-644 6459, 2018.
[32] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vetioo, Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in pytorch. 2017.
[33] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang. and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36. Springer, 2016.
[34] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[35] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zässerman. A short note on the kinetics-700 human action dataset. arXiv preprint 19Xiv:1977006987, 2019.
[36] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint 1rXiv:1212.0402, 2012.
[37] Hildegard Kuchne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video datahase for human motion recognition. 20/ International conference on computer vision, pages 2556-2563. IEFF, 2011.
[38] Antoine Miech, Jean- Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Rareact: A video dataset of unusual interactions. arXiv preprint arXiv:2008.01011 2020.
[39] Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks, In AAAI, volume 1, page 3, 2008.

---

[40] Alina Roitberg, Manuel Martinez, Monica Haurilet. and Rainer Stiefelhagen. Towards fair evaluation of zero-shot action recognition using externa data. In Proceedings of the European Conferencee Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[41] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick. Caroline Pantofaru, Yeqing Li. Sudheendra Vijayanarasimhan. George Toderici. Susanna Ricco, Rahul Sukthankar, al. Ava: video dataset of opatio-eemporally localized atomic visual actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6047-6056, 2018.
[42] Kellie Corona, Katie Osterdahl, Roderic Collins, and Anthony Hoogs. Meva: A large-scale multiview, multimodal video dataset for activity detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1060 1068, January 2021.
[43] Kitware inc, the multiview extended vidco with activitics (meva) dataset.
[44] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 244-254 2019.
[45] Laurens Van der Maaten and Geoffrey Hinton, Visualizing data using t-sne, Journal of machine learning research, 9(11), 2008.
12


Reviews:
[{"review_id": "cSGZixOWxP8", "reviewer": "Reviewer_CrDZ", "summary": "This work has two main contributions: (1) reformulation of the zero shot action recognition task for the multi-label case, and (2) a new model motivated by the authors idea that nearest neighbor is sub-optimal on a static semantic space. A pairwise scoring function is proposed in this work which first maps both visual and semantic embeddings into a joint space to predict match/no match probabilities of a 3D-CNN based visual representation and a Word2Vec/Sent2Vec based text representation. Their approach is evaluated not only on multi-label action recognition (where the zero shot performance is evaluated for the first time) dataset but also on single-label action recognition dataset, achieving state-of-the-art results. Evaluation protocols for single-label zero-shot action recognition datasets, UCF-101, HMDB, RareAct is proposed and another two evaluation protocols for multi-label zero-shot action recognition based on MEVA and EVA datasets are proposed in this work. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Several highlights and shortcomings are found during review.\nHighlights:\n1)\tThe authors reformulate zero-shot action recognition for a multi-label setting, which in my opinion is an important task.\n2)\tInteresting although a relatively simple idea for a new approach:  learning to infer semantic similarities given text label and a visual representation as input. The authors argued that the most leveraged nearest neighbor-based approach in zero-shot recognition is sub-optimal and proposed a scoring function-based approach to achieve one-stage zero-shot multi-label prediction.\n3)\tFor the dataset contributions, one evaluation protocol based on UCF-101 and HMDB leveraged random seed, and another one based on full RareAct dataset for single-label zero-shot action recognition are given. Besides these, two evaluation protocols for multi-label zero-shot recognition based on EVA and MEVA datasets are also given in this work.  \n4)\tExtensive evaluation results: Ablations for different refining modules of text encoder, such as no refining, single layer refining and full model refining, for text encoder and video encoder feature fusion modalities, such as concatenation, addition and multiplication and for different textual encoders such as Word2Vec, Sent2Vec and BERT are illustrated in their work, indicating clearly which modality setting is better for label text feature extraction and fusion between video embedding and text embedding.\n5)\tState-of-the-art results are achieved for both: the newly proposed multi-label and the existing single-label zero shot action recognition. \nShortcomings:\n1)\tSince the authors argue that for semantic similarity nearest neighbor approach is sub-optimal, for single-label and multi-label zero-shot recognition there is no comparison experiments between nearest neighbor-based approach and score function-based approach proposed by authors leveraging both semantic representation and video representation. \n2)\tThe parameterized function which maps an input video to the semantic space illustrated in line 109 has the same representation with the scoring function mentioned in line 115, which may cause confusion.\n3)\tThere is a small wrong typing in line 248. \nQuestion:\n1)\tThere is a question for formular (5) under line 142, does N forward passes be leveraged or only one forward pass for confident score prediction of different classes? Do the authors reinitialize the fc layer for different classes’ confidence score prediction?\nAfter the analysis among highlights, shortcomings and question, this is a good work, which provided interesting multi-label and single-label zero-shot recognition evaluation protocols on the well-known datasets and proposed a score function-based approach to take place with the common leveraged nearest neighbor method for zero-shot action recognition considering the similarity between video embedding and semantic embedding, while there are still several problems found in this work. The efficacy verification of their score function has not been illustrated in experiments by comparing the zero-shot recognition performance between nearest neighbor and score function proposed in their work while using same feature embedding setting. And also the forward pass of the score function is not illustrated clearly as the aforementioned question part. According to the above several analyses, this paper is recommend scored as marginal accept and we would kindly ask for clarification on the posed question.\n\n\nWe have read the other reviews and the author rebuttal and will stay with our initial assessment. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Since this work focuses on zero-shot recognition it makes a good step towards solving the data efficiency problem of the deep CNNs. The models are evaluated on public datasets and we do not see any ethical concerns beyond the conventional issues linked to visual recognition. ", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "ca five hours"}, "extra_text": {}}, {"review_id": "P9qZ_3lEg6f", "reviewer": "Reviewer_NxRY", "summary": "This work investigates zero-shot action recognition from a multi-label perspective. Rather than a nearest neighbour action selection, this work opts for a binary cross-entropy optimization. Experiments on common datasets as well as AVA and MEVA show the effectiveness of the approach.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nIn zero-shot action recognition, the commonly used datasets are multi-class, with one action relevant per video. This work goes against this common assumption. Zero-shot multi-label recognition has been investigated multiple times before in the image domain, but hardly in the video domain. It is good to see additional effort in this direction.\n\nMoreover, the approach is evaluated on five datasets, both multi-class and multi-label. The results are already good in the multi-class setting, which actually warrants further discussion as the approach is geared towards the multi-label setting. The main components are also studied individually to get a better sense of the most important elements in the overall architecture.\n\nWeaknesses:\n\nThe paper is low on novelty. While multi-label zero-shot action recognition has not received a lot of attention, it has been investigated before as indicated by the authors. For the method itself, the notion of binary pairing at the video-level, instead of the class level is interesting, the rest of the approach follows standard binary cross-entropy optimization. The experiments and especially the evaluations on AVA and MEVA are new.\n\nThe method itself is not tied to actions. What makes the method interesting for actions? And how does it perform on other multi-label zero-shot problems? More importantly, how does the method compare to multi-label approaches from the image domain on actions? I.e. what happens when the approaches of e.g. [25-27] are used on the same (pre-)training data? Unfortunately, evaluations such as on AVA are lacking these baselines. It is therefore not possible to assess the effectiveness of the proposed approach.\n\nThe paper also misses a number of relevant zero-shot action papers for discussion and comparison. The paper focuses mostly on Zhu et al. [2] and Brattoli et al. [4] in the discussion and as baselines, but a number of relevant papers are missed in the process, e.g.:\n\n[a] Mishra, Ashish, et al. \"A generative approach to zero-shot and few-shot action recognition.\" 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.\n\n[b] Jain, Mihir, et al. \"Objects2action: Classifying and localizing actions without any video example.\" Proceedings of the IEEE international conference on computer vision. 2015.\n\n[c] Tian, Yi, et al. \"Aligned Dynamic-Preserving Embedding for Zero-Shot Action Recognition.\" IEEE Transactions on Circuits and Systems for Video Technology 30.6 (2019): 1597-1612.\n\n[d] Mettes, Pascal, William Thong, and Cees GM Snoek. \"Object priors for classifying and localizing unseen actions.\" International Journal of Computer Vision (2021): 1-18.\n\nEspecially [b] and [d] are relevant as they do not transfer from seen to unseen actions, but from objects to unseen actions. A score is computed for each action separately. As these works focus on the multi-class setting, the argmax is taken, but it is equally possible to use a threshold as done in this paper. On UCF-101, [d] scores close to Brattoli et al., making it a viable candidate for comparison on AVA.\n\nConclusion:\n\nThis paper shins a brighter light on the multi-label recognition of actions in the zero-shot action, which is indeed needed. An approach that reasons on video-label pairs is proposed to make this possible. Experiments one five datasets show that the method is effective overall. The novelty of the paper is however low, as the task itself is not new, the method is not really new and especially not specific to actions, while the experimental comparisons are lacking. Moreover, relevant related work is not cited and discussed. Please address these issues in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention that limitations are discussed, but this can not directly be found back in the paper. A more explicit discussion is beneficial. Societal impact is not listed and discussed explicitly, which could be relevant due to the topic of action recognition.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DQm0bCuoeJt", "reviewer": "Reviewer_3jdc", "summary": "In this work, authors reformulate zero-shot action recognition such that does not rely on nearest neighbor\nclassification, but rather consists of a pairwise scoring function. Given a video and a set of action\nclasses, the method predicts a set of probabilities for each class, allowing for semantically distinct\nclasses to be predicted with high confidence. The proposed method improves on previous state-\nof-the-art zero-shot action detection models and is tested on UCF101, HMDB, AVA datasets among others. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is tackling the problem of Zero shot action recognition in videos. The core idea is to learn both video representations and semantic text embeddings in and end to end manner with and merge the representations with a cross entropy loss to handle multi label prediction. The authors did a good job describing the literature, writing a clean paper (a bunch of places with grammar issues but not major), doing significant experimental analysis of the method on various small and large scale datasets and also have ablation studies to backup the utility of the proposed method. \n\nThe only limitations if any would be that the proposed method is a simple extension of existing methods and for NeurIPS standards might not have the deep technical contributions. Nevertheless, i believe the work is valuable and is tackling an important problem of zero shot learning in videos.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations - The main limitation of the proposed work is it is a simple extension of backpropogating through two different models instead of freezing one model's output (text embedding) and using it with a cross entropy loss.  The video encoding, text encoding, jointly training the models and the final cross entropy loss are all very well studied modules and simpler extensions of existing works. So, the technical novelty of the proposed approach is a bit limited even though the results are good. The simplicity of the method though could be a positive in-spite of these limitations.\n\nOn Societal impact - N/A. Authors didn't touch on it in anyway but the subject area and the models built seem fairly safe from this angle that authors didn't have to think deeply about the societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "None, if any they would be about making sure word embedding models are trained properly without any biases.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "7cvhwN3vw3R", "reviewer": "Reviewer_KU7P", "summary": "This paper proposes a strategy for zero-shot action recognition. One of the contribution is to design the model such as it directly outputs a score given an unseen class name. This is different from past work that instead rely on nearest neighbor in a joint text-video space. The authors explore various ways to combine the visual and semantic class name in the architecture. They report results on multiple datasets UCF-101, HMDB, RareAct, AVA and MEVA where they show improvements over previous work trained on the same dataset [4].", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Zero-shot action classification is an important problem to tackle\n- The paper is clearly written\n- Compelling numbers on HMDB51 and UCF101 compared to previous work using the same setting\n- The idea of outputting a score directly for a given class allows to explore more complex interactions between the visual signal and the text contained in the class name than using simple nearest neighbors in a vector space. In particular, according to Table 6, it seems that the multiplicative interaction is the important architecture design of the paper.\n\n## Weaknesses and possible improvements\n\n- **More complex interactions**: Right now only concatenation, sum and multiplicative interactions are explored. As it seems to be quite important to improve the performance, have you explored other architecture for interactions? One could for example use attention mechanisms to combine the visual and the text features (e.g. similar to what is used in works such as [a] VilBERT). \n\n[a] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks, Lu et al.\n\n- **What happens on the regular classification accuracy?** Even if not the main focus of the work, I believe it would be worthwhile to provide the accuracy numbers obtained on Kinetics700 when training with that technique (the one illustrated in Figure 2) compared to standard softmax loss. Can we expect improved numbers on standard action recognition as well thanks to the information contained in word2vec and the proposed interaction architecture? Even if the result is not compelling I believe that would be an interesting datapoint to obtain.\n\n- **Comparing/discussing recent zero-shot work**: I think it would be worth discussing a recent trend in zero-shot recognition which consists in training on large corpora of webly supervised text+image/video datasets (see [b,c,d] below) and discuss how the two methods could potentially be merged together. In particular even if not comparable because the training datasets are very different, [c] obtains 80.3 on UCF101 zero shot even though it was only trained on still images (compared to 40.1 in the paper). Similarly [b] was trained on noisy instructional videos (HowTo100M dataset) and still perform quite well on the RareAct dataset (see next point).\n\n- **RareAct correct evaluation metric**: Please provide the metric of RareAct used in the paper (wAP). The authors provide an implementation of this score in [here](https://github.com/antoine77340/RareAct/blob/master/compute_score.py) and this should be compatible with your model as you only need to provide a matrix score containing the score for each video and for all classes. That way you will be able to compare to [b] and [c] which would give a point of reference in the future.\n\n- [4] further improves its results using SUN augmentation. Have you considered using this as well in your case to verify the improvements still hold?\n\n- Missing related work:\n  - On the multiplicative interactions, there are a few work that would be worth mentioning:\n    - FiLM: Visual Reasoning with a General Conditioning Layer, Perez et al.\n    - Squeeze-and-Excitation Networks, Hu et al.\n    - Learnable pooling with context gating for video classification, Miech et al.\n  - Some recent work that can perform zero-shot (mentioned above) and that would be worth discussing:\n    - [b] End-to-end learning of visual representations from uncurated instructional videos, CVPR2020 (for comparison on RareAct)\n    - [c] Learning Transferable Visual Models From Natural Language Supervision, Radford et al. (provide comparison on UCF101 and RareAct).\n    - [d] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision, Jia et al (ICML21). \n\n## Overall assessment\n\nThe paper proposes an interesting idea and the results seem to suggest that the proposed changes are effective as it improves over a recent baseline [4]. For this reason I am leaning towards acceptance however there are a few things that could improve the paper further and I will wait on the rebuttal to make my final decision.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was active engagement between the authors and reviewers following the rebuttal. 

It is important that the authors update the paper with the new results requested by the reviewers, as well as update the related work section with (i) missing citations for video ZSL, and (ii) related work on image ZSL, that should also be discussed. 

One other issue that could be more thoroughly discussed is the significance of the type of interaction, where multiplicative gives a subtantial performance improvement over the other choices.
Decision: Accept (Poster)