Reformulating Zero-shot Action Recognition for Multi-label Actions

The goal of zero-shot action recognition ( ZSAR ) is to classify action classes which were not previously seen during training . Traditionally , this is achieved by training a network to map , or regress , visual inputs to a semantic space where a nearest neighbor classifier is used to select the closest target class . We argue that this approach is sub-optimal due to the use of nearest neighbor on static semantic space and is ineffective when faced with multi-label videos - where two semantically distinct co-occurring action categories can not be predicted with high confidence . To overcome these limitations , we propose a ZSAR framework which does not rely on nearest neighbor classification , but rather consists of a pairwise scoring function . Given a video and a set of action classes , our method predicts a set of confidence scores for each class independently . This allows for the prediction of several semantically distinct classes within one video input . Our evaluations show that our method not only achieves strong performance on three single-label action classification datasets ( UCF-101 , HMDB , and RareAct ) , but also outperforms previous ZSAR approaches on a challenging multi-label dataset ( AVA ) and a real-world surprise activity detection dataset ( MEVA ) . 1 Introduction . Current image and video classification models require large labeled training datasets but they perform really well on previously seen classes . However , if novel classes are presented to these models , as is the case in many real-world applications , they tend to fail . The system has to be retrained with additional samples for this class to correctly predict these novel classes , which requires additional training time as well as computational resources . Therefore , it would be beneficial if we can train a single system on a fixed dataset which can be applied to new , previously unseen classes . To this end , zero-shot learning ( ZSL ) approaches [ 1 ] have been proposed which leverage semantic information , e.g . textual descriptions or class names , and relate them to new unseen class categories . In this work , we focus on zero-shot action recognition ( ZSAR ) , where the goal is to classify videos of unseen action categories . There has been a great progress focusing on ZSAR [ 2 , 3 , 4 ] , however , most of these existing methods utilize a similar fundamental approach . These approaches project a video representation to a fixed semantic space ( e.g . a text embedding space generated from a pre-trained Word2Vec [ 5 ] model ) and perform classification using a nearest neighbor operation . We argue that such a solution is sub-optimal because the class selection is performed on a static text-based semantic space . This formulation forces classes that are more similar in the semantic space to have closer classification boundaries , even if they are visually dissimilar ; conversely , actions that are visually similar , can appear further apart in the semantic space . For example , the action `` Tennis Swing '' is closer in semantic space to `` Swing '' ( i.e . a child on a swing ) than `` Table Tennis Shot '' even though it is more visually similar to the latter action . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In addition , this is detrimental in the multi-label case where multiple semantically distinct actions can occur within a sample . This is illustrated in Figure 1 ( top ) . In conventional ZSAR approaches , the video network can only generate a single representation in the semantic space so the actions `` Walking '' and `` Talking '' can not both be predicted without many incorrect action categories also being predicted . This issue could be solved by fine-tuning the textual encoder which generates the semantic space ( i.e . moving the representations of `` Walking '' and `` Talking '' closer together ) . However , previous approaches train the video models by regressing to the action class ’ s semantic feature vector . If the vector is not fixed ( i.e . the text encoder is fine-tuned ) , this would collapse to a trivial solution : it can map all classes to zero vectors which are easily regressed to by the video model . Several constraints could prevent this collapse , however such solutions to the best of our knowledge , have not been proposed in the ZSAR literature . We propose a novel solution to this problem where a model finds agreement between the features from a pair of inputs ( i.e . the video and a semantic/textual feature vector ) and outputs a match vs. no-match probability score . Our method consists of a pairwise scoring function which has two main benefits . First , the model can refine the input text features through additional parameterized layers , without collapsing to a trivial solution . Second , the method outputs individual action probability scores used for classification instead of relying on nearest neighbor . Figure 1 ( bottom ) depicts this behaviour . The Pairwise Scoring ZSAR ( PS-ZSAR ) approach is able to map the merged visual and semantic representations into a joint space , on which a decision boundary can be created to successfully predict both `` Walking '' and `` Talking '' without predicting irrelevant actions . PS-ZSAR is trained end-to-end using standard classification losses . We evaluate our proposed network on five video datasets . On three single-label action recognition datasets - UCF-101 , HMDB , and RareAct - it achieves strong performance when compared to previous approaches . We report , to the best of our knowledge , the first results for zero-shot multi-label action recognition on the AVA dataset , which we believe will be a challenging baseline for future ZSAR methods . Lastly , we use PS-ZSAR in the NIST ActEV challenge `` Surprise Activity '' task which demonstrates the method ’ s ability to scale to difficult real-world scenarios . 2 Related Work . There is a vast literature pertaining the zero-shot action recognition ( ZSAR ) [ 1 ] . There are bodies of work on transductive ZSAR [ 6 , 7 , 8 , 9 ] , where test data ( i.e . videos ) are available during training but classes are not , and generalized ZSAR ( GZSAR ) [ 10 , 11 , 12 , 13 , 14 , 15 ] which evaluates method performance on both seen and unseen classes . However , in this work , we concentrate on inductive ZSAR in which the test data and classes are unknown during training and the method is evaluated only on unseen classes . 2.1 Zero-shot Action Recognition . Typical zero-shot video classification systems [ 16 , 17 , 3 , 18 , 19 ] extract visual features from frames or 3D video clips using pre-trained image or video networks , resepctively ( e.g . ResNet [ 20 ] or C3D [ 21 ] models ) , and then train a separate model that maps the visual features to a semantic embedding space . For ZSAR methods , if video networks generalize well to the semantic space , the model can perform well on semantic embeddings for classes previously unseen during training . At test time , these methods predict the class which has semantic embedding that is the nearest neighbor of the model ’ s output . The semantic space usually takes the form of text embeddings from a Word2Vec [ 5 ] model . In other methods [ 22 , 23 , 24 ] , the semantic space consists of attributes manually selected by an expert ; however , such methods are difficult to apply to real-world scenarios . There have been a variety of methods to learn models which can map video representations to semantic space . Zhu et al . [ 2 ] treat ZSAR as a generalized multi-instance learning problem and learn a kernelized representation that can be directly compared with unseen action prototypes . Action2Vec [ 3 ] uses a combination of supervised classification loss and a ranking loss to map video features to textual space . Recently , Brattoli et al . [ 4 ] showed that large performance increases can be obtained by learning the video model rather than using pre-extracted video features . We find that this idea can be extended to the text model : by learning improved text representations , i.e . refining the static Word2Vec embeddings , we achieve improved generalization on unseen classes . 2.2 Multi-label Zero Shot Learning . Multi-label zero shot learning has been explored in limited capacity , primarily for images . Lee [ 25 ] utilized a graph structure to learn semantic dependencies between words in a class name to learn relevant textual features . Huynh and Elhamifar [ 26 ] used an attention framework to map important segments of an image to a join visual-label embedding space for classification . Huang [ 27 ] used a similar approach while also proposing transferring learned embedding spaces from systems trained on ImageNet . However , multi-label zero shot learning for action recognition is a far less explored space . To the best of our knowledge , Wang and Chen [ 28 ] propose the first work to address this problem . Their work focuses on learning temporal relationships between frames in a video , then using a ranking loss to learn alongside a semantic encoder . In this work , we simplify the multi-label ZSAR task by proposing a pairwise scoring network that can be trained end-to-end with cross-entropy loss . Furthermore , we evaluate on two multi-label datasets , AVA and MEVA , and show that our approach can scale to these challenging benchmarks . 3 Method . 3.1 Problem Formulation . Zero-shot action recognition aims to classify unseen actions , U , by transferring knowledge learned from seen training classes , S. The training set , DS = { ( x , y ) |x ∈ X , y ∈ S } , consists of labeled videos x in video space X and label y . The test set is defined as DU = { ( x , y ) |x ∈ X , y ∈ U } . The action classes in both datasets must be disjoint , i.e . S ∩ U = ∅ . The goal of ZSAR is to predict a label for a videos of unseen class ( activities ) , i.e . X → U . For each class , a semantic embedding vector ψ ( y ) ∈ RD of length D is obtained . These vectors are traditionally obtained by averaging word2vec [ 5 ] embeddings for each word in the class name , however different text encoders ( e.g . sent2vec [ 29 ] , BERT [ 30 ] ) could also be used . Conventional ZSAR methods learn a parameterized model fθ which maps an input video to the semantic space , i.e . fθ : X → RD . Then , for a given video , x , a classification is performed by a nearest neighbor in the set of test class embeddings , as defined by : F ( x ) = argmin y∈U d ( fθ ( x ) , ψ ( y ) ) , ( 1 ) where d is a distance metric ( e.g . cosine distance ) . We argue that this two stage approach - metric learning on the semantic space and nearest neighbor classification - is ineffective . Instead , we reformulate the ZSAR problem to learn a scoring function hθ which predicts a match vs. no-match probability when given a video and semantic embedding pair , i.e . hθ : X × RD → [ 0 , 1 ] . Once the network is trained , the final action classification can be obtained by selecting the class which achieves the highest probability : F ( x ) = argmax y∈U hθ ( x , ψ ( y ) ) . ( 2 ) This formulation allows for multi-label action recognition by selecting all classes with probabilities over the threshold τ , i.e . F ( x ) = { y ∈ U|hθ ( x , ψ ( y ) ) > τ } , ( 3 ) whereas nearest neighbor classification can not predict multiple semantically dissimilar actions to occur within a given video .

Summary: This work has two main contributions: (1) reformulation of the zero shot action recognition task for the multi-label case, and (2) a new model motivated by the authors idea that nearest neighbor is sub-optimal on a static semantic space. A pairwise scoring function is proposed in this work which first maps both visual and semantic embeddings into a joint space to predict match/no match probabilities of a 3D-CNN based visual representation and a Word2Vec/Sent2Vec based text representation. Their approach is evaluated not only on multi-label action recognition (where the zero shot performance is evaluated for the first time) dataset but also on single-label action recognition dataset, achieving state-of-the-art results. Evaluation protocols for single-label zero-shot action recognition datasets, UCF-101, HMDB, RareAct is proposed and another two evaluation protocols for multi-label zero-shot action recognition based on MEVA and EVA datasets are proposed in this work.