Unseen Action Recognition with Unpaired Adversarial Multimodal Learning

In this paper , we present a method to learn a joint multimodal representation space that allows for the recognition of unseen activities in videos . We compare the effect of placing various constraints on the embedding space using paired text and video data . Additionally , we propose a method to improve the joint embedding space using an adversarial formulation with unpaired text and video data . In addition to testing on publicly available datasets , we introduce a new , large-scale text/video dataset . We experimentally confirm that learning such shared embedding space benefits three difficult tasks ( i ) zero-shot activity classification , ( ii ) unsupervised activity discovery , and ( iii ) unseen activity captioning . 1 INTRODUCTION . Videos contain multiple data sources , such as visual , audio and text/caption data . Each data modality has distinct statistical properties capturing different aspects of the event . Current state-of-the-art activity recognition models ( Carreira & Zisserman , 2017 ; Tran et al. , 2017 ) only take visual data and class labels as input , which limits the information the model can learn from . For example , the sentence ‘ a group of men play basketball outdoors ’ contains rich information , such as ‘ outdoors ’ and ‘ group of men ’ compared to just the activity class of ‘ basketball. ’ Without multimodal learning , these models are unable to benefit from the additional information . In this paper , we explore multimodal learning from video and language data , each having its own representation . Video data is represented as a sequence of images ( spatio-temporal pixel data ) while text is represented as a sequence of word embeddings ( temporal data ) . Using a shared embedding space allows for learning the highly non-linear relationships between these modalities . The goal of learning such relationships is to capture similarities between concepts ( e.g. , basketball and volleyball both being sports with a ball ) and further generalize to concepts not seen during training . To enable this generalization , we rely on pre-trained word embeddings as they capture relationships between all words in the English language ( Mikolov et al. , 2013 ) . By learning a joint representation space , we can capture these relationships , allowing generalization to unseen video activities . To evaluate our approach , we test on several tasks based on recognizing activities not seen during training . First , we test on a zero-shot learning task , where we classify activity classes not seen during training . Our second task is unsupervised activity discovery , where no class information is provided during training , relying only on the relationship between text and videos to learn good embeddings . Finally , we evaluate on captioning videos with activities/words not seen during training . Many existing approaches to both zero-shot and embedding space learning require paired data examples , which can be expensive to obtain . By taking advantage of adversarial learning ( Goodfellow et al. , 2014 ) , we are able augment our method with unpaired data ( i.e. , random sentences and random videos without any labels or correspondence ) to further improve our learned shared embedding space . In this paper , we design a method capable of learning joint video/language embedding spaces using both paired and unpaired data and experimentally confirm its benefit to three challenging tasks ( i ) zero-shot activity recognition , ( ii ) unsupervised activity discovery , and ( iii ) unseen activity captioning . We show that the use of unpaired , multimodal data allows learning a shared embedding space that generalizes to unseen data better than existing approaches . 2 RELATED WORKS . Multimodal learning Previous approaches to multimodal learning have used Restricted Boltzmann Machines ( Srivastava & Salakhutdinov , 2012 ) or log-bilinear models ( Kiros et al. , 2014 ) to learn distributions over sentences and images . Ngiam et al . ( 2011 ) designed an autoencoder that learns joint audio-video representations , however they are unable to train the model end-to-end , instead relying on greedy , layer-by-layer training and further do not evaluate the effect of using unpaired data . Similarly , Chandar et al . ( 2016 ) proposed a auto-encoder able to learn correlations between different view of images . Frome et al . ( 2013 ) describe a model that maps images and words to a shared embedding . However , these works either learn a joint embedding by concatenating the different features or require a triplet consisting of positive and negative matches and do not evaluate the effects of using unpaired data . Text and vision Using both text and visual data has been studied for many tasks , such as image captioning ( Karpathy et al. , 2014 ; Johnson et al. , 2016 ; Karpathy & Fei-Fei , 2015 ) or video captioning ( Krishna et al. , 2017 ; Zhou et al. , 2018 ; Xu et al. , 2018 ) . Other works have explored the use of textual grounding for image/video retrieval ( Gupta & Mooney , 2010 ; Rohrbach et al. , 2016 ; Miech et al. , 2017 ; Hendricks et al. , 2017 ) . There have been various models proposed to learn a fixed text embedding space with mappings from video features into this embedding space ( Guadarrama et al. , 2013 ; Otani et al. , 2016 ; Song et al. , 2016 ; Wang et al. , 2016 ; Xian et al. , 2016 ) . However , these works all learn only a single directional mapping ( e.g. , only mapping from visual to text ) , only learn with paired text/image samples and require additional data in the form of positive/negative pairs . Learning with unpaired data Recently , there have been many works taking advantage of variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) or generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) to learn mappings between unpaired samples . CycleGan ( Zhu et al. , 2017 ) uses a cycle-consistency loss ( i.e. , the ability to go from a sample in one domain to a second domain then back to the source ) to learn unpaired image translation ( e.g. , image to sketch ) . Other works learn many-to-many mappings between images ( Almahairi et al. , 2018 ) or use two GANs to map between domains ( Yi et al. , 2017 ) . An autoencoder with shared weights for both domains has been used to learn a latent space for image-to-image translation ( Liu et al. , 2017 ) . However , these works all focus on learning mappings between unpaired data of the same modalitiy ( e.g . image to image ) , where the data is from the same underlying distribution . Our work focuses on learning from different modalities with very different distributions . Zero-shot activity recognition Many previous works have studied zero-shot activity recognition . Common approaches include using attributes ( Liu et al. , 2011 ; Palatucci et al. , 2009 ; Romera-Paredes & Torr , 2015 ) or word embeddings ( Xu et al. , 2015 ; 2017 ; Norouzi et al. , 2013 ; Socher et al. , 2013 ; Kodirov et al. , 2017 ) or learning a similarity metric ( Zhang & Saligrama , 2015 ; Chopra et al. , 2005 ) . Some works have explored using adversarial losses on the latent space ( Chen et al. , 2018 ) or use GANs to generate features for unseen classes ( Xian et al. , 2018 ) or used auto-encoders ( Wang et al. , 2017 ) . Our work differs from these previous works in three key ways : ( 1 ) we show the benefit of additional data augmentation using unpaired samples , ( 2 ) we experimentally compare the use of the embedding space for both zero-shot recognition , unseen recognition , and unseen video captioning , and ( 3 ) we learn a shared , multimodal embedding space with bi-directional mappings in an end-to-end fashion . 3 METHOD . To enable learning of a joint embedding space , we use a deep autoencoder architecture . Our model consists of 4 neural networks : Video Encoder EV : v 7→ zv Video Decoder GV : z 7→ v Text Encoder ET : t 7→ zt Text Decoder GT : z 7→ t where v is a sequence of video data and t is a sentence ( sequence of words ) . z is the embedded data in the shared space that we are learning . The encoders learn a compressed representation of the video or text while the decoders are trained to reconstruct the input : Lrecons ( v , t ) = ||GV ( EV ( v ) ) − v||2 + ||GT ( ET ( t ) ) − t||2 ( 1 ) As both text and video data are sequences , they often have different lengths . A joint embedding space requires that the features from both modalities have the same dimensions . Given a text representation of length L and a video representation of length T , we need to obtain a fixed-size representation . To learn a fixed-dimensional embedding , there are many choices for the encoder/decoder architecture , such as temporal pooling ( Ng et al. , 2015 ) , attention ( Piergiovanni et al. , 2017 ) or RNNs ( Krishna et al. , 2017 ) . We chose temporal attention filters ( Piergiovanni et al. , 2017 ) as they learn a mapping from any length input to a N -dimensional vector and have been shown to outperform temporal pooling and RNNs on activity recognition tasks . The attention filters consists of N Gaussians , each learning 2 parameters : a center ĝ and width σ . The filters are determined by : gn = 0.5 · T · ( g̃n + 1 ) µin = gn + ( i− 0.5N + 0.5 ) F [ n , t ] = 1 Z exp ( − ( t− µn ) 2 2σ2n ) n ∈ { 0 , 1 , . . . , N − 1 } , t ∈ { 0 , 1 , . . . , T − 1 } ( 2 ) The weights are applied by matrix multiplication with the video or text sequence : v′ = Fv . Additionally , we can learn a transposed version of these filters to reconstruct the input : v = FT v′ . To reconstruct the input , the decoders learn their own parameters with the tensors transposed , resulting in the matching output size . The architecture of the encoders is shown in Fig . 1 . 3.1 LEARNING A JOINT EMBEDDING SPACE . To learn a joint representation space , we minimize the L2 distance between the embeddings of a pair of text and video ( shown in Fig . 2 ( a ) ) : Ljoint ( v , t ) = ||EV ( v ) − ET ( t ) ||2 ( 3 ) This forces the joint embeddings to be similar and when combined with the reconstruction loss , ensures that the representations can still reconstruct the input . We can further constrain the networks and learned representation by forcing a cross-domain mapping from text to video and from video to text ( shown in Fig . 2 ( b ) ) : Lcross ( v , t ) = ||GT ( EV ( v ) ) − t||2 + ||GV ( ET ( t ) ) − v||2 ( 4 ) Additionally , we can use a ‘ cycle ’ loss to map from video to text and back to video . Note that while the previous losses all require paired examples , this loss does not . Lcycle ( v , t ) = ||GT ( EV ( GV ( ET ( t ) ) ) ) − t||2 + ||GV ( ET ( GT ( EV ( v ) ) ) ) − v||2 ( 5 ) To train the model to learn a joint embedding space , we minimize L ( v , t ) = Lrecons ( v , t ) + α1Ljoint ( v , t ) + α2Lcross ( v , t ) + α3Lcycle ( v , t ) ( 6 ) where αi are hyper-parameters weighting the various loss components .

Summary: This paper proposes a joint embedding model that aligns video sequences with sentences describing the context (caption) in a shared embedding space. With the space, various tasks such as zero-shot activity recognition and unseen video captioning can be performed. The problem tackled in this paper is interesting. However, the approach proposed is limited in novelty and there are some serious flaws in the experimental settings. So overall, this paper is not yet ready for publication.