Transductive Universal Transport for Zero-Shot Action Recognition

This work addresses the problem of recognizing action categories in videos for which no training examples are available . The current state-of-the-art enables such a zero-shot recognition by learning universal mappings from videos to a shared semantic space , either trained on large-scale seen actions or on objects . While effective , we find that universal action and object models are biased to their seen categories . Such biases are further amplified due to biases between seen and unseen categories in the semantic space . The amplified biases result in many unseen action categories simply never being selected during inference , hampering zero-shot progress . We seeks to address this limitation and introduce transductive universal transport for zero-shot action recognition . Our proposal is to re-position unseen action embeddings through transduction , i.e. , by using the distribution of the unlabelled test set . For universal action models , we first find an optimal mapping from unseen actions to the mapped test videos in the shared hyperspherical space . We then propose new target embeddings as weighted Fréchet means , with the weights given by the transport couplings . Finally , we introduce a re-positioning of unseen action embeddings along the geodesic between the original and target , as a form of semantic regularization . For universal object models , we outline a weighted transport variant from unseen action embeddings to object embeddings directly . Empirically , we show that our approach directly boosts universal action and object models , resulting in state-of-the-art performance for zero-shot classification and spatio-temporal localization . 1 INTRODUCTION . This paper investigates the problem of inferring actions in videos for which no training examples were provided . Akin to advances in the image domain ( He et al. , 2016 ; Huang et al. , 2017 ; Yuan et al. , 2021 ) , deeper and more expressive network architectures have in recent years been developed to continually improve the recognition of actions in videos ( Carreira & Zisserman , 2017 ; Feichtenhofer et al. , 2019 ; Arnab et al. , 2021 ) . To meet the corresponding demand for data , multiple new datasets – each spanning many hundreds of actions – have been created , e.g. , ( Carreira & Zisserman , 2017 ; Monfort et al. , 2019 ; Damen et al. , 2020 ) . While such datasets increase the coverage of the action space , we seek to recognize actions even when no examples are available during training . In zero-shot action recognition , one line of work has found success by mirroring progress in its image-based counterpart , for example by using attributes ( Liu et al. , 2011 ; Gan et al. , 2016b ) or feature synthesis ( Mishra et al. , 2020 ) to transfer knowledge from seen to unseen actions . Recently , state-of-the-art results have been achieved by taking a universal learning perspective , where largescale models are trained to map input videos to a shared semantic space occupied by both seen and unseen categories . In the first universal perspective , large-scale networks train a mapping from videos to a semantic space on hundreds of seen actions from e.g. , ActivityNet ( Zhu et al. , 2018 ) or Kinetics ( Brattoli et al. , 2020 ) . For a target dataset with unseen actions , zero-shot inference is directly possible through a nearest neighbour search between the video mappings and the embeddings of unseen actions . In the second perspective , networks are trained on thousands of objects ( Jain et al. , 2015 ; Mettes et al. , 2021 ) , where inference becomes a function of both the object likelihoods and the semantic relation between objects and actions . While effective , the first finding of this work is that both perspectives share a common limitation : they are strongly biased to subsets of the unseen actions . Compounding biases from videos to seen categories and from seen to unseen categories result in a mismatch between the embeddings of unseen actions and mapping of the videos covering the same actions . A significant part of the actions are as a result simply never selected , disrupting progress in zero-shot action recognition . We seek to address this limitation . In this work , we introduce transductive universal transport for zero-shot action recognition with universal models . The idea is to re-position the semantic embeddings of unseen actions by using information about the distribution of the entire test set . For universal action models , we first consider the optimal transport mapping between the semantic embeddings of unseen actions and all embedded videos . We then propose a target function for each unseen action as a weighted Fréchet mean given the optimal transport couplings . To avoid loosing semantic interpretation , we finally propose to re-position unseen actions along the geodesic spanned by the original and target embeddings . We view this as a form of semantic regularization imposed on the target embeddings . For universal object models , we perform the same approach with two differences : we map the unseen actions to object embeddings directly and we add transport weights to both the unseen actions and the objects , such that we focus on actions far away from objects and on objects that are more likely to occur in the test videos . Experiments on four datasets validate our approach , with state-of-the-art results for zero-shot action recognition and spatio-temporal action localization . 2 RE-POSITIONING ACTION EMBEDDINGS WITH UNIVERSAL TRANSPORT . For the problem of zero-shot action recognition , we are given a set of test videos Vu and a set of labels Lu denoting actions which have not been observed during training . We seek to assign a label l ∈ Lu to each test video . To that end , we start from two state-of-the-art universal learning directions in zero-shot action recognition , namely by transferring knowledge from large-scale seen actions and from objects . For both directions , we seek to overcome the long-tailed distributions in label assignment of unseen actions during zero-shot inference . Below , we first introduce transductive universal transport for the transfer from seen to unseen actions , see Figure 1 . Second , we extend our approach for re-positioning unseen action embeddings based on objects . 2.1 TRANSDUCTIVE UNIVERSAL TRANSPORT FOR UNIVERSAL ACTION MODELS . Inferring unseen actions by transferring knowledge from seen actions requires two transformation functions : a function ω that maps a label to a semantic embedding space and a function φ that maps a video to the same embedding space . The mapping function φ is learned on a set of training videos Vs with seen action labels Ls , with its loss given as : Ls = ∑ v∈Vs − 〈φ ( v ) , ω ( lv ) 〉 ||φ ( v ) || · ||ω ( lv ) || , ( 1 ) with lv ∈ Ls the action label for video v. The function ω is given by a pre-trained word embedding model ( Mikolov et al. , 2013b ) , where embeddings are commonly ` 2-normalized and optimized with the cosine distance . Due to the shared semantic nature of the embedding space , any test video that is projected to the embedding space using φ can also be assigned to a label from Lu through a nearest neighbour search with all unseen labels that have been mapped using ω . In this work , we argue that since φ is trained on actions not used during inference , the mapped video embeddings and unseen action embeddings are not aligned . We propose to improve zero-shot action recognition by re-positioning unseen action embeddings with transductive information . In order to better position unseen action embeddings , we first redefine the original seen and unseen action embeddings as discrete measures so that they can be used in the context of optimal transport : Definition 1 ( Actions as discrete measures ) . The sets of seen and unseen action labels are given as : µs = ∑ ls∈Ls wlsδω ( ls ) , µu = ∑ lu∈Lu wluδω ( lu ) , ( 2 ) where wlu ∈ wu and wls ∈ ws denote the set of weights for the action , such that ws ∈ Σ|Ls| and wu ∈ Σ|Lu| are both on the probability simplex and δx denotes the Dirac at position x . In similar fashion the definition of the mapped test videos is given as : Definition 2 ( Videos as a discrete measure ) . The set of mapped test videos is given as : µv = ∑ c∈Ck ( Vu ) wcδc , c = 1 |a ( Vu ; c ) | ∑ v∈a ( Vu ; c ) φ ( v ) , ( 3 ) with wc ∈ wc ∈ Σk akin to Definition 1 , where Ck denotes a k-component cluster aggregation over the set of videos , and where a ( Vu ; c ) denotes the set of videos assigned to cluster c. With the above definitions , seen and unseen actions are interpreted as discrete measures centered around their word embeddings . For the test videos , we define the corresponding discrete measure as a clustering over the mapped video outputs . This is to make the measure itself more robust to outliers and to increase the focus towards high-density regions . The seen actions are only used to train the universal network φ ( · ) ; the unseen actions and test videos are central in our approach . With the unseen action labels and videos defined as discrete measures in the same space , we first seek to find an optimal transport mapping between the two measures . Since we operate on the entire distribution of test videos rather than performing inference for each video independently , we view this as a transductive form of optimal transport , see Figure 1 ( left ) . Although the action embeddings are discrete entities , we find that it is better to view the transport as an instance of the relaxed Monge-Kantorovich problem , which allows for mass splitting from the unseen action measure to the aggregated video measure . Let C denote the cost matrix , with Cij defined as the cosine distance between action i and video cluster j . Then the transductive optimal transport form µu to µv is given as ( Peyré et al. , 2019 ) : LC ( wu , wv ) = min P∈U ( wu , wv ) 〈C , P〉 = ∑ ij CijPij , ( 4 ) where U denotes the admissible couplings , given as : U ( wu , wv ) = { P ∈ R|Lu|k+ : P1k = wu & P1|Lu| = wv } , P1k = ( ∑ j Pij ) i , P1|Lu| = ( ∑ i Pij ) j , ( 5 ) which is solved using the Lagrangian approach of Bonneel et al . ( 2011 ) . This results in a coupling matrix P , where Pij denotes the amount of mass of action i that moves to video cluster j . When working with universal action models , we set the weights of both measures uniformly . Given the optimal transport mapping , we propose to condense the corresponding coupling into a single target embedding per unseen action . Since since the semantic space on which we operate is a hypersphere , we define the target embedding as the weighted Fréchet mean ( Lou et al. , 2020 ; Miolane et al. , 2020 ) using normalized coupling values : ωtarget ( li ) = arg min s∈Sd−1 k∑ j=1 P̂ijd ( cj , s ) 2 , k∑ j=1 P̂ij = 1 , ( 6 ) with d the cosine similarity , see Figure 1 ( middle ) . The target provides a new location in embedding space for each unseen action label , guided by the distribution of mapped and clustered test videos . While we can directly use the new embeddings for inference , we want to avoid big changes in embedding space since that relates with losing the original semantic interpretation of the action . We therefore dictate that the final embedding of each unseen action is positioned along the geodesic spanned by the original and target embeddings , modelled through spherical interpolation : ω ? ( l ) = sin [ λΩ ] sin Ω ω ( l ) + sin [ ( 1− λ ) Ω ] sin Ω ωtarget ( l ) , cos Ω = 〈ω ( l ) , ωtarget ( l ) 〉 , ( 7 ) where λ denotes the balance between maintaining the original embedding and moving towards the transductive target embedding . In this manner , we move each unseen action towards its proposed target embedding , with the interpolation acting as a regularization towards the original embedding . Zero-shot action recognition is performed by computing the cosine similarity between the projected video v and each unseen action label l as saction ( l|v ) = 〈φ ( v ) , ω ? ( l ) 〉 ||φ ( v ) || · ||ω ? ( l ) || , ( 8 ) after which the action label with the highest similarity is selected . Figure 1 ( right ) provides an illustration of the final re-positioning , where we move the original unseen action embeddings along the greater arc of the hypersphere towards the optimal target embeddings .

Summary: The paper presents a zero-shot action recognition framework by learning an universal mapping from video to semantic space. The unseen action embeddings are re-positioned through leveraging the distribution of unlabelled test set. The universal mappings from unseen action to test videos are first defined and the target embeddings are treated as weighted Frechet means. The unseen action embeddings are re-positioned as a semantic regularization. The results on UCF101 and HMDB-51, UCF Sports and J-HMDB validates the proposed method.