Transductive Universal Transport for Zero-Shot Action Recognition

This work addresses the problem of recognizing action categories in videos for which no training examples are available . The current state-of-the-art enables such a zero-shot recognition by learning universal mappings from videos to a shared semantic space , either trained on large-scale seen actions or on objects . While effective , we find that universal action and object models are biased to their seen categories . Such biases are further amplified due to biases between seen and unseen categories in the semantic space . The amplified biases result in many unseen action categories simply never being selected during inference , hampering zero-shot progress . We seeks to address this limitation and introduce transductive universal transport for zero-shot action recognition . Our proposal is to re-position unseen action embeddings through transduction , i.e. , by using the distribution of the unlabelled test set . For universal action models , we first find an optimal mapping from unseen actions to the mapped test videos in the shared hyperspherical space . We then propose new target embeddings as weighted Fréchet means , with the weights given by the transport couplings . Finally , we introduce a re-positioning of unseen action embeddings along the geodesic between the original and target , as a form of semantic regularization . For universal object models , we outline a weighted transport variant from unseen action embeddings to object embeddings directly . Empirically , we show that our approach directly boosts universal action and object models , resulting in state-of-the-art performance for zero-shot classification and spatio-temporal localization . 1 INTRODUCTION . This paper investigates the problem of inferring actions in videos for which no training examples were provided . Akin to advances in the image domain ( He et al. , 2016 ; Huang et al. , 2017 ; Yuan et al. , 2021 ) , deeper and more expressive network architectures have in recent years been developed to continually improve the recognition of actions in videos ( Carreira & Zisserman , 2017 ; Feichtenhofer et al. , 2019 ; Arnab et al. , 2021 ) . To meet the corresponding demand for data , multiple new datasets – each spanning many hundreds of actions – have been created , e.g. , ( Carreira & Zisserman , 2017 ; Monfort et al. , 2019 ; Damen et al. , 2020 ) . While such datasets increase the coverage of the action space , we seek to recognize actions even when no examples are available during training . In zero-shot action recognition , one line of work has found success by mirroring progress in its image-based counterpart , for example by using attributes ( Liu et al. , 2011 ; Gan et al. , 2016b ) or feature synthesis ( Mishra et al. , 2020 ) to transfer knowledge from seen to unseen actions . Recently , state-of-the-art results have been achieved by taking a universal learning perspective , where largescale models are trained to map input videos to a shared semantic space occupied by both seen and unseen categories . In the first universal perspective , large-scale networks train a mapping from videos to a semantic space on hundreds of seen actions from e.g. , ActivityNet ( Zhu et al. , 2018 ) or Kinetics ( Brattoli et al. , 2020 ) . For a target dataset with unseen actions , zero-shot inference is directly possible through a nearest neighbour search between the video mappings and the embeddings of unseen actions . In the second perspective , networks are trained on thousands of objects ( Jain et al. , 2015 ; Mettes et al. , 2021 ) , where inference becomes a function of both the object likelihoods and the semantic relation between objects and actions . While effective , the first finding of this work is that both perspectives share a common limitation : they are strongly biased to subsets of the unseen actions . Compounding biases from videos to seen categories and from seen to unseen categories result in a mismatch between the embeddings of unseen actions and mapping of the videos covering the same actions . A significant part of the actions are as a result simply never selected , disrupting progress in zero-shot action recognition . We seek to address this limitation . In this work , we introduce transductive universal transport for zero-shot action recognition with universal models . The idea is to re-position the semantic embeddings of unseen actions by using information about the distribution of the entire test set . For universal action models , we first consider the optimal transport mapping between the semantic embeddings of unseen actions and all embedded videos . We then propose a target function for each unseen action as a weighted Fréchet mean given the optimal transport couplings . To avoid loosing semantic interpretation , we finally propose to re-position unseen actions along the geodesic spanned by the original and target embeddings . We view this as a form of semantic regularization imposed on the target embeddings . For universal object models , we perform the same approach with two differences : we map the unseen actions to object embeddings directly and we add transport weights to both the unseen actions and the objects , such that we focus on actions far away from objects and on objects that are more likely to occur in the test videos . Experiments on four datasets validate our approach , with state-of-the-art results for zero-shot action recognition and spatio-temporal action localization . 2 RE-POSITIONING ACTION EMBEDDINGS WITH UNIVERSAL TRANSPORT . For the problem of zero-shot action recognition , we are given a set of test videos Vu and a set of labels Lu denoting actions which have not been observed during training . We seek to assign a label l ∈ Lu to each test video . To that end , we start from two state-of-the-art universal learning directions in zero-shot action recognition , namely by transferring knowledge from large-scale seen actions and from objects . For both directions , we seek to overcome the long-tailed distributions in label assignment of unseen actions during zero-shot inference . Below , we first introduce transductive universal transport for the transfer from seen to unseen actions , see Figure 1 . Second , we extend our approach for re-positioning unseen action embeddings based on objects . 2.1 TRANSDUCTIVE UNIVERSAL TRANSPORT FOR UNIVERSAL ACTION MODELS . Inferring unseen actions by transferring knowledge from seen actions requires two transformation functions : a function ω that maps a label to a semantic embedding space and a function φ that maps a video to the same embedding space . The mapping function φ is learned on a set of training videos Vs with seen action labels Ls , with its loss given as : Ls = ∑ v∈Vs − 〈φ ( v ) , ω ( lv ) 〉 ||φ ( v ) || · ||ω ( lv ) || , ( 1 ) with lv ∈ Ls the action label for video v. The function ω is given by a pre-trained word embedding model ( Mikolov et al. , 2013b ) , where embeddings are commonly ` 2-normalized and optimized with the cosine distance . Due to the shared semantic nature of the embedding space , any test video that is projected to the embedding space using φ can also be assigned to a label from Lu through a nearest neighbour search with all unseen labels that have been mapped using ω . In this work , we argue that since φ is trained on actions not used during inference , the mapped video embeddings and unseen action embeddings are not aligned . We propose to improve zero-shot action recognition by re-positioning unseen action embeddings with transductive information . In order to better position unseen action embeddings , we first redefine the original seen and unseen action embeddings as discrete measures so that they can be used in the context of optimal transport : Definition 1 ( Actions as discrete measures ) . The sets of seen and unseen action labels are given as : µs = ∑ ls∈Ls wlsδω ( ls ) , µu = ∑ lu∈Lu wluδω ( lu ) , ( 2 ) where wlu ∈ wu and wls ∈ ws denote the set of weights for the action , such that ws ∈ Σ|Ls| and wu ∈ Σ|Lu| are both on the probability simplex and δx denotes the Dirac at position x . In similar fashion the definition of the mapped test videos is given as : Definition 2 ( Videos as a discrete measure ) . The set of mapped test videos is given as : µv = ∑ c∈Ck ( Vu ) wcδc , c = 1 |a ( Vu ; c ) | ∑ v∈a ( Vu ; c ) φ ( v ) , ( 3 ) with wc ∈ wc ∈ Σk akin to Definition 1 , where Ck denotes a k-component cluster aggregation over the set of videos , and where a ( Vu ; c ) denotes the set of videos assigned to cluster c. With the above definitions , seen and unseen actions are interpreted as discrete measures centered around their word embeddings . For the test videos , we define the corresponding discrete measure as a clustering over the mapped video outputs . This is to make the measure itself more robust to outliers and to increase the focus towards high-density regions . The seen actions are only used to train the universal network φ ( · ) ; the unseen actions and test videos are central in our approach . With the unseen action labels and videos defined as discrete measures in the same space , we first seek to find an optimal transport mapping between the two measures . Since we operate on the entire distribution of test videos rather than performing inference for each video independently , we view this as a transductive form of optimal transport , see Figure 1 ( left ) . Although the action embeddings are discrete entities , we find that it is better to view the transport as an instance of the relaxed Monge-Kantorovich problem , which allows for mass splitting from the unseen action measure to the aggregated video measure . Let C denote the cost matrix , with Cij defined as the cosine distance between action i and video cluster j . Then the transductive optimal transport form µu to µv is given as ( Peyré et al. , 2019 ) : LC ( wu , wv ) = min P∈U ( wu , wv ) 〈C , P〉 = ∑ ij CijPij , ( 4 ) where U denotes the admissible couplings , given as : U ( wu , wv ) = { P ∈ R|Lu|k+ : P1k = wu & P1|Lu| = wv } , P1k = ( ∑ j Pij ) i , P1|Lu| = ( ∑ i Pij ) j , ( 5 ) which is solved using the Lagrangian approach of Bonneel et al . ( 2011 ) . This results in a coupling matrix P , where Pij denotes the amount of mass of action i that moves to video cluster j . When working with universal action models , we set the weights of both measures uniformly . Given the optimal transport mapping , we propose to condense the corresponding coupling into a single target embedding per unseen action . Since since the semantic space on which we operate is a hypersphere , we define the target embedding as the weighted Fréchet mean ( Lou et al. , 2020 ; Miolane et al. , 2020 ) using normalized coupling values : ωtarget ( li ) = arg min s∈Sd−1 k∑ j=1 P̂ijd ( cj , s ) 2 , k∑ j=1 P̂ij = 1 , ( 6 ) with d the cosine similarity , see Figure 1 ( middle ) . The target provides a new location in embedding space for each unseen action label , guided by the distribution of mapped and clustered test videos . While we can directly use the new embeddings for inference , we want to avoid big changes in embedding space since that relates with losing the original semantic interpretation of the action . We therefore dictate that the final embedding of each unseen action is positioned along the geodesic spanned by the original and target embeddings , modelled through spherical interpolation : ω ? ( l ) = sin [ λΩ ] sin Ω ω ( l ) + sin [ ( 1− λ ) Ω ] sin Ω ωtarget ( l ) , cos Ω = 〈ω ( l ) , ωtarget ( l ) 〉 , ( 7 ) where λ denotes the balance between maintaining the original embedding and moving towards the transductive target embedding . In this manner , we move each unseen action towards its proposed target embedding , with the interpolation acting as a regularization towards the original embedding . Zero-shot action recognition is performed by computing the cosine similarity between the projected video v and each unseen action label l as saction ( l|v ) = 〈φ ( v ) , ω ? ( l ) 〉 ||φ ( v ) || · ||ω ? ( l ) || , ( 8 ) after which the action label with the highest similarity is selected . Figure 1 ( right ) provides an illustration of the final re-positioning , where we move the original unseen action embeddings along the greater arc of the hypersphere towards the optimal target embeddings .

Summary: This work introduces transductive universal transport for zero-shot action recognition, where no training examples for unseen classes are available. To address the biases of prior approaches towards seen classes during inference, this paper re-positions unseen action embeddings through transduction by using the distribution of the unlabelled test set. Experimental results on several action recognition datasets demonstrate the effectiveness of the proposed method.