Year: 2021

Abstract:
This work addresses the problem of recognizing action categories in videos for which no training examples are available. The current state-of-the-art enables such a zero-shot recognition by learning universal mappings from videos to a shared semantic space, either trained on large-scale seen actions or on objects. While effective, we find that universal action and object models are biased to their seen categories. Such biases are further amplified due to biases between seen and unseen categories in the semantic space. The amplified biases result in many unseen action categories simply never being selected during inference, hampering zero-shot progress. We seeks to address this limitation and introduce transductive universal transport for zero-shot action recognition. Our proposal is to re-position unseen action embeddings through transduction, i.e., by using the distribution of the unlabelled test set. For universal action models, we first find an optimal mapping from unseen actions to the mapped test videos in the shared hyperspherical space. We then propose new target embeddings as weighted Fréchet means, with the weights given by the transport couplings. Finally, we introduce a re-positioning of unseen action embeddings along the geodesic between the original and target, as a form of semantic regularization. For universal object models, we outline a weighted transport variant from unseen action embeddings to object embeddings directly. Empirically, we show that our approach directly boosts universal action and object models, resulting in state-of-the-art performance for zero-shot classification and spatio-temporal localization.

text: This work addresses the problem of recognizing action categories in videos for which no training examples are available. The current state-of-the-art enables such a zero-shot recognition by learning universal mappings from videos to a shared semantic space, either trained on large-scale seen actions or on objects. While effective, we find that universal action and object models are biased to their seen categories. Such biases are further amplified due to biases between seen and unseen categories in the semantic space. The amplified biases result in many unseen action categories simply never being selected during inference, hampering zero-shot progress. We seeks to address this limitation and introduce transductive universal transport for zero-shot action recognition. Our proposal is to re-position unseen action embeddings through transduction, i.e., by using the distribution of the unlabelled test set. For universal action models, we first find an optimal mapping from unseen actions to the mapped test videos in the shared hyperspherical space. We then propose new target embeddings as weighted Fréchet means, with the weights given by the transport couplings. Finally, we introduce a re-positioning of unseen action embeddings along the geodesic between the original and target, as a form of semantic regularization. For universal object models, we outline a weighted transport variant from unseen action embeddings to object embeddings directly. Empirically, we show that our approach directly boosts universal action and object models, resulting in state-of-the-art performance for zero-shot classification and spatio-temporal localization.

heading: 1 INTRODUCTION
text: This paper investigates the problem of inferring actions in videos for which no training examples were provided. Akin to advances in the image domain (He et al., 2016; Huang et al., 2017; Yuan et al., 2021), deeper and more expressive network architectures have in recent years been developed to continually improve the recognition of actions in videos (Carreira & Zisserman, 2017; Feichtenhofer et al., 2019; Arnab et al., 2021). To meet the corresponding demand for data, multiple new datasets – each spanning many hundreds of actions – have been created, e.g., (Carreira & Zisserman, 2017; Monfort et al., 2019; Damen et al., 2020). While such datasets increase the coverage of the action space, we seek to recognize actions even when no examples are available during training.
In zero-shot action recognition, one line of work has found success by mirroring progress in its image-based counterpart, for example by using attributes (Liu et al., 2011; Gan et al., 2016b) or feature synthesis (Mishra et al., 2020) to transfer knowledge from seen to unseen actions. Recently, state-of-the-art results have been achieved by taking a universal learning perspective, where largescale models are trained to map input videos to a shared semantic space occupied by both seen and unseen categories. In the first universal perspective, large-scale networks train a mapping from videos to a semantic space on hundreds of seen actions from e.g., ActivityNet (Zhu et al., 2018) or Kinetics (Brattoli et al., 2020). For a target dataset with unseen actions, zero-shot inference is directly possible through a nearest neighbour search between the video mappings and the embeddings of unseen actions. In the second perspective, networks are trained on thousands of objects (Jain et al., 2015; Mettes et al., 2021), where inference becomes a function of both the object likelihoods and the semantic relation between objects and actions.
While effective, the first finding of this work is that both perspectives share a common limitation: they are strongly biased to subsets of the unseen actions. Compounding biases from videos to seen categories and from seen to unseen categories result in a mismatch between the embeddings of unseen actions and mapping of the videos covering the same actions. A significant part of the actions are as a result simply never selected, disrupting progress in zero-shot action recognition. We seek to address this limitation.
In this work, we introduce transductive universal transport for zero-shot action recognition with universal models. The idea is to re-position the semantic embeddings of unseen actions by using information about the distribution of the entire test set. For universal action models, we first consider the optimal transport mapping between the semantic embeddings of unseen actions and all embedded videos. We then propose a target function for each unseen action as a weighted Fréchet mean given the optimal transport couplings. To avoid loosing semantic interpretation, we finally propose to re-position unseen actions along the geodesic spanned by the original and target embeddings. We view this as a form of semantic regularization imposed on the target embeddings. For universal object models, we perform the same approach with two differences: we map the unseen actions to object embeddings directly and we add transport weights to both the unseen actions and the objects, such that we focus on actions far away from objects and on objects that are more likely to occur in the test videos. Experiments on four datasets validate our approach, with state-of-the-art results for zero-shot action recognition and spatio-temporal action localization.

heading: 2 RE-POSITIONING ACTION EMBEDDINGS WITH UNIVERSAL TRANSPORT
text: For the problem of zero-shot action recognition, we are given a set of test videos Vu and a set of labels Lu denoting actions which have not been observed during training. We seek to assign a label l ∈ Lu to each test video. To that end, we start from two state-of-the-art universal learning directions in zero-shot action recognition, namely by transferring knowledge from large-scale seen actions and from objects. For both directions, we seek to overcome the long-tailed distributions in label assignment of unseen actions during zero-shot inference. Below, we first introduce transductive universal transport for the transfer from seen to unseen actions, see Figure 1. Second, we extend our approach for re-positioning unseen action embeddings based on objects.

heading: 2.1 TRANSDUCTIVE UNIVERSAL TRANSPORT FOR UNIVERSAL ACTION MODELS
text: Inferring unseen actions by transferring knowledge from seen actions requires two transformation functions: a function ω that maps a label to a semantic embedding space and a function φ that maps a video to the same embedding space. The mapping function φ is learned on a set of training videos
Vs with seen action labels Ls, with its loss given as: Ls = ∑ v∈Vs − 〈φ(v), ω(lv)〉 ||φ(v)|| · ||ω(lv)|| , (1)
with lv ∈ Ls the action label for video v. The function ω is given by a pre-trained word embedding model (Mikolov et al., 2013b), where embeddings are commonly `2-normalized and optimized with the cosine distance. Due to the shared semantic nature of the embedding space, any test video that is projected to the embedding space using φ can also be assigned to a label from Lu through a nearest neighbour search with all unseen labels that have been mapped using ω. In this work, we argue that since φ is trained on actions not used during inference, the mapped video embeddings and unseen action embeddings are not aligned. We propose to improve zero-shot action recognition by re-positioning unseen action embeddings with transductive information.
In order to better position unseen action embeddings, we first redefine the original seen and unseen action embeddings as discrete measures so that they can be used in the context of optimal transport:
Definition 1 (Actions as discrete measures). The sets of seen and unseen action labels are given as: µs = ∑ ls∈Ls wlsδω(ls), µu = ∑ lu∈Lu wluδω(lu), (2)
where wlu ∈ wu and wls ∈ ws denote the set of weights for the action, such that ws ∈ Σ|Ls| and wu ∈ Σ|Lu| are both on the probability simplex and δx denotes the Dirac at position x.
In similar fashion the definition of the mapped test videos is given as:
Definition 2 (Videos as a discrete measure). The set of mapped test videos is given as: µv = ∑
c∈Ck(Vu)
wcδc, c = 1 |a(Vu; c)| ∑
v∈a(Vu;c)
φ(v), (3)
with wc ∈ wc ∈ Σk akin to Definition 1, where Ck denotes a k-component cluster aggregation over the set of videos, and where a(Vu; c) denotes the set of videos assigned to cluster c.
With the above definitions, seen and unseen actions are interpreted as discrete measures centered around their word embeddings. For the test videos, we define the corresponding discrete measure as a clustering over the mapped video outputs. This is to make the measure itself more robust to outliers and to increase the focus towards high-density regions. The seen actions are only used to train the universal network φ(·); the unseen actions and test videos are central in our approach. With the unseen action labels and videos defined as discrete measures in the same space, we first seek to find an optimal transport mapping between the two measures. Since we operate on the entire distribution of test videos rather than performing inference for each video independently, we view this as a transductive form of optimal transport, see Figure 1 (left). Although the action embeddings are discrete entities, we find that it is better to view the transport as an instance of the relaxed Monge-Kantorovich problem, which allows for mass splitting from the unseen action measure to the aggregated video measure. Let C denote the cost matrix, with Cij defined as the cosine distance between action i and video cluster j. Then the transductive optimal transport form µu to µv is given as (Peyré et al., 2019):
LC(wu,wv) = min P∈U(wu,wv) 〈C,P〉 = ∑ ij CijPij , (4)
where U denotes the admissible couplings, given as:
U(wu,wv) = { P ∈ R|Lu|k+ : P1k = wu&P1|Lu| = wv } ,
P1k = ( ∑ j Pij)i, P1|Lu| = ( ∑ i Pij)j , (5)
which is solved using the Lagrangian approach of Bonneel et al. (2011). This results in a coupling matrix P, where Pij denotes the amount of mass of action i that moves to video cluster j. When working with universal action models, we set the weights of both measures uniformly.
Given the optimal transport mapping, we propose to condense the corresponding coupling into a single target embedding per unseen action. Since since the semantic space on which we operate is a hypersphere, we define the target embedding as the weighted Fréchet mean (Lou et al., 2020; Miolane et al., 2020) using normalized coupling values:
ωtarget(li) = arg min s∈Sd−1 k∑ j=1 P̂ijd(cj , s) 2,
k∑ j=1 P̂ij = 1, (6)
with d the cosine similarity, see Figure 1 (middle). The target provides a new location in embedding space for each unseen action label, guided by the distribution of mapped and clustered test videos. While we can directly use the new embeddings for inference, we want to avoid big changes in embedding space since that relates with losing the original semantic interpretation of the action. We therefore dictate that the final embedding of each unseen action is positioned along the geodesic spanned by the original and target embeddings, modelled through spherical interpolation:
ω?(l) = sin[λΩ]
sin Ω ω(l) + sin[(1− λ)Ω] sin Ω ωtarget(l), cos Ω = 〈ω(l), ωtarget(l)〉, (7)
where λ denotes the balance between maintaining the original embedding and moving towards the transductive target embedding. In this manner, we move each unseen action towards its proposed target embedding, with the interpolation acting as a regularization towards the original embedding. Zero-shot action recognition is performed by computing the cosine similarity between the projected video v and each unseen action label l as
saction(l|v) = 〈φ(v), ω?(l)〉 ||φ(v)|| · ||ω?(l)|| , (8)
after which the action label with the highest similarity is selected. Figure 1 (right) provides an illustration of the final re-positioning, where we move the original unseen action embeddings along the greater arc of the hypersphere towards the optimal target embeddings.

heading: 2.2 TRANSDUCTIVE UNIVERSAL TRANSPORT FOR UNIVERSAL OBJECT MODELS
text: Universal object models form a competitive and similarly biased alternative for zero-shot action recognition. To tackle these biases, we also redefine objects as discrete measures to make them suitable in the context of optimal transport:
Definition 3 (Objects as a discrete measure). The sets of object labels are given as: µo = ∑ o∈Os woδω(o), Os = {o ∈ O | max v∈Vu p(o|v) ≥ τ}, (9)
with wo ∈ oc and where p(o|v) denotes the likelihood of object o occurring in video v.
Different from both the action and video measure definitions, the discrete measure for objects is based on a subset Os ∈ O, i.e., we define a degenerate distribution over the objects. The subset is determined by again taking a transductive view; we exclude any object which does not have a likelihood over a low threshold τ in any test video. The idea behind this definition is to avoid a bias in the optimal transport to objects which do not actually occur in videos.
Akin to our approach for universal action models, we compute a coupling between the unseen action measure and the object measure, as given by Equations 4 and 5. Differently however, we set non-uniform weights for both the actions and objects. The objects are weighted according to their transductive maximum score, wo = maxv∈Vu p(o|v)/Zo, with Zo a normalization constant over all objects in Os. The unseen action weights are given as wa = (1 − ((maxo∈Os〈ω(a), ω(o)〉/2) + 0.5))/Za, with Za a normalization constant over all actions. The intuition behind the object weights is to focus the attention of the transductive universal transport on objects with a higher visual likelihood. The weights for the unseen actions are given as the inverse over the maximum word embedding similarity with respect to the objects, under the notion that actions without obvious relations to objects should have a more prominent spot in the transport coupling. With the optimal transport computed between unseen actions and objects, action embeddings are again interpolated following Equations 6 and 7, with the updated embedding for action label l now denoted as ω‡(l).
For the final zero-shot action inference from objects, we follow the same setup as current objectbased approaches, where the score for each action label l in video v is determined based on the top relevant objects for that action (Jain et al., 2015; Mettes et al., 2021):
sobject(l|v) = ∑ o∈Ol p(o|v) · 〈ω(o), ω ‡(l)〉 ||ω(o)|| · ||ω‡(l)|| . (10)
with Ol the set of most semantically similar objects for action label l. Finally, the transductive action-based and object-based scores from respectively Equations 8 and 10 can also be fused as sfusion(l|v) = · saction(l|v) + (1− ) · sobject(l|v).

heading: 3 EXPERIMENTAL SETUP

heading: 3.1 DATASETS
text: Source datasets. We employ two source datasets for universal representation learning, namely Kinetics-700 for actions and ImageNet for objects. For Kinetics-700, we follow Brattoli et al. (2020) and use a subset with 664 action categories to avoid any overlap with action categories in datasets used for zero-shot action recognition. For ImageNet, we follow Mettes et al. (2021) and use the reorganized variant containing 12,988 object categories (Mettes et al., 2020).
Target datasets. The classification evaluation is performed on the two datasets used most often in zero-shot action recognition: UCF-101 and HMDB51. The UCF-101 dataset consists of 13,320 videos covering 101 action categories. Next to 101-way zero-shot evaluation, we also evaluate on settings with 20 and 50 test actions. For these settings, we rerun our approach on 10 runs with randomly selected actions and we report the mean and standard deviation over the runs. We note that in the 20- and 50-way zero-shot recognition, we do not use the other actions for training, they are simply not used in our approach. The HMDB51 dataset consists of 6,766 videos covering 51 action categories. Next to 51-way evaluation, we also investigate 10- and 25-way zero-shot recognition.
We also investigate the potential of our approach for zero-shot spatio-temporal action localization on UCF Sports and J-HMDB. UCF Sports consists of 150 videos with 10 actions and J-HMDB consists of 928 videos with 21 actions. For the evaluation, we follow Jain et al. (2015) and report results with the AUC metric across five overlap thresholds.

heading: 3.2 IMPLEMENTATION DETAILS
text: For the universal action network φ, we employ the R(2+1)D network (Tran et al., 2018) as given by Brattoli et al. (2020), which is pre-trained on the 664 action categories from Kinetics. For each video, we obtain its corresponding video embedding by randomly selecting a 16 frame shot and passing the shot through φ. For a fair comparison to the state-of-the-art by Brattoli et al. (2020), we also use the same word embedding ω, namely a word2vec model (Mikolov et al., 2013a), resulting in a 300-dimensional representation per word. For any action or object with more than one word, the word representations are averaged. The object scores in a video are obtained following the protocol of Mettes et al. (2021), where two frames per second are sampled, each fed to the pretrained ImageNet model, and with the object probabilities averaged over the sampled frames. For the clustering of the video embeddings in Definition 2, we use k-means clustering along with `2- normalization, akin to Banerjee et al. (2005). For the optimal transport, we employ the Lagrangian approach of Bonneel et al. (2011) as implemented in (Flamary et al., 2021). Lastly for spatiotemporal localization, we start from the tubes made available by Mettes et al. (2021). To each tube, we add the corresponding video-level action scores from our approach to improve the ranking of the action tubes over the entire dataset. All code will be made publicly available.

heading: 4 EXPERIMENTAL RESULTS
text: We focus on three experiments: (i) evaluations on universal action models, (ii) evaluations on universal object models along with the fusion between both, and (iii) state-of-the-art comparison for zero-shot action recognition and zero-shot spatio-temporal action localization.

heading: 4.1 EVALUATING TRANSDUCTIVE UNIVERSAL TRANSPORT FROM SEEN ACTIONS
text: Setup. For the first experiment, we evaluate on UCF-101 using all 101 actions for classification. We investigate the two variables that come with our approach in the context of universal action models, namely the granularity of the cluster aggregation over all test videos and the interpolation ratio between the original and target embeddings of the unseen actions.
Results. The results for six cluster sizes and three interpolation ratios are shown in Figure 2a. An interpolation rate of 1 denotes only using the original action embeddings and 0 only the target embeddings. With the original embeddings, we obtain an accuracy of 39.2%. Using the target embeddings from the universal transport (λ = 0) directly boosts the classification accuracy when using many clusters, while results degrade for few clusters. The best performance is obtained by positioning the unseen actions halfway along the geodesic between the original and target embeddings. With 1,000 clusters the accuracy becomes 42.4%, bcompared to 40.1% without interpolation.
Analysis. An explanation for our improvements is shown in Figure 2b. We show the distributions of selected actions across all interpolation ratios when using 1,000 clusters. With the original unseen action embeddings, this distribution is highly uneven, with 23% (!) of the actions never being selected, naturally leading to zero accuracy for these actions. With our approach, the distributions become more uniform, highlighting the deduced bias present in our approach. We conclude that transductive universal transport improves zero-shot action recognition for universal action models. Throughout the rest of the experiments, we use 1,000 clusters with an interpolation ratio of 0.5.

heading: 4.2 EVALUATING TRANSDUCTIVE UNIVERSAL TRANSPORT FROM OBJECTS
text: Setup. Next, we investigate our approach on universal object models. We again use UCF-101 with all 101 actions for evaluation, with the interpolation ratio fixed to 0.5. We evaluate three threshold levels that come with the definition of objects as discrete measure, along with the universal object model itself and a vanilla uniformly-weighted optimal transport using all objects as baselines.
Results. In Table 1 on the left, we show the zero-shot action results for our approach when maintaining the top 2,500, 1,000, and 500 objects according to their transductive maximum probability over all test videos. We first find that using a baseline optimal transport approach akin to the setup for seen actions provides only a marginal boost from 29.2% to 30.1%. In contrast, using the proposed weights for the unseen actions and objects, combined with a filtering of objects never present in a
test video, provides a boost to 31.6% with the top 1,000 objects. We find that as long as the threshold is not set too strictly (e.g., keeping 1,000 objects or more) provides stable zero-shot results.
In Table 1 on the right, we show that the proposed weighting matters. For this Table we keep the top 1,000 objects and investigate all four combinations of uniform and proposed weighting. With uniform weights for both unseen actions and objects, the results are similar to the baseline objectbased setup, while the results improve when incorporating either or both of the weight vectors to the proposed setup. We conclude that also in the universal object-based model for zero-shot action recognition, transductive universal transport is beneficial.
Fusion. With our approach shown to be effective for both universal action and object approaches separately, we investigate the potential of fusing both perspectives. In Figure 3, we show the effect of fusing both approaches with and without transductive universal transport on UCF-101 and HMDB51. On UCF-101, we find that combining both approaches has a clear positive effect. Our results improve from 42.4% (universal actions) and 31.5% (universal objects) to 47.9% when balancing both equally. When setting the fusion proportion to 0.3, the results can even be further improved to 48.9. Due to the zero-shot nature of our approach, we stick to an a priori equal balance between both setups. Averaged over all fusion proportions, our approach provides a boost of 3.0 percent point compared to the baseline fusion.
In Figure 3, we also show the fusion results for HMDB51. Our approach improves the accuracy from 24.9% to 28.1% when using universal action models only. The results are further improved to 29.4% with an equal fusion of our universal object representations. We conclude that universal seen actions and objects are complementary for zero-shot action recognition and their fusion benefits from transductive universal transport.
UCF-101 HMDB51 20 50 101 10 25 51
UCF Sports J-HMDB 0.1 0.2 0.3 0.4 0.5 0.1 0.2 0.3 0.4 0.5

heading: 4.3 STATE-OF-THE-ART COMPARISON
text: Action recognition. In Table 2, we compare our results on UCF-101 and HMDB51 to the state-of-the-art in zero-shot action recognition. Similar to other universal approaches, in the scenarios with random sub-selection of the test actions (20 and 50 for UCF-101, 10 and 25 for HMDB51) we do not use the remaining actions and their videos for network training. On both datasets, the current state-of-the-art is given by Brattoli et al. (2020). On UCF101 with 101 test actions, they obtain an accuracy of 39.8%. With our transductive universal transport applied and fused over universal action and object models, the results improve to 47.9%. We also improve over the recent universal object model of Mettes et al. (2021)
and the action signature approach of Kim et al. (2021) across all numbers of test actions. On HMDB51, we improve the state-of-the-art with 2.5 p.p. (51 actions), 3.3 p.p. (25 actions), and 7.0 p.p. (10 actions). We conclude that transductive universal transport is a viable solution for universal action and object models in zeroshot action recognition. Lastly, we also include a one-to-one comparison to universal action and object models in Table 3 for fair comparisons.
Spatio-temporal action localization. Lastly, we showcase the potential of our approach for improving zero-shot spatio-temporal action localization as first introduced by Jain et al. (2015). Since our approach operates over entire videos, we start from the spatio-temporal tubes made publicly available by Mettes et al. (2021). For each tube, we simply add the score for each action from the entire video as given by our approach. In Table 4, we report the AUC scores for UCF Sports and J-HMDB. Across datasets and overlap thresholds, we find that the global scores from our approach boosts spatio-temporal localization. This is because the scores of our approach help to distinguish and rank tubes from different videos as they encode contextual information.

heading: 5 RELATED WORK
text: Zero-shot action recognition refers to the task of assigning an action label to a test video given a pool of action labels not observed during training. A common approach is learn and transfer a shared representation on seen actions with training examples to be able to perform inference on unseen actions. A well-known shared space is given by attributes (Liu et al., 2011; Zhang et al., 2015; Gan et al., 2016b). By projecting test videos to an attribute space, inference is possible through a neighbour search with unseen actions manually defined in the same space. Since attributes require manual annotations for every action, more recent works prefer to use word embeddings (Gan et al., 2016a;c; Li et al., 2016; Bishay et al., 2019) or action hierarchies (Long et al., 2020) to provide a share space for seen and unseen actions.
State-of-the-art approaches to zero-shot action recognition take a universal learning perspective towards the use of semantic word embeddings as the shared space for knowledge transfer. Rather than relying on a small set of seen actions from the same dataset, large-scale models are trained on hundreds or thousands of seen categories to learn a direct mapping from videos to the shared space. The first universal learning direction relies on large-scale actions with training videos. Zhu et al. (2018) were the first to propose a large-scale universal action perspective by learning a video network on 200 actions from ActivityNet (Caba Heilbron et al., 2015). Recently, Brattoli et al. (2020) have obtained the best performance in zero-shot action recognition by scaling this perspective to 664 actions from Kinetics (Carreira & Zisserman, 2017). Due to the large amount of seen actions, care needs to be taken to avoid (near) duplicates between the seen and unseen actions, as also noted by Roitberg et al. (2018).
Next to universal action models, several works have shown competitive results by taking a universal object perspective. Large-scale networks are trained in the image domain on thousands of object labels (Jain et al., 2015; Wu et al., 2016; Mettes & Snoek, 2017; Liu et al., 2019; Mettes et al., 2021). Once trained any action can be inferred based on the object likelihoods in test video and the semantic similarities between objects and unseen actions. Regardless of the universal perspective, the current state-of-the-art shares the same fate: zero-shot inference results in a long-tailed distribution over selected actions. Many unseen actions are simply never selected during persistent biases. We tackle this problem with the transductive universal transport over universally trained models. This paper introduces a new way of dealing with biases in universal models, hence the focus on the video domain over the image domain, where universal models are not state-of-the-art.
Within zero-shot action recognition, a transductive view is commonly applied, as noted in (Estevam et al., 2021). Rohrbach et al. (2013) provide a foundation for transduction in zero-shot context by exploiting the inter-sample similarity over the test set. In similar spirit, several works have proposed transductive extensions for zero-shot action recognition (Fu et al., 2014; Kodirov et al., 2015; Alexiou et al., 2016; Xu et al., 2017; Gao et al., 2019; Mandal et al., 2019). While transductive learning no longer operates on individual test videos, many applications require evaluating entire video batches/collections simultaneously. Different from existing approaches, we use the distribution over all test videos to improve the unseen action embeddings in the shared output space of universal models by building on optimal transport. Moreover, our approach can naturally switch between inductive and transductive settings, as only the positioning of action embeddings are updated.

heading: 6 CONCLUSIONS
text: In this work, we investigate a persistent limitation in current universal learning models for zeroshot action recognition, namely selection biases in the assignment of unseen actions to test videos. We introduce transductive universal transport to alleviate this bias. Our approach consists of three stages: (i) finding an optimal transport mapping from unseen action embeddings to the mapped test videos (in universal action models) or object embeddings (in universal object models), (ii) calculating a target embedding for each unseen action as a weighted Fréchet mean with the weights given by the transport couplings, and (iii) re-positioning the unseen actions along the geodesic spanned by the original and target embeddings. Empirical evaluation on four datasets shows the effectiveness of our approach for debiasing action assignments and improving zero-shot recognition as a result, outperforming existing approaches for zero-shot action classification and zero-shot spatio-temporal action localization. Our approach is general and can be used to improve any universal model.
ETHICS AND REPRODUCIBILITY STATEMENT
Action recognition has a broad range of applications, for example in autonomous driving and robotics. While such applications hopefully have a positive societal effect, the research can also potentially be used for applications such as surveillance and this should be taken into account. To ensure reproducibility, we start from off-the-shelf models as provided by current universal learning models. In the paper, we have outlined the hyperparameters of our approach and we will release the code to reproduce the experiments.

Review summaries:
The paper presents a zero-shot action recognition framework by learning an universal mapping from video to semantic space. The unseen action embeddings are re-positioned through leveraging the distribution of unlabelled test set. The universal mappings from unseen action to test videos are first defined and the target embeddings are treated as weighted Frechet means. The unseen action embeddings are re-positioned as a semantic regularization. The results on UCF101 and HMDB-51, UCF Sports and J-HMDB validates the proposed method.

The paper targets transductive zero-shot action recognition.  To alleviate models biased to seen categories, the authors propose to re-position unseen action embedding through transduction. There are three steps in the proposed method: first, finding an optimal mapping from unseen actions to the mapped video in the shared hyperspherical space. Second, defining target embeddings as weighted Frechet means with the weight given by the transport couplings. Third, re-positioning unseen action embeddings along the geodesic between the original and target. The zero-shot classification performance of the proposed method is tested on the UCF-101 and HMDB datasets. The zero-shot spatio-temporal localization performance is tested on the UCF Sport and J-HMDB datasets.  

This work tries to address the problem of zero-shot action recognition. Particularly, the paper aims at preventing the case that many unseen action categories in the target domain are simply never being selected during inference. Using the distribution of the unlabelled test set, the embeddings of unseen actions in the target domain are reweighted and repositioned along the geodesic such that they are better aligned with embeddings of training actions in the source domain. In experiments, Empirically, the proposed method has been evaluated on  benchmark datasets for tasks zero-shot action classification and spatio-temporal action localization.

This work introduces transductive universal transport for zero-shot action recognition, where no training examples for unseen classes are available. To address the biases of prior approaches towards seen classes during inference, this paper re-positions unseen action embeddings through transduction by using the distribution of the unlabelled test set. Experimental results on several action recognition datasets demonstrate the effectiveness of the proposed method.