Transductive Universal Transport for Zero-Shot Action Recognition
Authors: Pascal Mettes
Venue: ICLR 2022 Submitted (2022)
Keywords: zero-shot learning
action recognition
action localization
optimal transport

Abstract: This work addresses the problem of recognizing action categories in videos for which no training examples are available. The current state-of-the-art enables such a zero-shot recognition by learning universal mappings from videos to a shared semantic space, either trained on large-scale seen actions or on objects. While effective, universal action and object models are biased to their seen categories. Such biases are further amplified due to biases between seen and unseen categories in the semantic space. The amplified biases result in many unseen action categories simply never being selected during inference, hampering zero-shot progress. We seeks to address this limitation and introduce transductive universal transport for zero-shot action recognition. Our proposal is to re-position unseen action embeddings through transduction, \ie by using the distribution of the unlabelled test set. For universal action models, we first find an optimal mapping from unseen actions to the mapped test videos in the shared hyperspherical space. We then define target embeddings as weighted Fr\'echet means, with the weights given by the transport couplings. Finally, we re-position unseen action embeddings along the geodesic between the original and target, as a form of semantic regularization. For universal object models, we outline a weighted transport variant from unseen action embeddings to object embeddings directly. Empirically, we show that our approach directly boosts universal action and object models, resulting in state-of-the-art performance for zero-shot classification and spatio-temporal localization.

Under review as a conference paper at ICLR 2022
TRANSDUCTIVE UNIVERSAL TRANSPORT FOR ZERO-SHOT ACTION RECOGNITION
Anonymous authors Paper under double-bling review
ABSTRACT
This work addresses the problem of recognizing action categories in videos for which no training examples are available. The current state-of-the-art enables such zero-shot recognition by learning universal mappings from videos shared semantic space, either trained on large-scale seen actions or on objects. While ef- fective, we find that universal action and object models are biased to their seen egories. Such biases are further amplified due to biases between seen and unseen categories in the semantic space. The amplified biases result in many unseen ac- tion categories simply never being selected during inference, hampering zero-shot progress. We seeks to address this limitation and introduce transductive universal transport for zero-shot action recognition. Our proposal is to re- --pssiiion unseen action embeddings through transduction, i.e., by using the distribution of the unla- belled test set. For universal action models, we first find optimal mapping from unseen actions to the mapped test videos in the shared hyperspherical space, We then propose new target embeddings as weighted Fréchet means, with the weights given by the transport couplings. Finally, we introduce a reeppsitionnin of unseen action embeddings along the geodesic between the original and target, of semantic regularization. For universal object models, we outline a weighted transport variant from unseen action embeddings object embeddings directly. Empirically, we show that our approach directly boosts universal action and ob- ject models, resulting in state-of-the-art performance for zero-shot classification and spatio-temporal localization.
form
INTRODUCTION
This paper investigates the problem of inferring actions in videos for which no training examples were provided. Akin to advances the image domain (He et al. 2016 Huang et al. 2017 Yuan et al. 202I), deeper and more expressive network architectures have in recent years been developed to continually improve the recognition of actions in videos (Carreira & Zisserman 2017 Feichten- hofer al. 2019 Arnab et al. 202T). To meet the corresponding demand for data, multiple new datasets - each spanning many hundreds of actions - have been created, e.g., (Carreira & Zisserman, 2017 Monfort al., 2019 Damen al., (0020). While such datasets increase the coverage of the action space, we seek to recognize actions even when no examples are available during training.
In zero-shot action recognition, one line of work has found success by mirroring progress in its image-based counterpart, for example by using attributes (Liu et al., 2011 Gan et al., 2016b) or feature synthesis (Mishra et al., (200)) to transfer knowledge from seen unseen actions. Recently, state-of-ther -ftte-- results have been achieved by taking a universal learning perspective, where large- scale models are trained to map input videos to a shared semantic space occupied by both seen and unseen categories. In the first universal perspective, large-scale networks train mapping from videos to a semantic space on hundreds of seen actions from egg., ActivityNet (Zhu et al., 2018) or Kinetics (Brattoli et al.) 2020). For target dataset with unseen actions, zero-shot inference is di- rectly possible through a nearest neighbour search between the video mappings and the embeddings of unseen actions. In the second perspective, networks trained thousands of objects (Jain jet al. 20133] |Mttes et al. (02T). where inference becomes function of both the object likelihoods and the semantic relation between objects and actions.

---

Under review as a conference paper at ICLR 2022
Unseen action embedding
Test video
f Universal transport target
Figure/ 1: Overview of transductive universal transport for universal action models. First, we find an optimal mapping from the discrete measure representing unseen actions the discrete measure representing the mapped test videos. Second, we define the target embedding for each unseen action as Fréchet means over the test videos, weighted by their optimal coupling. Third, we re-position unseen action embeddings along the geodesic spanned by the original and target embeddings.
While effective, the first finding of this work is that both perspectives share a common limitation: they are strongly biased to subsets of the unseen actions. Compounding biases from videos to seen categories and from seen to unseen categories result in a mismatch between the embeddings of unseen actions and mapping of the videos covering the same actions. A significant part of the actions are as a result simply never selected, disrupting progress zero-shot action recognition. We seek to address this limitation.
In this work, we introduce transductive universal transport for zero-shot action recognition with universal models. The idea is to re-position the semantic embeddings of unseen actions by using information about the distribution the entire test set. For universal action models, we first consider the optimal transport mapping between the semantic embeddings of unseen actions and all embedded videos. We then propose a target function for each unseen action as a weighted Fréchet mean given finally propose to
the optimal transport couplings. To avoid loosing semantic interpretation, re-position unseen actions along the geodesic spanned by the original and target embeddings. We view this form of semantic regularization imposed the target embeddings. For universal object models. we perform the same approach with two differences: we map the unseen actions to object embeddings directly and we add transport weights both the unseen actions and the objects, such that we focus on actions far away from objects and on objects that are more likely to occur in the test videos. Experiments on four datasets validate our approach, with state-of-the-are results for zero-shot action recognition and spatio-temporal action localization.
2 RE-POSITIONING ACTION EMBEDDINGS WITH UNIVERSAL TRANSPORT
For the problem of zero-shot action recognition, we are given set of test videos Va and set of labels L, denoting actions which have not been observed during training. We seek to assign label E L., to each test video. that end, we start from two state-of-the-art universal learning directions in zero-shot action recognition, namely by transferring knowledge from large-scale seen actions and from objects. For both directions, we seek to overcome the long-tailed distributions in label assignment of unseen actions during zero-shot inference. Below, first introduce transductive universal transport for the transfer from seen to unseen actions, see Figure Second, we extend our approach for re-positioning unseen action embeddings based on objects.
2.1 TRANSDUCTIVE UNIVERSAL TRANSPORT FOR UNIVERSAL ACTION MODELS
Inferring unseen actions by transferring knowledge from seen actions requires two transformation functions: a function w that maps a label to semantic embedding space and a function ó that maps video to the same embedding space. The mapping function is learned set of training videos
2

---

Under review as a conference paper at ICLR 2022
V& with seen action labels L. with its loss given
(o(v),<((.)) VEV. |[o())] [|((1)]
c. Σ
(1)
with 1, E L. the action label for video v. The function w is given by a pre-trained word embedding model (Mikolov et al. 2013b), where embeddings commonly fz-normalized and optimized with the cosine distance. Due to the shared semantic nature of the embedding space, any test video that projected to the embedding space using 6 can also be assigned to a label from L,, through nearest neighbour search with all unseen labels that have been mapped using w. In this work, we argue that since & is trained on actions not used during inference, the mapped video embeddings and unseen action embeddings are not aligned. We propose to improve zero-shot action recognition by re-positioning unseen action embeddings with transductive information.
In order to better position unseen action embeddings, we first redefine the original seen and unseen action embeddings as discrete measures so that they can be used in the context of optimal transport:
Definition 1 (Actions discrete measures). The sets of seen and unseen action labels are given
Σ wu,,) L.E.,
Σ wwttwwlt))
LEEL.
where wi, E W. and Wt, E w. denote the set of weights for the action, such that W, E/e,a and
W, E S/pcul are both on the probubility simplex and bg denotes the Dirac at position
In similar fashion the definition of the mapped test videos is given as:
Definition 2 (Videos as discrete measure). The of mapped test videos is given as:
Mu =
wedes
Σ ((v),
(3)
[a(Vaic)
(EC*(VA)
VVa(V
with we We E Eg akin Definition 1, where Ck denotes k-component cluster aggregation over the set of videos, and where c(Vu;c) denotes the set of videos assigned to cluster c.
With the above definitions, seen and unseen actions are interpreted as discrete measures centered around their word embeddings. For the test videos, we define the corresponding discrete measure as a clustering over the mapped video outputs. This is make the measure itself more robust to outliers and to increase the focus towards high-density regions. The seen actions are only used to train the universal network <()); the unseen actions and test videos are central our approach.
With the unseen action labels and videos defined as discrete measures in the same space, we first seek to find an optimal transport mapping between the two measures. Since we operate on the entire distribution of test videos rather than performing inference for each video independently, we view this as a transductive form of optimal transport, see Figure (left). Although the action embeddings discrete entities, find that it is better to view the transport as instance of the relaxed Hongg-Kantoroovich problem, which allows for mass splitting from the unseen action measure to the aggregated video measure. Let C denote the cost matrix, with Ctj defined as the cosine distance between action i and video cluster j. Then the transductive optimal transport form pa to H.y is given as (Peyré et al. 2019):
Lo(Wu, w.) min PEU(W...w.) (C.P) C ij
(4)
where U denotes the admissible couplings, given as:
U(w., we) = P (Culk P1: = = Wo
(5)
P1 = E Pl|c. = 3 Piy)s
which is solved using the Lagrangian approach of Bonneel et al. (2011). This results in coupling matrix P, where P.j denotes the amount mass action i that moves to video cluster j. When working with universal action models, set the weights of both measures uniformly.
3

---

Under review as a conference paper at ICLR 2022
Given the optimal transport mapping, we propose condense the corresponding coupling into single target embedding per unseen action. Since since the semantic space on which we operate is hypersphere, define the target embedding as the weighted Fréchet mean (Lou et al. 2020
Miolane et al. 2020) using normalized coupling values:
k Σ
Jargel (1:) = arg min
Σ Pijd(c,,s)3 1=1
Pij 1, 3=1
(6)
eE3M-1
with d the cosine similarity, see Figure T (middle). The target provides a new location in embedding space for each unseen action label, guided by the distribution of mapped and clustered test videos. While we can directly use the new embeddings for inference, we want to avoid big changes in embedding space since that relates with losing the original semantic interpretation of the action. We therefore dictate that the final embedding of each unseen action is positioned along the geodesic spanned by the original and target embeddings, modelled through spherical interpolation:
w (1) sin (() + sin((1 (S(y (arget())
sin[As]
cos Ω (w) larget (I)),
sin
where À denotes the balance between maintaining the original embedding and moving towards the transductive target embedding. this manner, we move each unseen action towards its proposed larget embedding, with the interpolation acting regularization towards the original embedding. Zero-shot action recognition is performed by computing the cosine similarity between the projected video and each unseen action label I as
(o(v),
Saction (1|u) [ ((](())|
after which the action label with the highest similarity selected. Figure | (right) provides an illustration of the final re-positioning, where we move the original unseen action embeddings along the greater arc of the hypersphere towards the optimal target embeddings.
2.2 TRANSDUCTIVE UNIVERSAL TRANSPORT FOR UNIVERSAL OBJECT MODELS
Universal object models form a competitive and similarly biased alternative for zero-shot action recognition. To tackle these biases, we also redefine objects discrete measures to make them suitable the context of optimal transport:
Definition 3 (Objects as a discrete measure). The sets of object labels are given as:
Ho = Σ WoDw(o)
O, (
max p(olu) 2 27
vEV.
oEO,
with Wo Og and where p() denotes the likelihood of object occurring in video v.
Different from both the action and video measure definitions, the discrete measure for objects is based subset O, E O, Le., define degenerate distribution over the objects. The subset determined by again taking a transductive view; we exclude any object which does not have a likelihood over low threshold 7 in any test video. The idea behind this definition is to avoid bias in the optimal transport to objects which do not actually occur in videos.
Akin to approach for universal action models, compute a coupling between the unseen ac- tion measure and the object measure, as given by Equations 4 and S Differently however, we set non-uniform weights for both the actions and objects. The objects are weighted according to their transductive maximum score, w, = max,ev, p(o|u)/ Z,, with 2, a normalization constant over all objects in O.. The unseen action weights are given as Wa (1 - ((max,go, (w(a), w(o)) /2) + 0.5))/ Z., with 2, a normalization constant over all actions. The intuition behind the object weights to focus the attention of the transductive universal transport on objects with a higher visual likeli- hood. The weights for the unseen actions are given as the inverse over the maximum word embed- ding similarity with respect to the objects, under the notion that actions without obvious relations to objects should have more prominent spot in the transport coupling. With the optimal transport computed between unseen actions and objects, action embeddings are again interpolated following Equations and 7 with the updated emhedding for action label now denoted as (+))..
4

---

Under review as a conference paper at ICLR 2022
For the final zero-shot action inference from objects, we follow the same setup current object- based approaches, where the score for each action label in video e is determined based on the top
relevant objects for that action (Jin al. 2015 Mettes et al. 2021:
(w)()) w (t)
Sobjoct (1lu)
Σ p(olv) dEO,
(10)
[[(()]| (1)l
with Or the set of most semantically similar objects for action label Finally, the transductive
action-based and object-based scores from respectively Equations g and [0 can also be fused as
Stusion (1|) = Saction (4|1) + (1 (1-e) Sobjecr(
EXPERIMENTAL SETUP
3.1 DATASETS
Source datasets. We employ two source datasets for universal representation learning, namely Kinetics-700 for actions and ImageNet for objects. For Kinetics-700, we follow Brattoli et al. (0200 and use a subset with 664 action categories to avoid any overlap with action categories in datasets used for zero-shot action recognition. For ImageNet, we follow Mettes et al.) (202T) and use the reorganized variant containing 12,988 object categories (Mettes et al. 2020).
Target datasets. The classification evaluation is performed the two datasets used most often in sero-shot action recogmition: UCF-101 and HMDB51. The UCF-101 dataset consists of 13,320 videos covering 101 action categories. Next to 101-way zero-shot evaluation, we also evaluate on settings with 20 and 50 test actions. For these settings, we rerun our approach on 10 runs with randomly selected actions and we report the mean and standard deviation We note that in the 20- and 50-way zero-shot recognition, we do not use the other actions for training, they simply not used in our approach. The HMDB51 dataset consists of 6, 6,766 videos covering 51 action categories. Next to 51-way evaluation, we also investigate 10- and 25-way zero-shot recognition.
the runs.
We also investigate the potential of our approach for zero-shot spatio-temporal action localization on UCF Sports and J-HMDB. UCF Sports consists of 150 videos with 10 actions and J-HMDB consists of 928 videos with 21 actions. For the evaluation, we follow Jain et al. (2015) and report results with the AUC metric across five overlap thresholds.
3.2 IMPLEMENTATION DETAILS
For the universal action network é, we employ the R(2+1)D network (Tran et al. 2018) as given by Brattoli et al. (2020), which is pre-trained on the 664 action categories from Kinetics. For each video, obtain its corresponding video embedding by randomly selecting 16 frame shot and passing the shot through Φ. For a fair comparison to the state-of-the-ae by Brattoli et al. (2020), we also the same word embedding w, namely word2vee model (Mikolov et al. (013a). resulting in a 300-dimensional representation per word. For any action or object with more than one word, the word representations are averaged. The object scores in a video obtained following the protocol of [Mettes et al. (202T), where two frames per second are sampled, each fed to the pre- trained ImageNet model, and with the object probabilities averaged over the sampled frames. For the clustering of the video embeddings in Definition 2 we use k-means clustering along with 4-- normalization, akin to Banerjee et al. (2005). For the optimal transport, we employ the Lagrangian approach of [Bonneel et al. (2011) implemented in (Flamary et al.] (021). Lastly for spatio- temporal localization, we start from the tubes made available by Mettes et al. (20TT) To each tube, we add the corresponding video-level action scores from our approach to improve the ranking of the action tubes over the entire dataset. All code will be made publicly available.
EXPERIMENTAL RESULTS
We focus on three experiments: (i) evaluations universal action models, (ii) evaluations uni- versal object models along with the fusion between both, and (iii) state-of-the-ar comparison for zero-shot action recognition and zero-shot spatio-temporal action localization.
5

---

Under review as a conference paper at ICLR 2022
50
0.06
Interpolation ratio
1.00
0.5
0.05
(4.36)
45
0.00 (4.43)
0.04
40
Frequency
35
0.01
25
0.00
101
500
1000
4000
13320
20
6.
80
100
Number clusters
Action index
(a) Cluster ablation.
(b) Distribution analysis.
Figure 2: Evaluating transductive universal transport from seen actions UCF-101. Left: The effect of the number of clusters and the interpolation ratio on the recognition performance, showing the positive effect of re-positioning unseen actions and the semantic regularization given by the spherical interpolation. Right: An intuition behind the obtained results. Using the original unseen action embeddings results large biases during zero-shot inference, here using 1,000 clusters. With out approach, this imbalance is reduced, as indicated by the more even action distributions in the plot and the corresponding higher entropy scores brackets in the legend.
4.1 EVALUATING TRANSDUCTIVE UNIVERSAL TRANSPORT FROM SEEN ACTIONS
Setup. For the first experiment, we evaluate on UCF-101 using all 101 actions for elassification. We investigate the two variables that come with our approach in the context of universal action models, namely the granularity the cluster aggregation over all test videos and the interpolation ratio between the original and target embeddings of the unseen actions.
The results for six cluster sizes and three interpolation ratios are shown Figure 2a An interpolation rate of 1 denotes only using the original action embeddings and 0 only the target embeddings. With the original embeddings, we obtain an accuracy of 39.2%. Using the target em- beddings from the universal transport (A 0) directly boosts the classification accuracy when using many clusters, while results degrade for few clusters. The best performance is obtained by posi- tioning the unseen actions halfway along the geodesic between the original and target embeddings. With 1,000 clusters the accuracy becomes 42.4%, bcompared to 40. % without interpolation.
Results.
Analysis. An explanation for our improvements is shown in Figure 2b We show the distributions of selected actions across all interpolation ratios when using 1,000 clusters. With the original unseen action embeddings, this distribution is highly uneven, with 23% (!) of the actions never being selected, naturally leading to zero accuracy for these actions. With our approach, the distributions become more uniform, highlighting the deduced bias present in our approach. We conclude that transductive universal transport improves zero-shot action recognition for universal action models. Throughout the rest of the experiments, we use 1,000 clusters with interpolation ratio of 0.5.
4.2 EVALUATING TRANSDUCTIVE UNIVERSAL TRANSPORT FROM OBJECTS
Setup. Next, we investigate approach universal object models. We again use UCF-101 with all 101 actions for evaluation, with the interpolation ratio fixed to 0.5. We evaluate three threshold levels that come with the definition of objects discrete measure, along with the universal object model itself and a vanilla uniformly-weighted optimal transport using objects as baselines.
Results. In Table D the left, we show the zero-shot action results for our approach when maintain- ing the top 2,500, 1,000, and 500 objects according to their transductive maximum probability over all test videos. We first find that using baseline optimal transport approach akin to the setup for seen actions provides only a marginal boost from 29.2% to 30.1% contrast, using the proposed weights for the unseen actions and objects, combined with filtering of objects never present in

---

Under review as a conference paper at ICLR 2022
UCF-101
UCF-101
O
Accuracy
Weighting
Accuracy
Baseline
29.9
Actions
Objects
Baseline transport
12,988
30.1
uniform
uniform
29.8
Proposed transport
2,500
31.4
inverse
uniform
30,4
Proposed transport
1,000
31.6
uniform
transductive
30.6
Proposed transport
500
30.2
inverse
transductive
31.6
Table I: Evaluating transductive universal transport from objects on UCF-101. Our proposed approach also enhances universal object-based approaches for zero-shot action recognition, due to both object filtering (left) and the proposed weighting of unseen actions and objects (right).
50.0
30.0
47.5
27.5
42.5
Accuracy
22.5
37.5
17.5
32.5
w/o transductciie universal transport transductive universal transport
12.5
w/o transductive universal transport transduttivee universal transport
30.0
0.25
0.5
0.75
0.25
0.5
0.75
Fusion proportion
Fusion proportion
(a) UCF-101.
(b) HMDB51.
Figure 3: Fusing action and object information for zero-shot recognition on UCF-101 and HMDB51. Combining universal action and object information benefits zero-shot recognition, with transductive universal transport preferred regardless of the fusion proportion hetween both sources.
test video. provides boost to 31.6% with the top 1,000 objects. We find that as long the threshold not set too strictly (e.g., keeping 1,000 objects or more) provides stable zero-shot results.
In Table on the right, show that the proposed weighting matters. For this Table keep the top 1,000 objects and investigate all four combinations of uniform and proposed weighting. With uniform weights for both unseen actions and objects, the results similar to the baseline object- based setup, while the results improve when incorporating either or both of the weight vectors to the proposed setup. We conclude that also in the universal object-based model for zero-shot action recognition, transductive universal transport is beneficial.
Fusion. With our approach shown to be effective for both universal action and object approaches separately, we investigate the potential of fusing both perspectives. In Figure 3 we show the ef- feet of fusing both approaches with and without transductive universal transport on UCF-101 and HMDB51. On UCF-101, we find that combining both approaches has a clear positive effect. Our results improve from 42.4% (universal actions) and 31.5% (universal objects) to 47.9% when bal- ancing both equally. When setting the fusion proportion to 0.3, the results can even be further improved to 48.9. Due the zero-shot nature of our approach, we stick to an a priori equal balance between both setups. Averaged over all fusion proportions, our approach provides boost of 3.0 percent point compared to the baseline fusion.
In Figure 3 we also show the fusion results for HMDB51. Our approach improves the accuracy from 24.9% to 28.1% when using universal action models only. The results are further improved to 29.4% with equal fusion of our universal object representations. We conclude that universal seen actions and objects are complementary for zero-shot action recognition and their fusion benefits from transductive universal transport.

---

Under review as a conference paper at ICLR 2022
UCF-101
HMDBS1
20
50
101
10
25
51
Hahn et al.
(2019)
36.5
22.1
40.1
23.5
Bishay et al
2019
42.7 + 5.4
23.2 =: 2.9
19.5 = 4.2
Mishra et
2020
26.1 3.0
21.3 = 3.2
Mandal ei al.
(2019)
38.3 3.0
30.2 2.7
Mettes Snock
(2017
51.2 + 5.0
40.4 1.0
32.8
Zhu et al. (0018)
53.8
42.5 1
34.2
Mettes al. (2021
61.1
47.3 +
36.3
Kim et al. (2021)
48.9 = 5.8
Brattoli et al. 2020)
49.2
39.8
32.7
26.9
This paper
64.1 10.5
54.4
47.9
47.1 8.8
36.0
5.6
29.4
Table 2: State-of-the-art comparison on UCF-101 and HMDB51 for different numbers test actions. Across both datasets and different numbers of test actions, our approach obtains the highest zero-shot action classification performance.
UCF Sports
J-HMDB
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Jain et al. 2015
38.8
23.2
16.2
9.9
7.2
Mettes Snock 2207
43.5
39.3
37.1
35.7
31.1
34.6
33.3
30.5
26.8
23.0
Mettes el al. (2021)
47.3
43.0
40.7
37.9
33.1
37.3
37.1
33.9
31.0
26.7
This paper
51.1
47.8
45.7
41.2
33.5
44.2
43.5
40.1
35.5
30.8
Table State-of-the-art comparison on UCF Sports and J-HMDB for five overlap thresholds. Across datasets and overlap thresholds. we obtain the highest scores, highlighting the effectiveness of our approach in the context of zero-shot apatio-tempora localization.
4.3 OFATE-OF-THE-ARE COMPARISON
Action recognition. In Table we compare our results on UCF-101 and HMDB51 to the state-of-the-art in zero-shot action recogmition. Similar to other universal approaches, in the scenarios with random sub-selection of the test actions (20 and 50 for UCF-101, 10 and 25 for HMDB51) we do not use the remaining actions and their videos for network training- On both datasets, the current state-of-the-art is given by Brattoli et al.) (2020). On UCF- 101 with 101 test actions, they obtain an accuracy of 39.8%. With our transductive univer- sal transport applied and fused over universal action and object models, the results improve to 47.9%. We also improve over the recent universal object model of Mettes et al. (202T) and the action signature approach Kim et al. (202T) across all numbers of test actions. On HMDB51, improve the state-of-the-art with 2.5 p.p. (51 actions), 3.3 p.p. (25 actions), and 7.0 p.p. (10 actions). We conclude that trans- ductive universal transport is a viable solution for universal action and object models in zero- shot action recognition. Lastly, we also include one-to-one comparison to universal action and object models in Table 3 for fair comparisons.
Baseline
+ This paper
Universal action model
39.2
42.4
Universal object model 29.9
31.6
pre-trainsd model and word omboddings from (Brattoli all 2020) pre-trained object model action-scoring given by (2021), word embeddings from (Bratiohi et al. 2020)
Table 3:
One-to-one overview of our positive im- pact on universal action and object models.
Spatio-temporal action localization. Lastly, we showcase the potential our approach for im- proving zero-shot spatio-temporal action localization as first introduced by Jajn et al. (2015). Since our approach operates over entire videos, we start from the spatio-temporal tubes made publicly available by Mettes et al. (2021). For each tube, we simply add the score for each action from the entire video as given by our approach. Table 4 we report the AUC scores for UCF Sports and find that the global scores from approach boosts spatio-temporal localization. This is because the scores of our approach help to distinguish and rank tubes from different videos they encode contextual information.
J-HMDB. Across datasets and overlap thresholds,
8

---

Under review as a conference paper at ICLR 2022
5 RELATED WORK
Zero-shot action recognition refers to the task of assigning an action label to test video given a pool of action labels not observed during training. A common approach is learn and transfer a shared representation on seen actions with training examples to be able to perform inference on unseen actions. A well-known shared space is given by attributes (Liu et al.) 2011 Zhang et al., 2015 Gan et al. 20T6b). By projecting test videos to an attribute space, inference is possible through a neighbour search with unseen actions manually defined in the same space. Since attributes require manual annotations for every action, more recent works prefer to use word embeddings (Gan et al.
2016a Li al. 2016 Bishay et al. 2019) or action hierarchies (Long et al. 2020 to provide share space for seen and unseen actions.
ffate-of-the-ar approaches to zero-shot action recognition take universal learning perspective to- wards the use of semantic word embeddings as the shared space for knowledge transfer. Rather than relying small set of seen actions from the same dataset, large-scale models trained on hundreds or thousands of seen categories to learn a direct mapping from videos to the shared space. The first universal learning direction relies on large-scale actions with training videos. [Zhu et al. 2018) were the first to propose large-scale universal action perspective by learning a video net- work on 200 actions from ActivityNet (Caba Heilbron et al. 2015). Recently, [Bratolli et al. (2020) have obtained the best performance in zero-shot action recognition by scaling this perspective to 664 actions from Kinetics (Carreira & Zisserman 2017). Due to the large amount of seen actions, care needs to be taken to avoid (near) duplicates between the seen and unseen actions, as also noted by |Rotbbeg et al. (2018).
Next to universal action models, several works have shown competitive results by taking universal object perspective. Large-scale networks are trained in the image domain on thousands of object labels (Jain et al. 2013 Wu et al. | & Snoek 2017 Liu et al. 2019 Mettes et al. 202T). Once trained any action can be inferred based on the object likelihoods in test video and the semantic similarities between objects and unseen actions. Regardless of the universal perspective, the current state-of-the-an shares the same fate: zero-shot inference results in a long-tailed distribution over selected actions. Many unseen actions simply never selected during persistent biases. We tackle this problem with the transductive universal transport over universally trained models. This paper introduces a new way of dealing with biases in universal models, hence the focus on the video domain over the image domain, where universal models are not sate--of-the-art
Mettes
Within zero-shot action recognition, transductive view is commonly applied, as noted in (Estevam et al. 2021). Rohrbach et al. (2013) provide a foundation for transduction in zero-shot context by exploiting the inter-sample similarity over the test set. In similar spirit, several works have pro- posed transductive extensions for zero-shot action recognition (Fu et al. 2014, Kodirov et al.) 2015 Alexiou et al., 2016 Xu et al. 2017 Gao et al.) 2019 Mandal et al. 2019). While transductive learn- ing no longer operates on individual test videos, many applications require evaluating entire video batches/collections simultancously. Different from existing approaches, we use the distribution over all test videos to improve the unseen action embeddings in the shared output space of universal mod- els by building on optimal transport. Moreover, our approach can naturally switch between inductive and transductive settings, as only the positioning of action embeddings updated.
CONCLUSIONS
In this work, we investigate a persistent limitation in current universal learning models for zero- shot action recognition, namely selection biases in the assignment of unseen actions test videos. We introduce transductive universal transport to alleviate this bias. Our approach consists of three stages: (i) finding an optimal transport mapping from unseen action embeddings to the mapped test videos (in universal action models) or object embeddings (in universal object models), (ii) calculat- ing a target embedding for each unseen action as a weighted Fréchet mean with the weights given by the transport couplings, and (iii) re-positioning the unseen actions along the geodesic spanned by the original and target embeddings. Empirical evaluation on four datasets shows the effectiveness of our approach for debiasing action assignments and improving zero-shot recognition outperforming existing approaches for zero-shot action classification and zero-shot tentioaa action localization. Our approach is general and can be used to improve any universal model.
result,

---

Under review as a conference paper at ICLR 2022
ETHICS AND REPRODUCIBILITY STATEMENT
Action recognition has a broad range of applications, for example in autonomous driving and robotics. While such applications hopefully have positive societal effect, the research can also potentially be used for applications such as surveillance and this should be taken into account. To ensure reproducibility, we start from off-thee-hellf models as provided by current universal learning models. In the paper, we have outlined the hyperparameters of our approach and we will release the code to reproduce the experiments.
REFERENCES
loannis Alexiou, Tao Xiang, and Shaogang Gong. Exploring synonyms as context in zero-shot action recognition. In ICIP, 2016.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. CoRR, 2021.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. JMLR, 2005.
Mina Bishay, Georgios Zoumpourlis, and Ioannis Patras. Tarn: Temporal attentive relation network for few-shot and zero-shot action recognition. In BMVC, 2019.
Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich. Displacement interpolation using lagrangian mass transport. In SIGGRAPH Asia, 2011.
Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. CVPR, 2020.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Dima Damen, Hazel Doughty, Giovanni Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kaza- kos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and baselines. TPAMI, 2020.
Valter Estevam, Helio Pedrini, and David Menotti. Zero-shot action recognition in videos: survey. Neurocomputing, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. ICCV, 2019.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Zahdi Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Foumnie, al. Pot: Python optimal transport. JMLR, 2021.
Yanwei Fu, Timothy M Hospedales, Tao Xiang, Zhenyong Fu, and Shaogang Gong. Transductive multi-view embedding for zero-shot recognition and annotation. In ECCV, 2014.
Chuang Gan, Ming Lin, Yi Yang, Gerard De Melo, and Alexander G Hauptmann. Concepts not
alone: Exploring pairwise relationships for zero-shot video activity recognition. In AAAI, 2016a.
Chuang Gan, Tianbao Yang, and Boqing Gong. Learning attributes equals multi-source domain generalization. In CVPR, 2016b.
Chuang Gan, Yi Yang, Linchao Zhu, Deli Zhao, and Yueting Zhuang. Recognizing an action using its name: knowledge-based approach. IJCV, 2016c.
Junyu Gao, Tianzhu Zhang, and Changsheng Xu. I know the relationships: Zero-shot action recog- nition via two-stream graph convolutional networks and knowledge graphs. In AAAI, 2019.
10

---

Under review as a conference paper at ICLR 2022
Meera Hahn, Andrew Silva, and James M Rehg. Action2vec: A crossmodal embedding approach to action learning. CVPRw, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Mihir Jain, Jan C Van Gemert, Thomas Mensink, and Cees GM Snoek. Objects2action: Classifying and localizing actions without any video example. In ICCV, 2015.
Tae Soo Kim, Jonathan Jones, Michael Peven, Zihao Xiao, Jin Bai, Yi Zhang, Weichao Qiu, Alan Yuille, and Gregory D Hager. Daszl: Dynamic action signatures for zero-shot learning. In AAAI, 2021.
Elyor Kodirov, Tao Xiang, Zhenyong Fu, and Shaogang Gong. Unsupervised domain adaptation for zero-shot learning. In ICCV, 2015.
Yikang Li, Sheng-hung Hu, and Baoxin Li. Recognizing unseen actions in a domain em- bedding space. In ICIP, 2016.
Jingen Liu, Benjamin Kuipers, and Silvio Savarese. Recognizing human actions by attributes. In CVPR, 2011.
Kun Liu, Wu Liu, Huadong Ma, Wenbing Huang, and Xiongxiong Dong. Generalized zero-shot learning for action recognition with web-scale video data. WWW, 2019.
Teng Long, Pascal Mettes, Heng Tao Shen, and Cees GM Snoek. Searching for actions on the hyperbole. In CVPR, 2020.
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean. In ICML, 2020.
Devraj Mandal, Sanath Narayan, Sai Kumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shah- baz Khan, and Ling Shao. Qut-of-distribution detection for generalized zero-shot action recogni- tion. In CVPR, 2019.
Pascal Mettes and Cees GM Snoek. Spatial-aware object embeddings for zero-shot localization and classification of actions. In ICCV, 2017.
Pascal Mettes, Dennis C Koclma, and Cees G M Snock. Shuffled imagenet banks for video event detection and search. ACM TOMM, 2020.
Pascal Mettes, William Thong, and Cees G M Snoek. Object priors for classifying and localizing unseen actions. IJCV, 2021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. NeurlPS, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. NeurIPS, 2013b.
Nina Miolane, Nicolas Guigui, Alice Le Brigant, Johan Mathe, Benjamin Hou, Yann Thanwerdas, Stefan Heyder, Olivier Peltre, Niklas Koep, Hadi Zaatiti, al. Geomstats: python package for riemannian geometry in machine learning. JMLR, 2020.
Ashish Mishra, Anubha Pandey, and Hema A Murthy. Zero-shot learning for action recognition using synthesized features. Neurocomputing, 2020.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. TPAMI, 2019.

---

Under review as a conference paper at ICLR 2022
Gabriel Peyré, Marco Cuturi, al. Computational optimal transport: With applications data science. Foundations and Trends@ in Machine Learning, 2019.
Marcus Rohrbach, Sandra Ebert, and Bernt Schiele. Transfer learning in a transductive setting. NeurIPS, 2013.
Alina Roitherg, Manuel Martinez, Monica Haurilet, and Rainer Stiefelhagen. Towards fair evalu- ation of zero-shot action recognition using external data. In ECCVw, 2018.
Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. CVPR, 2018.
Zuxuan Wu, Yanwei Fu, Yu-Gang Jiang, and Leonid Sigal. Harnessing object and scene semantics for large-scale video understanding. In CVPR, 2016.
Xun Xu, Timothy Hospedales, and Shaogang Gong. Transductive zero-shot action recognition by word-vector embedding. IJCV, 2017.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. CoRR, 2021.
Zhong Zhang, Chunheng Wang, Baihua Xiao, Wen Zhou, and Shuang Liu. Robust relative attributes for human action recognition. PAA, 2015.
Yi Zhu, Yang Long, Yu Guan, Shawn Newsam, and Ling Shao. Towards universal representation for unseen action recognition. CVPR, 2018
12


Reviews:
[{"review_id": "ozHhAqBZYhc", "reviewer": "Reviewer_2kAP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper demonstrates state-of-the-art results, one important concern is about fairer comparisons, especially the zero-shot spatio-temporal localization experiments (Table 3). \n- The setting of the proposed method is different from some compared papers.  For example, the authors focus on transductive ZSL, while Mettes et al. (2021), Kim et al. (2021),  and Brattoli et al. (2020) focus on inductive ZSL. \n- The proposed method uses both action and object information, while Brattoli et al. (2020) use action information only, and Mettes et al. (2021) use object information only. \n\nWithout fairer comparisons, it is hard to assess the effectiveness of the proposed method.\n\nAnother concern is that the importance of some critical components is not adequately evaluated.\n- This is also related to the comments above mentioned. The proposed method uses both video features and object information. Is this critical to obtain a good performance? The importance of video features and object information is not properly evaluated. One way to show this is to evaluate the performance of the proposed method using only one type of modality. Partial information is given in Figure 3 and the Fusion paragraph on page 7. Based on Figure 3 and the discussion, it seems that the proposed method does not outperform Brattoli et al. (2020) and Mettes et al. (2021) under the same experimental settings as the compared methods?\n\nTypo: in 3.2 implementation details: (2+1)D -> R(2+1)D\n", "summary_of_the_paper": "The paper targets transductive zero-shot action recognition.  To alleviate models biased to seen categories, the authors propose to re-position unseen action embedding through transduction. There are three steps in the proposed method: first, finding an optimal mapping from unseen actions to the mapped video in the shared hyperspherical space. Second, defining target embeddings as weighted Frechet means with the weight given by the transport couplings. Third, re-positioning unseen action embeddings along the geodesic between the original and target. The zero-shot classification performance of the proposed method is tested on the UCF-101 and HMDB datasets. The zero-shot spatio-temporal localization performance is tested on the UCF Sport and J-HMDB datasets. \n", "summary_of_the_review": "The paper can be further strengthened by demonstrating fairer comparisons and adequately evaluating the importance of the critical components.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gBeaxz10uR", "reviewer": "Reviewer_Gn5W", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The use of transductive universal transport for zero-shot action recognition is new.\n2. The experimental results show the effectiveness of this new method and also better performance than prior states of the art.\n\nWeaknesses\n1. The approach needs access to the entire testing video set to obtain distribution information of testing videos. This is an unrealistic setting. When used in practice, a machine learning model should expect to see one testing example a time.\n\n2. Many symbols are not clearly defined, making the math descriptions in this paper hard to read. For example, in Eq. (2), it is unclear what w_s, w_u, sum_|Lu| sum_|Ls| are, what weights for labels, i.e., w_lu and w_ls, mean, and why u_s and u_u are sets of labels.\n\n3. Does the use of transductive universal transport bring any computational overhead to the zero-shot learning model? It is interesting to see comparison of inference time and complexity with the baseline and some states of the art.", "summary_of_the_paper": "This work introduces transductive universal transport for zero-shot action recognition, where no training examples for unseen classes are available. To address the biases of prior approaches towards seen classes during inference, this paper re-positions unseen action embeddings through transduction by using the distribution of the unlabelled test set. Experimental results on several action recognition datasets demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The idea of using transductive universal transport for zero-shot action recognition is new, and the performance is good. But the core setting that the entire testing set is available during training to get the distribution information is unrealistic. The writing, especially the math part, needs improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y13krlh7-I", "reviewer": "Reviewer_cLxn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The zero-shot learning for action recognition is good direction however the applicability is limited in real world scenario.\n2. Figure 1 is very clear to give the illustration of the concept and the high-level idea of the whole paper.\n3. The performance are very stronger when compared with other baselines.\n\nWeakness:\n1. The technical details is a little hard to follow since I am not very familiar with the zero-shot learning. In my opinion, the proposed method is a general approach across video and image domains. Why not implement the experiments of such method on the image zero-shot learning?\n2.  Is there any visualization to intuitively show the learning results by the transductive universal transport?", "summary_of_the_paper": "The paper presents a zero-shot action recognition framework by learning an universal mapping from video to semantic space. The unseen action embeddings are re-positioned through leveraging the distribution of unlabelled test set. The universal mappings from unseen action to test videos are first defined and the target embeddings are treated as weighted Frechet means. The unseen action embeddings are re-positioned as a semantic regularization. The results on UCF101 and HMDB-51, UCF Sports and J-HMDB validates the proposed method.", "summary_of_the_review": "The paper gives a new method for zero-shot action recognition by learning the transduction from unseen action to test videos in hyperspherical space. The performances are good when compared with other state-of-the-art methods. However, the details of the technique  is a little hard to follow for me. In my opinion, the paper forms well and the writing is very good. I give accept currently because of my unfamiliarity of such domain. By the way, I will follow up the work in the following reviewing sections. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ak1q4Y2_zJ", "reviewer": "Reviewer_EmnY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n\n- Work on the problem of reducing the biases between seen categories in the source domain and unseen categories in the target domain\nthe semantic space.\n\n- Evaluate the approach on benchmark datasets for two tasks:  zero-shot action classification and spatio-temporal action localization.\n\nWeaknesses \n\n- Novelty seems incremental. The proposed transductive universal transport algorithm for embedding reposition seems like a simple weighting method guided by the distribution of the unlabelled test set. The paper merely uses existing approaches to solve the transductive optimal transport problem but does NOT bring any new insights.\n\n- Generalization seems a concern. The proposed approach heavily depends on the distribution of the unlabelled test set. It seems sensitive to the distribution and the number of clusters.  As shown in Figure 2, using the target embeddings seems on par with repositioned embeddings. Also, the proposed approach seems to only works in the case with a small number of clusters. ", "summary_of_the_paper": "This work tries to address the problem of zero-shot action recognition. Particularly, the paper aims at preventing the case that many unseen action categories in the target domain are simply never being selected during inference. Using the distribution of the unlabelled test set, the embeddings of unseen actions in the target domain are reweighted and repositioned along the geodesic such that they are better aligned with embeddings of training actions in the source domain. In experiments, Empirically, the proposed method has been evaluated on \nbenchmark datasets for tasks zero-shot action classification and spatio-temporal action localization.", "summary_of_the_review": "This paper proposes a sensible solution to reduce the bias between the source domain and the target domain for the task of action recognition. But novelty seems incremental. Also, some experiments seems a bit unconvincing and the approach seems not to scale to general settings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field and received mixed scores (1 borderline accept, 3 borderline reject). The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper. AC feels that this work has great potential, but needs more work to better clarify the contribution and include additional ablated study. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject