Title: UNSEEN ACTION RECOGNITION WITH UNPAIRED ADVERSARIAL MULTIMODAL LEARNING

Abstract: In this paper, we present a method to learn a joint multimodal representation space that allows for the recognition of unseen activities in videos. We compare the effect of placing various constraints on the embedding space using paired text and video data. Additionally, we propose a method to improve the joint embedding space using an adversarial formulation with unpaired text and video data. In addition to testing on publicly available datasets, we introduce a new, large-scale text/video dataset. We experimentally confirm that learning such shared embedding space benefits three difficult tasks (i) zero-shot activity classification, (ii) unsupervised activity discovery, and (iii) unseen activity captioning.

1. INTRODUCTION:

Videos contain multiple data sources, such as visual, audio and text/caption data. Each data modality has distinct statistical properties capturing different aspects of the event. Current state-of-the-art activity recognition models (Carreira & Zisserman, 2017; Tran et al., 2017) only take visual data and class labels as input, which limits the information the model can learn from. For example, the sentence 'a group of men play basketball outdoors' contains rich information, such as 'outdoors' and 'group of men' compared to just the activity class of 'basketball.' Without multimodal learning, these models are unable to benefit from the additional information.

In this paper, we explore multimodal learning from video and language data, each having its own representation. Video data is represented as a sequence of images (spatio-temporal pixel data) while text is represented as a sequence of word embeddings (temporal data). Using a shared embedding space allows for learning the highly non-linear relationships between these modalities. The goal of learning such relationships is to capture similarities between concepts (e.g., basketball and volleyball both being sports with a ball) and further generalize to concepts not seen during training. To enable this generalization, we rely on pre-trained word embeddings as they capture relationships between all words in the English language (Mikolov et al., 2013) . By learning a joint representation space, we can capture these relationships, allowing generalization to unseen video activities.

To evaluate our approach, we test on several tasks based on recognizing activities not seen during training. First, we test on a zero-shot learning task, where we classify activity classes not seen during training. Our second task is unsupervised activity discovery, where no class information is provided during training, relying only on the relationship between text and videos to learn good embeddings. Finally, we evaluate on captioning videos with activities/words not seen during training.

Many existing approaches to both zero-shot and embedding space learning require paired data examples, which can be expensive to obtain. By taking advantage of adversarial learning (Goodfellow et al., 2014) , we are able augment our method with unpaired data (i.e., random sentences and random videos without any labels or correspondence) to further improve our learned shared embedding space.

In this paper, we design a method capable of learning joint video/language embedding spaces using both paired and unpaired data and experimentally confirm its benefit to three challenging tasks (i) zero-shot activity recognition, (ii) unsupervised activity discovery, and (iii) unseen activity captioning. We show that the use of unpaired, multimodal data allows learning a shared embedding space that generalizes to unseen data better than existing approaches.

2. RELATED WORKS:

Multimodal learning Previous approaches to multimodal learning have used Restricted Boltzmann Machines (Srivastava & Salakhutdinov, 2012) or log-bilinear models (Kiros et al., 2014) to learn distributions over sentences and images. Ngiam et al. (2011) designed an autoencoder that learns joint audio-video representations, however they are unable to train the model end-to-end, instead relying on greedy, layer-by-layer training and further do not evaluate the effect of using unpaired data. Similarly, Chandar et al. (2016) proposed a auto-encoder able to learn correlations between different view of images. Frome et al. (2013) describe a model that maps images and words to a shared embedding. However, these works either learn a joint embedding by concatenating the different features or require a triplet consisting of positive and negative matches and do not evaluate the effects of using unpaired data.

Text and vision:

Using both text and visual data has been studied for many tasks, such as image captioning (Karpathy et al., 2014; Johnson et al., 2016; Karpathy & Fei-Fei, 2015) or video captioning (Krishna et al., 2017; Zhou et al., 2018; Xu et al., 2018) . Other works have explored the use of textual grounding for image/video retrieval (Gupta & Mooney, 2010; Rohrbach et al., 2016; Miech et al., 2017; Hendricks et al., 2017) . There have been various models proposed to learn a fixed text embedding space with mappings from video features into this embedding space (Guadarrama et al., 2013; Otani et al., 2016; Song et al., 2016; Wang et al., 2016; Xian et al., 2016) . However, these works all learn only a single directional mapping (e.g., only mapping from visual to text), only learn with paired text/image samples and require additional data in the form of positive/negative pairs.

Learning with unpaired data Recently, there have been many works taking advantage of variational autoencoders (VAEs) (Kingma & Welling, 2014) or generative adversarial networks (GANs) (Goodfellow et al., 2014) to learn mappings between unpaired samples. CycleGan (Zhu et al., 2017) uses a cycle-consistency loss (i.e., the ability to go from a sample in one domain to a second domain then back to the source) to learn unpaired image translation (e.g., image to sketch). Other works learn many-to-many mappings between images (Almahairi et al., 2018) or use two GANs to map between domains (Yi et al., 2017 ). An autoencoder with shared weights for both domains has been used to learn a latent space for image-to-image translation (Liu et al., 2017) . However, these works all focus on learning mappings between unpaired data of the same modalitiy (e.g. image to image), where the data is from the same underlying distribution. Our work focuses on learning from different modalities with very different distributions.

Zero-shot activity recognition Many previous works have studied zero-shot activity recognition. Common approaches include using attributes (Liu et al., 2011; Palatucci et al., 2009; Romera-Paredes & Torr, 2015) or word embeddings (Xu et al., 2015; 2017; Norouzi et al., 2013; Socher et al., 2013; Kodirov et al., 2017) or learning a similarity metric (Zhang & Saligrama, 2015; Chopra et al., 2005) . Some works have explored using adversarial losses on the latent space (Chen et al., 2018) or use GANs to generate features for unseen classes (Xian et al., 2018) or used auto-encoders (Wang et al., 2017) .

Our work differs from these previous works in three key ways: (1) we show the benefit of additional data augmentation using unpaired samples, (2) we experimentally compare the use of the embedding space for both zero-shot recognition, unseen recognition, and unseen video captioning, and (3) we learn a shared, multimodal embedding space with bi-directional mappings in an end-to-end fashion.

3. METHOD:

To enable learning of a joint embedding space, we use a deep autoencoder architecture. Our model consists of 4 neural networks:

Video Encoder E

V : v → z v Video Decoder G V : z → v Text Encoder E T : t → z t

Text Decoder G T : z → t where v is a sequence of video data and t is a sentence (sequence of words). z is the embedded data in the shared space that we are learning. The encoders learn a compressed representation of the video or text while the decoders are trained to reconstruct the input: As both text and video data are sequences, they often have different lengths. A joint embedding space requires that the features from both modalities have the same dimensions. Given a text representation of length L and a video representation of length T , we need to obtain a fixed-size representation. To learn a fixed-dimensional embedding, there are many choices for the encoder/decoder architecture, such as temporal pooling (Ng et al., 2015) , attention (Piergiovanni et al., 2017) or RNNs (Krishna et al., 2017) . We chose temporal attention filters (Piergiovanni et al., 2017) as they learn a mapping from any length input to a N -dimensional vector and have been shown to outperform temporal pooling and RNNs on activity recognition tasks.

L recons (v, t) = ||G V (E V (v)) -v|| 2 + ||G T (E T (t)) -t|| 2 (1)

The attention filters consists of N Gaussians, each learning 2 parameters: a center ĝ and width σ. The filters are determined by:

EQUATION

The weights are applied by matrix multiplication with the video or text sequence: v = F v. Additionally, we can learn a transposed version of these filters to reconstruct the input: v = F T v . To reconstruct the input, the decoders learn their own parameters with the tensors transposed, resulting in the matching output size. The architecture of the encoders is shown in Fig. 1 .

3.1. LEARNING A JOINT EMBEDDING SPACE:

To learn a joint representation space, we minimize the L 2 distance between the embeddings of a pair of text and video (shown in Fig. 2 (a)):

L joint (v, t) = ||E V (v) -E T (t)|| 2 (3)

This forces the joint embeddings to be similar and when combined with the reconstruction loss, ensures that the representations can still reconstruct the input. Mapping from text to video using the cross-domain (Eq. 4) loss. 

V T z E v E T L joint L recons G T G v V T z E v E T G T G v (a) (b) V z L 2 (c) L recons L cross L cross

V T z E v E T G T G v D v ( ) = 1 D v ( ) = 0 D T ( ) = 0 D T ( ) = 1 D z ( ) = 0 D z ( ) = 1

EQUATION

Additionally, we can use a 'cycle' loss to map from video to text and back to video. Note that while the previous losses all require paired examples, this loss does not.

L cycle (v, t) = ||G T (E V (G V (E T (t)))) -t|| 2 + ||G V (E T (G T (E V (v)))) -v|| 2 (5)

To train the model to learn a joint embedding space, we minimize

EQUATION

where α i are hyper-parameters weighting the various loss components.

3.2. SEMI-SUPERVISED LEARNING WITH UNPAIRED DATA:

To learn using unpaired data (i.e., unrelated text and video), we use an adversarial formulation. We treat the encoders and decoders as generator networks. We then learn an additional 3 discriminator networks which constrain the generators and embedding space and force the encoders and decoders to be consistent:

(1) D z which learns to discriminate between latent text representations and latent video representations. Conceptually, this constrains the learned embeddings to appear to be from the same distribution.

(2) D V which learns to discriminate between true video data and generated video data G V (E T (t)).

(3) D T which learns to discriminate between true text data and generated text data, G T (E V (v)).

Given these discriminators, we minimize the following losses:

EQUATION

Using the discriminators, we can train the generators (encoders and decoders) to minimize the following loss based on unpaired data:

EQUATION

Note that in this formulation, v and t are not paired.

These networks are trained in an adversarial setting. For example, for the text-to-video generator (i.e., v = G V (E T (t)) and video discriminator, D V , we optimize the following minimax equation:

EQUATION

This equation is similarly applied for video-to-text. For learning the embedding space with the video and text encoders, E V , E T and the discriminator D z , we optimize the following minimax equation:

EQUATION

As training GANs can be unstable, we developed an method to allow for more stable training of the joint embedding space, shown in Algorithm 1. We initialize both the generator and discriminator networks by training only on paired data. After several iterations of this, we train with both unpaired and paired data. We found the initial training of the generators and discriminators was important for stability, without it the loss often diverges and the learned embedding did not generalize to unseen activities.

Algorithm 

4. EXPERIMENTS:

We compare our various approaches on different tasks (i) zero-shot activity recognition, (ii) unsupervised activity discovery and (iii) unseen activity captioning. These tasks test various combinations of our encoders and decoders and how well the shared representation generalizes to unseen data. We experimentally confirm the benefits of our methods using multiple public datasets: AcitivtyNet (Heilbron et al., 2015; Krishna et al., 2017) , HMDB (Kuehne et al., 2011) , UCF101 (Soomro et al., 2012) , and MLB-YouTube (Piergiovanni & Ryoo, 2018) .

4.1. IMPLEMENTATION/TRAINING DETAILS:

We implement our models in PyTorch. For the per-segment video CNN, we use I3D (Carreira & Zisserman, 2017) to obtain a 1024 × T video representation. We trained a version of I3D based on Kinetics-600, but withheld all classes that appear in ActivityNet, HMDB51, or UCF101 so that the classes are truly unseen. This resulted in a training set with 478 classes and 278k videos. Since generating videos is an extremely challenging task, the video autoencoders start with and generate the I3D feature. We use GloVe word embeddings (Pennington et al., 2014) to obtain a language representation. We set N = 4 for the temporal attention filters and apply 4 fully connected layers. These layers are followed by L 2 normalization so that the embedding space has unit length (Tygert et al., 2015) . We train for 200 epochs and use stochastic gradient descent with momentum to minimize the loss function with a learning rate of 0.01. After every 50 epochs, we decay the learning rate by a factor of 10. When training in the adversarial setting (e.g., Algorithm 1), we initialize the network training for 50 epochs on paired data followed by 200 on the paired + unpaired data.

Baselines For baselines, we compare to a fixed-text embedding space, were only a mapping from video data into the text embedding space is learned (e.g., (Otani et al., 2016) ). We also compare to learning a shared embedding space with the 'recons' (Eq. 1) and 'cross' (Eq. 4) terms (e.g., (Ngiam et al., 2011) ).

4.2. ZERO-SHOT ACTIVITY RECOGNITION:

Zero-shot activity recognition is the problem of classifying a video that belongs to a class not seen during training. Given training videos of seen classes together with paired text descriptions, our approach learns a shared embedding that maps videos/texts from multiple seen classes. The objective is to classify videos of unseen classes solely based on the learned embedding space and the text samples.

We use the ActivityNet captions (Krishna et al., 2017) dataset to learn the joint embedding space, as this dataset has both sentence descriptions for each video as well as activity classes. We randomly choose a set of K activity classes and withhold all videos/sentences belonging to those classes during training. For testing, we take a subset of sentences for the unseen classes and map the sentences into the joint embedding space, z t = E T (t). We then map the videos into the space, z v = E V (v) and use nearest neighbors to match each video (z v ) to text (z t ), using the class of the nearest sentence as the classification for the video.

In Table 1 , we compare the effect of the various loss components. For each method, we run 10 trials each with a different set of unseen activity classes and average the results. We find that previous methods of learning a fixed language embedding (e.g., (Socher et al., 2013; Xu et al., 2015; 2017) ) are significantly outperformed by learning a joint representation. Previous methods learning embedding spaces without the 'joint' term (e.g. (Ngiam et al., 2011 )), we found yield nearly random performance on these tasks, suggesting that forcing the representations to match in the embedding space is important. Further, adding the reconstruction, cross-domain, and cycle losses all improve performance. We also compare to a standard triplet loss (e.g., (Guadarrama et al., 2013) ) which requires additional positive/negative samples. We find that the triplet loss outperforms the 'joint' loss, but is surpassed by adding the 'cycle' and 'cross' terms, which use less data.

We also compare the various components of the adversarial loss. We compare to having just the adversarial loss on the embedding space (D z ), like (Chen et al., 2018) , and compare just the adversary on the generated videos/sentences. We find that the use of all terms in important for performance.

To obtain unpaired data, we use the sentence descriptions from the Charades (Sigurdsson et al., 2016) dataset, which contains many activities in a home setting. The unpaired video data is sampled from HMDB and UCF101. While previous works such as (Ngiam et al., 2011) can support learning with unpaired data, we find that the adversarial loss provides better results than just the 'cycle' and 'recons' terms, and further improves over training with just paired data.

In Table 2 , we compare our approach to previous zero-shot learning methods on HMDB and UCF101. The paired training data for these models is drawn from ActivityNet with any classes belonging to HMDB or UCF101 withheld. The unpaired text data is sampled from Charades and the video data comes from either HMDB (when testing on UCF101) or UCF101 (when testing on HMDB). As HMDB and UCF101 have no text descriptions, we created a sentence description for each activity class (included in Appendix B). We find that our joint embedding space outperforms the previous approaches on these datasets and unpaired adversarial learning further improves our performance. Table 2 : Results on HMDB51 and UCF101 compared to previous state-of-the-art results. We find that learning a shared embedding space is beneficial and that augmented with unpaired data provides the best results.

Feature HMDB51 UCF101:

SJE (Akata et al., 2015) IDT 12.0 ± 2.6 9.3 ± 1.7 ConSe (Norouzi et al., 2013) IDT 15.0 ± 2.7 11.6 ± 2.1 ZSECOC (Qin et al., 2017) IDT 22.6 ± 1.2 15.1 ± 1.7 Semantic Embedding (Xu et al., 2015) IDT 21.2 ± 3.0 18.6 ± 2.2 Manifold Ridge Regression (Xu et al., 2017) IDT 24.1 ± 3.8 22.1 ± 2.5 SAE (Kodirov et al., We explore different strategies for obtaining unpaired data. For these experiments, we keep a fixed set of paired text and videos and explore adding various sources of unpaired data. We compare (i) 10k random sentences from Wikipedia, (ii) 10k random dictionary definitions, and (iii) 10k dictionary He is an aggressive third baseman and he can really play over there you know. He definitely takes pride in his defense as well.

He got right on top of that pitch, Pederson, and shot and way out of here. Three-run blast.

That has been a feat in this series for both teams, nobody is hitting with two strikes. That's how good the pitching has been.

They would suspend him at the beginning of next year as opposed to for a game during this World Series. definitions of verbs. We also compare adding 10k random videos from YouTube as additional video data. We show our results using 10 unseen classes in ActivityNet in Table 4 . We find that augmenting with similar unpaired data improves performance, while using irrelevant data harms performance. We find that dictionary verb definitions improve performance the most, as they capture important semantic information regarding the activities we are learning. The use of additional video data is further beneficial.

4.4. UNSUPERVISED ACTIVITY DISCOVERY:

To further evaluate our joint embedding space, we conducted experiments on unsupervised activity discovery. For this task, we expanded the MLB-YouTube dataset (Piergiovanni & Ryoo, 2018) by densely annotating the videos with a transcription of the announcers' commentary, resulting in approximately 50 hours of aligned text and video. Examples of this data are shown in Fig. 4 . The MLB-YouTube dataset is designed for fine-grained activity recognition, where the difference between activities is quite small. Additionally, these captions only roughly describe what is happening in the video, and often contain unrelated stories or commentary on a previous event, making this a challenging task. The dataset will be made publicly available. To train our joint embedding space, we split each baseball video into 30 second intervals and use the corresponding text as paired data, resulting in 6,089 paired training samples.

We evaluate our joint embedding using the segmented videos from MLB-YouTube. For each video, we compute the embedded features and apply k-means clustering (k = 8, the number of classes). Each segmented video is assigned to a cluster and votes for the cluster label based on its ground truth label. We use that cluster assignment for classification on the MLB-YouTube test set. We report our findings in Table 5 . As a baseline, we cluster I3D features pre-trained on Kinetics. We find that our methods improve the embedding space. However, we note that when using unpaired data from Charades, the performance drops. This is likely due to Charades data being very different from MLB-YouTube data. We collected additional captions and baseball videos to augment the MLB-YouTube dataset, and confirmed that unpaired data helps when it is from a similar distribution.

In Table 6 we compare our various methods for unsupervised activity discovery on HMDB and UCF101. Here, we learn a joint embedding space using the ActivityNet videos and captions. We withhold any videos belonging to a class in HMDB or UCF101. Unlike MLB-YouTube, on these datasets, we find that using the unpaired training with Charades further improves performance. This confirms that when the additional data is similar to the target dataset, using the adversarial learning setting further improves the embedding space.

4.5. UNSEEN VIDEO CAPTIONING:

As our model learns a bi-directional mappings, we can apply our model to generate video captions.

Existing video captioning models are unable to create realistic captions for unseen activities, as without training data they do not know the words to describe the video. Given a video, v, we can generate a caption by mapping the video to text t = G T (E V (v)). For each word, we then use nearest neighbors matching with the GloVe embeddings to obtain the words to form a sentence. In Table 7 , we report the commonly used METEOR (Banerjee & Lavie, 2005) and CIDEr (Vedantam et al., 2015) scores of our various models, measured with the unseen classes from the ActivityNet dataset. We find that learning a joint representation is beneficial and using unpaired samples further improves the task. In Fig. 5 , we show example captioned videos. Note that this task is extremely challenging, as it requires the model to generate captions using activity words (e.g., basketball) not seen during training.

5. CONCLUSION:

We proposed an approach to learn a joint language/text embedding space using various constraints. We further extended the model to be able to learn with unpaired video and text data using an adversarial formulation. We experimentally confirmed that learning such an embedding space benefits three difficult tasks (i) zero-shot activity classification, (ii) unsupervised activity discovery, and (iii) unseen activity captioning. We find that the use of related unpaired data is beneficial. We presented several strategies for obtaining unpaired data and confirmed the benefit of adding additional, relevant unpaired data. To confirm that temporal attention is beneficial, we compare different forms of temporal pooling (i) max-pooling, (ii) sum-pooling, (iii) LSTM, and (iv) temporal attention filters (Piergiovanni et al., 2017) . In Table 8 , we compare these temporal pooling methods learning the joint embedding space. We confirm that using the temporal attention filters performs best.

A.2 COMPARISON OF DIFFERENT RATIOS OF PAIRED AND UNPAIRED DATA:

We compare different ratios of paired and unpaired data to see how much paired data we require and how much unpaired data is beneficial. For these experiments, we use all the loss terms (i.e., what provided us the best results). Note that in these experiments, the total number of samples was the same for each method (40k examples) so that we can directly compare the effects of unpaired data vs. paired data. Thus not all the available data was used.

In Table 9 , we show the results. We find that using no paired data results in nearly random performance, but using using some paired data greatly improves the embedding space. The model using 100% paired data performs best, as all the others are using less overall paired data.

We also compare augmenting our 40k paired training samples with different amounts of unpaired data. Since UCF101 and HMDB only have 13k and 7k examples, to get up to 60k samples, we also use videos from the Kinetics dataset (Carreira & Zisserman, 2017) . The results, shown in Table 10 , show that adding the initial 10k samples is most beneficial, while additional samples do not seem to meaningfully improve results. However, due to our training method where each batch consists of 50% paired data and 50% unpaired data, the additional unpaired data does not harm results either. In Fig. 6 , we compare t-SNE embeddings of the fixed text representation and our joint embedding space. This visually shows that learning a joint embedding space gives more distinct class distributions.

A.3.1 MLB-YOUTUBE CAPTIONS:

As a baseline for the MLB-YouTube captions dataset, we compared several different models for standard video captioning (i.e., all activity classes are seen). This task is quite challenging compared to other datasets as the announcers commentary is not always a direct description of the current events. Often the announcers tell loosely related stories and attempt to describe events differently each time to avoid repetition. Additionally, the descriptions contain on average 150 words for each 30 second interval and current captioning approaches usually only trained and tested on 10-20 word sentences. Due to these factors, this task is quite challenging the standard evaluation metrics do not account for these factors. In Table 11 , we report our results on this task.

B HMDB AND UCF101 SENTENCES:

For the HMBD and UCF101 datasets, we created sentences to describe each activity class. Our sentences descriptions are included in this appendix.

These sentences are written for each activity class (by randomly selecting a single video per class) and are shared for all instances of the activity. Depending on what video was randomly chosen for the class, some sentences describe the actor as a 'man', 'woman', or 'person' which could confuse the model. Ideally, the CNN embedding needs to learn to ignore the impact of such pronoun changes.

We conducted experiments comparing randomly replacing the pronouns to determine if there was any bias introduced by the pronouns. We show the results in Table 12 . We find that the choice of pronouns does not impact performance, as our model automatically learns to focus more on verbs rather than pronouns. When examining the temporal attention filters on the sentences, we found that they placed very little 'attention' on the start of the sentence, where the pronoun usually is, suggesting that the pronoun has very little effect on the embedding space we learned.

:



Figures and Tables:

Figure 1: Illustration of the encoder models used to learn a joint embedding space. Videos and sentences are mapped into a low-dimensional space by applying CNNs and temporal attention. Then several fully-connected layers map into the joint embedding space. The decoders follow this same architecture with the weights transposed.

Figure 2: Visualization of several constrains on the shared embedding space. Circles are video data, ovals are reconstructed video. Diamonds are text data, and pentagons are reconstructed text. (a) The reconstruction (Eq. 1) and joint (Eq. 3) losses. (b) Mapping from text to video using the cross-domain (Eq. 4) loss.

Figure 3: Visualization of the adversarial formulation to learn with unpaired data. We create 3 discriminators, (1) D z learns to discriminate examples of text/video in the latent space. (2) D V learns to discriminate video generated from text compared to video. (3) D T learns to discriminate generated text compared to text.

Figure 4: Example video sequences from the MLB-YouTube dataset with the commentary caption. Top: Sentences that describe the occurring activities. Bottom: Sentences that do not describe the current activities.

Figure 5: Example captions for unseen activities. Left: Using a joint embedding space allows the model to correctly caption this video as basketball, despite never seeing an example of basketball during training. Right: An example of a caption for the unseen water-ski activity. Here the model fails to correctly caption the activity.

Figure 6: t-SNE mapping of (a) fixed text representation and (b) joint embedding with all paired losses for the MLB-YouTube dataset. The joint embedding space provides most distinct representations for the activities. Each color represents the activity class of the video (e.g., swing, hit, foul ball, etc.).

Comparison of various methods on ActivityNet for 5, 10, 20 or 50 unseen classes. These results are averaged over 10 trials where each trial has a different set of unseen activity classes.

Table content: <table><tr><td/><td colspan="4">5 Unseen 10 Unseen 20 Unseen 50 Unseen</td></tr><tr><td>Paired Data</td><td/><td/><td/><td/></tr><tr><td>Fixed Text Representation</td><td>41.9</td><td>38.4</td><td>29.4</td><td>15.6</td></tr><tr><td>Triplet Loss</td><td>56.8</td><td>44.9</td><td>38.8</td><td>23.3</td></tr><tr><td>joint</td><td>54.3</td><td>41.7</td><td>36.1</td><td>21.2</td></tr><tr><td>recons + cross</td><td>21.1</td><td>12.6</td><td>7.6</td><td>2.9</td></tr><tr><td>joint + recons</td><td>70.1</td><td>54.4</td><td>42.6</td><td>27.5</td></tr><tr><td>joint + recons + cycle</td><td>70.4</td><td>54.3</td><td>42.1</td><td>26.8</td></tr><tr><td>joint + recons + cross</td><td>72.6</td><td>55.4</td><td>43.2</td><td>27.8</td></tr><tr><td>joint + recons + cross + cycle</td><td>76.4</td><td>56.9</td><td>45.5</td><td>28.8</td></tr><tr><td colspan="4">With Adversarial Losses (joint + recons + cross + cycle + Adv.)</td><td/></tr><tr><td>+ Dz</td><td>78.4</td><td>57.3</td><td>45.9</td><td>29.1</td></tr><tr><td>+ Dv + Dt</td><td>77.2</td><td>57.2</td><td>45.6</td><td>28.9</td></tr><tr><td>+ Dz + Dv + Dt</td><td>79.7</td><td>58.3</td><td>46.5</td><td>29.7</td></tr><tr><td>Paired + Unpaired Data</td><td/><td/><td/><td/></tr><tr><td>recons + cycle</td><td>22.8</td><td>13.6</td><td>8.4</td><td>4.2</td></tr><tr><td>joint + recons + cycle</td><td>72.6</td><td>58.4</td><td>44.7</td><td>29.3</td></tr><tr><td>joint + recons + cross + cycle</td><td>73.4</td><td>59.1</td><td>45.3</td><td>29.2</td></tr><tr><td>All terms</td><td>82.5</td><td>60.4</td><td>46.2</td><td>30.1</td></tr></table>

Unseen activity recognition results on ActivityNet, HMDB51 and UCF101, evaluated by using both unseen and seen classes for the testing.

Table content: <table><tr><td/><td colspan="4">ActNet (10 unseen) ActNet (50 unseen) HMDB51 UCF101</td></tr><tr><td>Fixed Text Representation</td><td>55.7</td><td>46.8</td><td>24.5</td><td>26.8</td></tr><tr><td>Triplet Loss</td><td>57.7</td><td>48.5</td><td>27.6</td><td>29.8</td></tr><tr><td>joint</td><td>62.1</td><td>50.2</td><td>29.8</td><td>30.6</td></tr><tr><td>joint + recons</td><td>64.4</td><td>52.6</td><td>30.4</td><td>31.3</td></tr><tr><td>joint + recons + cross + cycle</td><td>69.6</td><td>58.5</td><td>35.6</td><td>36.5</td></tr><tr><td>Paired + Unpaired Data</td><td/><td/><td/><td/></tr><tr><td>All terms</td><td>71.6</td><td>65.9</td><td>38.9</td><td>42.1</td></tr></table>

Comparison of various source of unpaired data on ActivityNet with 10 unseen classes.

Table content: <table><tr><td>Paired Data</td><td>58.3</td><td>69.6</td></tr><tr><td>+ Random Wikipedia Sentences</td><td>55.8</td><td>66.4</td></tr><tr><td>+ Random Dictionary Defs.</td><td>56.3</td><td>68.2</td></tr><tr><td>+ Verb Dictionary Defs.</td><td>59.2</td><td>70.7</td></tr><tr><td>+ Random YouTube Videos</td><td>58.7</td><td>70.1</td></tr><tr><td>+ Verbs + Random Videos</td><td>60.3</td><td>71.2</td></tr></table>

Comparison of unsupervised activity classification on MLB-YouTube.Several men are playing basketballPeople are swimming in the ocean

Table content: <table><tr><td/><td/><td/><td colspan="3">Table 6: Comparison of unsupervised activity</td></tr><tr><td/><td colspan="2">Accuracy mAP</td><td colspan="2">classification on HMDB and UCF101.</td><td/></tr><tr><td>Baseline I3D features Fixed Text Representation</td><td>23.4 27.9</td><td>32.6 34.7</td><td/><td colspan="2">HMDB UCF101</td></tr><tr><td>Joint</td><td>34.5</td><td>41.6</td><td>Baseline I3D features</td><td>26.6</td><td>42.5</td></tr><tr><td>Joint + recons</td><td>37.9</td><td>43.7</td><td>Joint</td><td>32.4</td><td>57.7</td></tr><tr><td>Joint + recons + cycle</td><td>44.2</td><td>48.6</td><td>Joint + recons</td><td>33.5</td><td>59.0</td></tr><tr><td>Joint + recons + cross</td><td>43.7</td><td>49.3</td><td>All (paired)</td><td>34.6</td><td>59.5</td></tr><tr><td>All (paired)</td><td>48.4</td><td>51.2</td><td>All (+ unpaired)</td><td>34.9</td><td>59.9</td></tr><tr><td>All (+ unrelated unpaired)</td><td>39.7</td><td>43.9</td><td/><td/><td/></tr><tr><td>All (+ related unpaired)</td><td>49.1</td><td>54.3</td><td/><td/><td/></tr></table>

Comparison of several models for unseen activity captioning using the ActivityNet dataset, using METEOR and CIDEr scores. This evaluation was done on 10 unseen classes held out during training. Higher values are better.

Table content: <table><tr><td/><td colspan="2">METEOR CIDEr</td></tr><tr><td>Fixed Text Representation</td><td>3.64</td><td>8.95</td></tr><tr><td>Joint</td><td>4.21</td><td>9.23</td></tr><tr><td>All (paired)</td><td>5.31</td><td>11.21</td></tr><tr><td>All (paired + unpaired)</td><td>6.89</td><td>13.95</td></tr></table>

Comparison of temporal pooling methods for 5 unseen classes in the ActivityNet dataset.

Table content: <table><tr><td/><td>Accuracy</td></tr><tr><td>Max Pooling</td><td>23.4</td></tr><tr><td>Sum Pooling</td><td>24.1</td></tr><tr><td>LSTM</td><td>42.3</td></tr><tr><td>Temporal Attention Filters</td><td>55.2</td></tr></table>

Comparison of different ratios of paired and unpaired data methods for 5 unseen classes in the ActivityNet dataset.

Table content: <table><tr><td colspan="2">Paired/Unpaired Accuracy</td></tr><tr><td>100% / 0%</td><td>74.2</td></tr><tr><td>75% / 25%</td><td>73.2</td></tr><tr><td>50% / 50%</td><td>69.7</td></tr><tr><td>25% / 75%</td><td>62.6</td></tr><tr><td>0% / 100%</td><td>24.5</td></tr><tr><td>A ADDITIONAL EXPERIMENTS</td><td/></tr></table>

Comparison using 40k paired examples and varying amounts of unpaired samples for 5 unseen classes in the ActivityNet dataset.

Table content: <table><tr><td colspan="2">Unpaired Samples Accuracy</td></tr><tr><td>0</td><td>77.1</td></tr><tr><td>10k</td><td>82.4</td></tr><tr><td>20k</td><td>83.9</td></tr><tr><td>40k</td><td>83.6</td></tr><tr><td>60k</td><td>83.5</td></tr></table>

Comparison of several models for standard, seen video captioning using the MLB-YouTube dataset, using Bleu, METEOR and CIDEr scores. Higher values are better.

Table content: <table><tr><td/><td colspan="3">Bleu METEOR CIDEr</td></tr><tr><td colspan="2">Fixed Text Representation 0.12</td><td>0.04</td><td>0.12</td></tr><tr><td>Joint Representation</td><td>0.14</td><td>0.08</td><td>0.15</td></tr><tr><td>Joint + all paired</td><td>0.15</td><td>0.10</td><td>0.18</td></tr><tr><td>Joint + paired + unpaired</td><td>0.10</td><td>0.02</td><td>0.08</td></tr><tr><td>A.3 MLB-YOUTUBE CAPTIONS</td><td/><td/><td/></tr></table>

Comparison of various pronouns on the UCF101 dataset with 50 unseen classes.

Table content: <table><tr><td/><td>Accuracy</td></tr><tr><td>Baseline Sentences</td><td>33.4</td></tr><tr><td>All 'man'</td><td>33.2</td></tr><tr><td>All 'woman'</td><td>33.3</td></tr><tr><td>All 'person'</td><td>33.4</td></tr><tr><td>Random pronoun</td><td>33.4</td></tr></table>

Peer review (iclr2019):
 This paper proposes a joint embedding model that aligns video sequences with sentences describing the context (caption) in a shared embedding space. With the space, various tasks such as zero-shot activity recognition and unseen video captioning can be performed. The problem tackled in this paper is interesting. However, the approach proposed is limited in novelty and there are some serious flaws in the experimental settings. So overall, this paper is not yet ready for publication. Pros: • The overall bidirectional encoder-decoder architecture for learning a shared embedding space is sensible. It is also interesting that adversarial training is introduced so that unlabelled data can be utilized. • Additional annotations are provided to two activity recognition datasets, creating new benchmarks. Cons • My first concern is the limited novelty of the work. Although I am not aware of a joint embedding learning model that has exactly the same architecture and formulation, the model is closely related to many existing ones both in zero-shot learning and beyond. More specifically, o The overall framework is similar to “correlational neural networks”, Neural Computation, 2016 by Chandar et al. This should be acknowledged. o The connections to CyclyGan and its variants for image-to-image style transfer is obvious, as pointed out by the authors. o More importantly, there are quite a few closely related zero-shot learning (ZSL) papers published recently. Although they focus on static images and class name, rather that image sequences and sentences, I don’t see any reason why these models cannot be applied to solve the same problem tackled in this paper. In particular, the autoencoder architecture was first used in ZSL in E. Kodirov, T. Xiang and S. Gong, "Semantic Autoencoder for Zero-Shot Learning", in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, July 2017. This work is further extended in Chen et al, “Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Network”, cvpr18, now with adversarial learning. Similarly, variational autoencoder is adopted in Wang et al, Zero-Shot Learning via Class-Conditioned Deep Generative Models, AAAI 2018. Note that the joint embedding spaces in these studies are the semantic spaces – attribute or word vector spaces representing the classes. In contrast, since the semantic modality is a variable-length word sequences, this is not possible, so a third space (other than the visual feature space or semantic space) is used as the embedding space. Beyond these autoencoder based models, there are also a number of recent ZSL works that use a conditional generative model with adversarial loss. Instead of learning a joint embedding space where the visual and text modalities are aligned and compared for recognition, these works use the text modality as condition to the generative model to synthesize visual features for the unseen classes followed by a conventional supervised classifier. The representative one of this line of work is Xian et al, “Feature Generating Networks for Zero-Shot Learning, cvpr18”. o In summary, there are too many existing works that are similar to the proposed one in one or more aspects. The authors failed to acknowledge most of them; moreover, it is not argued theoretically or demonstrated empirically, why combining different approaches together is necessary/making fundamental differences. • My second main concern is the experiment setting. This paper adopts a conventional ZSL setting in two aspects: (1) the visual features are obtained by a video CNN, I3D, which is pretrained on the large (400 or 600 classes depending on which version is used) Kinetics dataset. This dataset have classes overlapping with those in ActivityNet, HMDB and UCF101. So if these overlapped classes are used in the unseen class partition, then the ZSL assumption (the target classes are ‘unseen’) is violated. (2): The test data only contains unseen class samples. In practice, one will face a test set composed of a mix of seen and unseen classes. Under this more realistic setting (termed generalized ZSL in the ZSL community), a ZSL must avoid the bias towards the seen classes which provide the only visual data available during training. The two problems have been identified in the ZSl community when static images are considered. As a result, the conventional setting has been largely abandoned in the last two years and the ‘pure’ and ‘generalized’ settings become the norm; that is, there is no overlapping classes between the test classes and classes used to pretrain the visual feature extraction network; and both seen and unseen class samples are used for testing. The ZSL evaluation is only meaningful under this more rigorous and realistic setting. In summary, the experimental results presented in this paper are obtained under the wrong setting and the proposed model is not compared with a number of closely related ZSL models listed above, so it is not possible to judge how effective the proposed model is.