Title: UNSEEN ACTION RECOGNITION WITH UNPAIRED ADVERSARIAL MULTIMODAL LEARNING

Abstract: In this paper, we present a method to learn a joint multimodal representation space that allows for the recognition of unseen activities in videos. We compare the effect of placing various constraints on the embedding space using paired text and video data. Additionally, we propose a method to improve the joint embedding space using an adversarial formulation with unpaired text and video data. In addition to testing on publicly available datasets, we introduce a new, large-scale text/video dataset. We experimentally confirm that learning such shared embedding space benefits three difficult tasks (i) zero-shot activity classification, (ii) unsupervised activity discovery, and (iii) unseen activity captioning.

1. INTRODUCTION:

Videos contain multiple data sources, such as visual, audio and text/caption data. Each data modality has distinct statistical properties capturing different aspects of the event. Current state-of-the-art activity recognition models (Carreira & Zisserman, 2017; Tran et al., 2017) only take visual data and class labels as input, which limits the information the model can learn from. For example, the sentence 'a group of men play basketball outdoors' contains rich information, such as 'outdoors' and 'group of men' compared to just the activity class of 'basketball.' Without multimodal learning, these models are unable to benefit from the additional information.

In this paper, we explore multimodal learning from video and language data, each having its own representation. Video data is represented as a sequence of images (spatio-temporal pixel data) while text is represented as a sequence of word embeddings (temporal data). Using a shared embedding space allows for learning the highly non-linear relationships between these modalities. The goal of learning such relationships is to capture similarities between concepts (e.g., basketball and volleyball both being sports with a ball) and further generalize to concepts not seen during training. To enable this generalization, we rely on pre-trained word embeddings as they capture relationships between all words in the English language (Mikolov et al., 2013) . By learning a joint representation space, we can capture these relationships, allowing generalization to unseen video activities.

To evaluate our approach, we test on several tasks based on recognizing activities not seen during training. First, we test on a zero-shot learning task, where we classify activity classes not seen during training. Our second task is unsupervised activity discovery, where no class information is provided during training, relying only on the relationship between text and videos to learn good embeddings. Finally, we evaluate on captioning videos with activities/words not seen during training.

Many existing approaches to both zero-shot and embedding space learning require paired data examples, which can be expensive to obtain. By taking advantage of adversarial learning (Goodfellow et al., 2014) , we are able augment our method with unpaired data (i.e., random sentences and random videos without any labels or correspondence) to further improve our learned shared embedding space.

In this paper, we design a method capable of learning joint video/language embedding spaces using both paired and unpaired data and experimentally confirm its benefit to three challenging tasks (i) zero-shot activity recognition, (ii) unsupervised activity discovery, and (iii) unseen activity captioning. We show that the use of unpaired, multimodal data allows learning a shared embedding space that generalizes to unseen data better than existing approaches.

2. RELATED WORKS:

Multimodal learning Previous approaches to multimodal learning have used Restricted Boltzmann Machines (Srivastava & Salakhutdinov, 2012) or log-bilinear models (Kiros et al., 2014) to learn distributions over sentences and images. Ngiam et al. (2011) designed an autoencoder that learns joint audio-video representations, however they are unable to train the model end-to-end, instead relying on greedy, layer-by-layer training and further do not evaluate the effect of using unpaired data. Similarly, Chandar et al. (2016) proposed a auto-encoder able to learn correlations between different view of images. Frome et al. (2013) describe a model that maps images and words to a shared embedding. However, these works either learn a joint embedding by concatenating the different features or require a triplet consisting of positive and negative matches and do not evaluate the effects of using unpaired data.

Text and vision:

Using both text and visual data has been studied for many tasks, such as image captioning (Karpathy et al., 2014; Johnson et al., 2016; Karpathy & Fei-Fei, 2015) or video captioning (Krishna et al., 2017; Zhou et al., 2018; Xu et al., 2018) . Other works have explored the use of textual grounding for image/video retrieval (Gupta & Mooney, 2010; Rohrbach et al., 2016; Miech et al., 2017; Hendricks et al., 2017) . There have been various models proposed to learn a fixed text embedding space with mappings from video features into this embedding space (Guadarrama et al., 2013; Otani et al., 2016; Song et al., 2016; Wang et al., 2016; Xian et al., 2016) . However, these works all learn only a single directional mapping (e.g., only mapping from visual to text), only learn with paired text/image samples and require additional data in the form of positive/negative pairs.

Learning with unpaired data Recently, there have been many works taking advantage of variational autoencoders (VAEs) (Kingma & Welling, 2014) or generative adversarial networks (GANs) (Goodfellow et al., 2014) to learn mappings between unpaired samples. CycleGan (Zhu et al., 2017) uses a cycle-consistency loss (i.e., the ability to go from a sample in one domain to a second domain then back to the source) to learn unpaired image translation (e.g., image to sketch). Other works learn many-to-many mappings between images (Almahairi et al., 2018) or use two GANs to map between domains (Yi et al., 2017 ). An autoencoder with shared weights for both domains has been used to learn a latent space for image-to-image translation (Liu et al., 2017) . However, these works all focus on learning mappings between unpaired data of the same modalitiy (e.g. image to image), where the data is from the same underlying distribution. Our work focuses on learning from different modalities with very different distributions.

Zero-shot activity recognition Many previous works have studied zero-shot activity recognition. Common approaches include using attributes (Liu et al., 2011; Palatucci et al., 2009; Romera-Paredes & Torr, 2015) or word embeddings (Xu et al., 2015; 2017; Norouzi et al., 2013; Socher et al., 2013; Kodirov et al., 2017) or learning a similarity metric (Zhang & Saligrama, 2015; Chopra et al., 2005) . Some works have explored using adversarial losses on the latent space (Chen et al., 2018) or use GANs to generate features for unseen classes (Xian et al., 2018) or used auto-encoders (Wang et al., 2017) .

Our work differs from these previous works in three key ways: (1) we show the benefit of additional data augmentation using unpaired samples, (2) we experimentally compare the use of the embedding space for both zero-shot recognition, unseen recognition, and unseen video captioning, and (3) we learn a shared, multimodal embedding space with bi-directional mappings in an end-to-end fashion.

3. METHOD:

To enable learning of a joint embedding space, we use a deep autoencoder architecture. Our model consists of 4 neural networks:

Video Encoder E

V : v → z v Video Decoder G V : z → v Text Encoder E T : t → z t

Text Decoder G T : z → t where v is a sequence of video data and t is a sentence (sequence of words). z is the embedded data in the shared space that we are learning. The encoders learn a compressed representation of the video or text while the decoders are trained to reconstruct the input: As both text and video data are sequences, they often have different lengths. A joint embedding space requires that the features from both modalities have the same dimensions. Given a text representation of length L and a video representation of length T , we need to obtain a fixed-size representation. To learn a fixed-dimensional embedding, there are many choices for the encoder/decoder architecture, such as temporal pooling (Ng et al., 2015) , attention (Piergiovanni et al., 2017) or RNNs (Krishna et al., 2017) . We chose temporal attention filters (Piergiovanni et al., 2017) as they learn a mapping from any length input to a N -dimensional vector and have been shown to outperform temporal pooling and RNNs on activity recognition tasks.

L recons (v, t) = ||G V (E V (v)) -v|| 2 + ||G T (E T (t)) -t|| 2 (1)

The attention filters consists of N Gaussians, each learning 2 parameters: a center ĝ and width σ. The filters are determined by:

EQUATION

The weights are applied by matrix multiplication with the video or text sequence: v = F v. Additionally, we can learn a transposed version of these filters to reconstruct the input: v = F T v . To reconstruct the input, the decoders learn their own parameters with the tensors transposed, resulting in the matching output size. The architecture of the encoders is shown in Fig. 1 .

3.1. LEARNING A JOINT EMBEDDING SPACE:

To learn a joint representation space, we minimize the L 2 distance between the embeddings of a pair of text and video (shown in Fig. 2 (a)):

L joint (v, t) = ||E V (v) -E T (t)|| 2 (3)

This forces the joint embeddings to be similar and when combined with the reconstruction loss, ensures that the representations can still reconstruct the input. Mapping from text to video using the cross-domain (Eq. 4) loss. 

V T z E v E T L joint L recons G T G v V T z E v E T G T G v (a) (b) V z L 2 (c) L recons L cross L cross

V T z E v E T G T G v D v ( ) = 1 D v ( ) = 0 D T ( ) = 0 D T ( ) = 1 D z ( ) = 0 D z ( ) = 1

EQUATION

Additionally, we can use a 'cycle' loss to map from video to text and back to video. Note that while the previous losses all require paired examples, this loss does not.

L cycle (v, t) = ||G T (E V (G V (E T (t)))) -t|| 2 + ||G V (E T (G T (E V (v)))) -v|| 2 (5)

To train the model to learn a joint embedding space, we minimize

EQUATION

where α i are hyper-parameters weighting the various loss components.

3.2. SEMI-SUPERVISED LEARNING WITH UNPAIRED DATA:

To learn using unpaired data (i.e., unrelated text and video), we use an adversarial formulation. We treat the encoders and decoders as generator networks. We then learn an additional 3 discriminator networks which constrain the generators and embedding space and force the encoders and decoders to be consistent:

(1) D z which learns to discriminate between latent text representations and latent video representations. Conceptually, this constrains the learned embeddings to appear to be from the same distribution.

(2) D V which learns to discriminate between true video data and generated video data G V (E T (t)).

(3) D T which learns to discriminate between true text data and generated text data, G T (E V (v)).

Given these discriminators, we minimize the following losses:

EQUATION

Using the discriminators, we can train the generators (encoders and decoders) to minimize the following loss based on unpaired data:

EQUATION

Note that in this formulation, v and t are not paired.

These networks are trained in an adversarial setting. For example, for the text-to-video generator (i.e., v = G V (E T (t)) and video discriminator, D V , we optimize the following minimax equation:

EQUATION

This equation is similarly applied for video-to-text. For learning the embedding space with the video and text encoders, E V , E T and the discriminator D z , we optimize the following minimax equation:

EQUATION

As training GANs can be unstable, we developed an method to allow for more stable training of the joint embedding space, shown in Algorithm 1. We initialize both the generator and discriminator networks by training only on paired data. After several iterations of this, we train with both unpaired and paired data. We found the initial training of the generators and discriminators was important for stability, without it the loss often diverges and the learned embedding did not generalize to unseen activities.

Algorithm 

4. EXPERIMENTS:

We compare our various approaches on different tasks (i) zero-shot activity recognition, (ii) unsupervised activity discovery and (iii) unseen activity captioning. These tasks test various combinations of our encoders and decoders and how well the shared representation generalizes to unseen data. We experimentally confirm the benefits of our methods using multiple public datasets: AcitivtyNet (Heilbron et al., 2015; Krishna et al., 2017) , HMDB (Kuehne et al., 2011) , UCF101 (Soomro et al., 2012) , and MLB-YouTube (Piergiovanni & Ryoo, 2018) .

4.1. IMPLEMENTATION/TRAINING DETAILS:

We implement our models in PyTorch. For the per-segment video CNN, we use I3D (Carreira & Zisserman, 2017) to obtain a 1024 × T video representation. We trained a version of I3D based on Kinetics-600, but withheld all classes that appear in ActivityNet, HMDB51, or UCF101 so that the classes are truly unseen. This resulted in a training set with 478 classes and 278k videos. Since generating videos is an extremely challenging task, the video autoencoders start with and generate the I3D feature. We use GloVe word embeddings (Pennington et al., 2014) to obtain a language representation. We set N = 4 for the temporal attention filters and apply 4 fully connected layers. These layers are followed by L 2 normalization so that the embedding space has unit length (Tygert et al., 2015) . We train for 200 epochs and use stochastic gradient descent with momentum to minimize the loss function with a learning rate of 0.01. After every 50 epochs, we decay the learning rate by a factor of 10. When training in the adversarial setting (e.g., Algorithm 1), we initialize the network training for 50 epochs on paired data followed by 200 on the paired + unpaired data.

Baselines For baselines, we compare to a fixed-text embedding space, were only a mapping from video data into the text embedding space is learned (e.g., (Otani et al., 2016) ). We also compare to learning a shared embedding space with the 'recons' (Eq. 1) and 'cross' (Eq. 4) terms (e.g., (Ngiam et al., 2011) ).

4.2. ZERO-SHOT ACTIVITY RECOGNITION:

Zero-shot activity recognition is the problem of classifying a video that belongs to a class not seen during training. Given training videos of seen classes together with paired text descriptions, our approach learns a shared embedding that maps videos/texts from multiple seen classes. The objective is to classify videos of unseen classes solely based on the learned embedding space and the text samples.

We use the ActivityNet captions (Krishna et al., 2017) dataset to learn the joint embedding space, as this dataset has both sentence descriptions for each video as well as activity classes. We randomly choose a set of K activity classes and withhold all videos/sentences belonging to those classes during training. For testing, we take a subset of sentences for the unseen classes and map the sentences into the joint embedding space, z t = E T (t). We then map the videos into the space, z v = E V (v) and use nearest neighbors to match each video (z v ) to text (z t ), using the class of the nearest sentence as the classification for the video.

In Table 1 , we compare the effect of the various loss components. For each method, we run 10 trials each with a different set of unseen activity classes and average the results. We find that previous methods of learning a fixed language embedding (e.g., (Socher et al., 2013; Xu et al., 2015; 2017) ) are significantly outperformed by learning a joint representation. Previous methods learning embedding spaces without the 'joint' term (e.g. (Ngiam et al., 2011 )), we found yield nearly random performance on these tasks, suggesting that forcing the representations to match in the embedding space is important. Further, adding the reconstruction, cross-domain, and cycle losses all improve performance. We also compare to a standard triplet loss (e.g., (Guadarrama et al., 2013) ) which requires additional positive/negative samples. We find that the triplet loss outperforms the 'joint' loss, but is surpassed by adding the 'cycle' and 'cross' terms, which use less data.

We also compare the various components of the adversarial loss. We compare to having just the adversarial loss on the embedding space (D z ), like (Chen et al., 2018) , and compare just the adversary on the generated videos/sentences. We find that the use of all terms in important for performance.

To obtain unpaired data, we use the sentence descriptions from the Charades (Sigurdsson et al., 2016) dataset, which contains many activities in a home setting. The unpaired video data is sampled from HMDB and UCF101. While previous works such as (Ngiam et al., 2011) can support learning with unpaired data, we find that the adversarial loss provides better results than just the 'cycle' and 'recons' terms, and further improves over training with just paired data.

In Table 2 , we compare our approach to previous zero-shot learning methods on HMDB and UCF101. The paired training data for these models is drawn from ActivityNet with any classes belonging to HMDB or UCF101 withheld. The unpaired text data is sampled from Charades and the video data comes from either HMDB (when testing on UCF101) or UCF101 (when testing on HMDB). As HMDB and UCF101 have no text descriptions, we created a sentence description for each activity class (included in Appendix B). We find that our joint embedding space outperforms the previous approaches on these datasets and unpaired adversarial learning further improves our performance. Table 2 : Results on HMDB51 and UCF101 compared to previous state-of-the-art results. We find that learning a shared embedding space is beneficial and that augmented with unpaired data provides the best results.

Feature HMDB51 UCF101:

SJE (Akata et al., 2015) IDT 12.0 ± 2.6 9.3 ± 1.7 ConSe (Norouzi et al., 2013) IDT 15.0 ± 2.7 11.6 ± 2.1 ZSECOC (Qin et al., 2017) IDT 22.6 ± 1.2 15.1 ± 1.7 Semantic Embedding (Xu et al., 2015) IDT 21.2 ± 3.0 18.6 ± 2.2 Manifold Ridge Regression (Xu et al., 2017) IDT 24.1 ± 3.8 22.1 ± 2.5 SAE (Kodirov et al., We explore different strategies for obtaining unpaired data. For these experiments, we keep a fixed set of paired text and videos and explore adding various sources of unpaired data. We compare (i) 10k random sentences from Wikipedia, (ii) 10k random dictionary definitions, and (iii) 10k dictionary He is an aggressive third baseman and he can really play over there you know. He definitely takes pride in his defense as well.

He got right on top of that pitch, Pederson, and shot and way out of here. Three-run blast.

That has been a feat in this series for both teams, nobody is hitting with two strikes. That's how good the pitching has been.

They would suspend him at the beginning of next year as opposed to for a game during this World Series. definitions of verbs. We also compare adding 10k random videos from YouTube as additional video data. We show our results using 10 unseen classes in ActivityNet in Table 4 . We find that augmenting with similar unpaired data improves performance, while using irrelevant data harms performance. We find that dictionary verb definitions improve performance the most, as they capture important semantic information regarding the activities we are learning. The use of additional video data is further beneficial.

4.4. UNSUPERVISED ACTIVITY DISCOVERY:

To further evaluate our joint embedding space, we conducted experiments on unsupervised activity discovery. For this task, we expanded the MLB-YouTube dataset (Piergiovanni & Ryoo, 2018) by densely annotating the videos with a transcription of the announcers' commentary, resulting in approximately 50 hours of aligned text and video. Examples of this data are shown in Fig. 4 . The MLB-YouTube dataset is designed for fine-grained activity recognition, where the difference between activities is quite small. Additionally, these captions only roughly describe what is happening in the video, and often contain unrelated stories or commentary on a previous event, making this a challenging task. The dataset will be made publicly available. To train our joint embedding space, we split each baseball video into 30 second intervals and use the corresponding text as paired data, resulting in 6,089 paired training samples.

We evaluate our joint embedding using the segmented videos from MLB-YouTube. For each video, we compute the embedded features and apply k-means clustering (k = 8, the number of classes). Each segmented video is assigned to a cluster and votes for the cluster label based on its ground truth label. We use that cluster assignment for classification on the MLB-YouTube test set. We report our findings in Table 5 . As a baseline, we cluster I3D features pre-trained on Kinetics. We find that our methods improve the embedding space. However, we note that when using unpaired data from Charades, the performance drops. This is likely due to Charades data being very different from MLB-YouTube data. We collected additional captions and baseball videos to augment the MLB-YouTube dataset, and confirmed that unpaired data helps when it is from a similar distribution.

In Table 6 we compare our various methods for unsupervised activity discovery on HMDB and UCF101. Here, we learn a joint embedding space using the ActivityNet videos and captions. We withhold any videos belonging to a class in HMDB or UCF101. Unlike MLB-YouTube, on these datasets, we find that using the unpaired training with Charades further improves performance. This confirms that when the additional data is similar to the target dataset, using the adversarial learning setting further improves the embedding space.

4.5. UNSEEN VIDEO CAPTIONING:

As our model learns a bi-directional mappings, we can apply our model to generate video captions.

Existing video captioning models are unable to create realistic captions for unseen activities, as without training data they do not know the words to describe the video. Given a video, v, we can generate a caption by mapping the video to text t = G T (E V (v)). For each word, we then use nearest neighbors matching with the GloVe embeddings to obtain the words to form a sentence. In Table 7 , we report the commonly used METEOR (Banerjee & Lavie, 2005) and CIDEr (Vedantam et al., 2015) scores of our various models, measured with the unseen classes from the ActivityNet dataset. We find that learning a joint representation is beneficial and using unpaired samples further improves the task. In Fig. 5 , we show example captioned videos. Note that this task is extremely challenging, as it requires the model to generate captions using activity words (e.g., basketball) not seen during training.

5. CONCLUSION:

We proposed an approach to learn a joint language/text embedding space using various constraints. We further extended the model to be able to learn with unpaired video and text data using an adversarial formulation. We experimentally confirmed that learning such an embedding space benefits three difficult tasks (i) zero-shot activity classification, (ii) unsupervised activity discovery, and (iii) unseen activity captioning. We find that the use of related unpaired data is beneficial. We presented several strategies for obtaining unpaired data and confirmed the benefit of adding additional, relevant unpaired data. To confirm that temporal attention is beneficial, we compare different forms of temporal pooling (i) max-pooling, (ii) sum-pooling, (iii) LSTM, and (iv) temporal attention filters (Piergiovanni et al., 2017) . In Table 8 , we compare these temporal pooling methods learning the joint embedding space. We confirm that using the temporal attention filters performs best.

A.2 COMPARISON OF DIFFERENT RATIOS OF PAIRED AND UNPAIRED DATA:

We compare different ratios of paired and unpaired data to see how much paired data we require and how much unpaired data is beneficial. For these experiments, we use all the loss terms (i.e., what provided us the best results). Note that in these experiments, the total number of samples was the same for each method (40k examples) so that we can directly compare the effects of unpaired data vs. paired data. Thus not all the available data was used.

In Table 9 , we show the results. We find that using no paired data results in nearly random performance, but using using some paired data greatly improves the embedding space. The model using 100% paired data performs best, as all the others are using less overall paired data.

We also compare augmenting our 40k paired training samples with different amounts of unpaired data. Since UCF101 and HMDB only have 13k and 7k examples, to get up to 60k samples, we also use videos from the Kinetics dataset (Carreira & Zisserman, 2017) . The results, shown in Table 10 , show that adding the initial 10k samples is most beneficial, while additional samples do not seem to meaningfully improve results. However, due to our training method where each batch consists of 50% paired data and 50% unpaired data, the additional unpaired data does not harm results either. In Fig. 6 , we compare t-SNE embeddings of the fixed text representation and our joint embedding space. This visually shows that learning a joint embedding space gives more distinct class distributions.

A.3.1 MLB-YOUTUBE CAPTIONS:

As a baseline for the MLB-YouTube captions dataset, we compared several different models for standard video captioning (i.e., all activity classes are seen). This task is quite challenging compared to other datasets as the announcers commentary is not always a direct description of the current events. Often the announcers tell loosely related stories and attempt to describe events differently each time to avoid repetition. Additionally, the descriptions contain on average 150 words for each 30 second interval and current captioning approaches usually only trained and tested on 10-20 word sentences. Due to these factors, this task is quite challenging the standard evaluation metrics do not account for these factors. In Table 11 , we report our results on this task.

B HMDB AND UCF101 SENTENCES:

For the HMBD and UCF101 datasets, we created sentences to describe each activity class. Our sentences descriptions are included in this appendix.

These sentences are written for each activity class (by randomly selecting a single video per class) and are shared for all instances of the activity. Depending on what video was randomly chosen for the class, some sentences describe the actor as a 'man', 'woman', or 'person' which could confuse the model. Ideally, the CNN embedding needs to learn to ignore the impact of such pronoun changes.

We conducted experiments comparing randomly replacing the pronouns to determine if there was any bias introduced by the pronouns. We show the results in Table 12 . We find that the choice of pronouns does not impact performance, as our model automatically learns to focus more on verbs rather than pronouns. When examining the temporal attention filters on the sentences, we found that they placed very little 'attention' on the start of the sentence, where the pronoun usually is, suggesting that the pronoun has very little effect on the embedding space we learned.

:



Figures and Tables:

Figure 1: Illustration of the encoder models used to learn a joint embedding space. Videos and sentences are mapped into a low-dimensional space by applying CNNs and temporal attention. Then several fully-connected layers map into the joint embedding space. The decoders follow this same architecture with the weights transposed.

Figure 2: Visualization of several constrains on the shared embedding space. Circles are video data, ovals are reconstructed video. Diamonds are text data, and pentagons are reconstructed text. (a) The reconstruction (Eq. 1) and joint (Eq. 3) losses. (b) Mapping from text to video using the cross-domain (Eq. 4) loss.

Figure 3: Visualization of the adversarial formulation to learn with unpaired data. We create 3 discriminators, (1) D z learns to discriminate examples of text/video in the latent space. (2) D V learns to discriminate video generated from text compared to video. (3) D T learns to discriminate generated text compared to text.

Figure 4: Example video sequences from the MLB-YouTube dataset with the commentary caption. Top: Sentences that describe the occurring activities. Bottom: Sentences that do not describe the current activities.

Figure 5: Example captions for unseen activities. Left: Using a joint embedding space allows the model to correctly caption this video as basketball, despite never seeing an example of basketball during training. Right: An example of a caption for the unseen water-ski activity. Here the model fails to correctly caption the activity.

Figure 6: t-SNE mapping of (a) fixed text representation and (b) joint embedding with all paired losses for the MLB-YouTube dataset. The joint embedding space provides most distinct representations for the activities. Each color represents the activity class of the video (e.g., swing, hit, foul ball, etc.).

Comparison of various methods on ActivityNet for 5, 10, 20 or 50 unseen classes. These results are averaged over 10 trials where each trial has a different set of unseen activity classes.

Table content: <table><tr><td/><td colspan="4">5 Unseen 10 Unseen 20 Unseen 50 Unseen</td></tr><tr><td>Paired Data</td><td/><td/><td/><td/></tr><tr><td>Fixed Text Representation</td><td>41.9</td><td>38.4</td><td>29.4</td><td>15.6</td></tr><tr><td>Triplet Loss</td><td>56.8</td><td>44.9</td><td>38.8</td><td>23.3</td></tr><tr><td>joint</td><td>54.3</td><td>41.7</td><td>36.1</td><td>21.2</td></tr><tr><td>recons + cross</td><td>21.1</td><td>12.6</td><td>7.6</td><td>2.9</td></tr><tr><td>joint + recons</td><td>70.1</td><td>54.4</td><td>42.6</td><td>27.5</td></tr><tr><td>joint + recons + cycle</td><td>70.4</td><td>54.3</td><td>42.1</td><td>26.8</td></tr><tr><td>joint + recons + cross</td><td>72.6</td><td>55.4</td><td>43.2</td><td>27.8</td></tr><tr><td>joint + recons + cross + cycle</td><td>76.4</td><td>56.9</td><td>45.5</td><td>28.8</td></tr><tr><td colspan="4">With Adversarial Losses (joint + recons + cross + cycle + Adv.)</td><td/></tr><tr><td>+ Dz</td><td>78.4</td><td>57.3</td><td>45.9</td><td>29.1</td></tr><tr><td>+ Dv + Dt</td><td>77.2</td><td>57.2</td><td>45.6</td><td>28.9</td></tr><tr><td>+ Dz + Dv + Dt</td><td>79.7</td><td>58.3</td><td>46.5</td><td>29.7</td></tr><tr><td>Paired + Unpaired Data</td><td/><td/><td/><td/></tr><tr><td>recons + cycle</td><td>22.8</td><td>13.6</td><td>8.4</td><td>4.2</td></tr><tr><td>joint + recons + cycle</td><td>72.6</td><td>58.4</td><td>44.7</td><td>29.3</td></tr><tr><td>joint + recons + cross + cycle</td><td>73.4</td><td>59.1</td><td>45.3</td><td>29.2</td></tr><tr><td>All terms</td><td>82.5</td><td>60.4</td><td>46.2</td><td>30.1</td></tr></table>

Unseen activity recognition results on ActivityNet, HMDB51 and UCF101, evaluated by using both unseen and seen classes for the testing.

Table content: <table><tr><td/><td colspan="4">ActNet (10 unseen) ActNet (50 unseen) HMDB51 UCF101</td></tr><tr><td>Fixed Text Representation</td><td>55.7</td><td>46.8</td><td>24.5</td><td>26.8</td></tr><tr><td>Triplet Loss</td><td>57.7</td><td>48.5</td><td>27.6</td><td>29.8</td></tr><tr><td>joint</td><td>62.1</td><td>50.2</td><td>29.8</td><td>30.6</td></tr><tr><td>joint + recons</td><td>64.4</td><td>52.6</td><td>30.4</td><td>31.3</td></tr><tr><td>joint + recons + cross + cycle</td><td>69.6</td><td>58.5</td><td>35.6</td><td>36.5</td></tr><tr><td>Paired + Unpaired Data</td><td/><td/><td/><td/></tr><tr><td>All terms</td><td>71.6</td><td>65.9</td><td>38.9</td><td>42.1</td></tr></table>

Comparison of various source of unpaired data on ActivityNet with 10 unseen classes.

Table content: <table><tr><td>Paired Data</td><td>58.3</td><td>69.6</td></tr><tr><td>+ Random Wikipedia Sentences</td><td>55.8</td><td>66.4</td></tr><tr><td>+ Random Dictionary Defs.</td><td>56.3</td><td>68.2</td></tr><tr><td>+ Verb Dictionary Defs.</td><td>59.2</td><td>70.7</td></tr><tr><td>+ Random YouTube Videos</td><td>58.7</td><td>70.1</td></tr><tr><td>+ Verbs + Random Videos</td><td>60.3</td><td>71.2</td></tr></table>

Comparison of unsupervised activity classification on MLB-YouTube.Several men are playing basketballPeople are swimming in the ocean

Table content: <table><tr><td/><td/><td/><td colspan="3">Table 6: Comparison of unsupervised activity</td></tr><tr><td/><td colspan="2">Accuracy mAP</td><td colspan="2">classification on HMDB and UCF101.</td><td/></tr><tr><td>Baseline I3D features Fixed Text Representation</td><td>23.4 27.9</td><td>32.6 34.7</td><td/><td colspan="2">HMDB UCF101</td></tr><tr><td>Joint</td><td>34.5</td><td>41.6</td><td>Baseline I3D features</td><td>26.6</td><td>42.5</td></tr><tr><td>Joint + recons</td><td>37.9</td><td>43.7</td><td>Joint</td><td>32.4</td><td>57.7</td></tr><tr><td>Joint + recons + cycle</td><td>44.2</td><td>48.6</td><td>Joint + recons</td><td>33.5</td><td>59.0</td></tr><tr><td>Joint + recons + cross</td><td>43.7</td><td>49.3</td><td>All (paired)</td><td>34.6</td><td>59.5</td></tr><tr><td>All (paired)</td><td>48.4</td><td>51.2</td><td>All (+ unpaired)</td><td>34.9</td><td>59.9</td></tr><tr><td>All (+ unrelated unpaired)</td><td>39.7</td><td>43.9</td><td/><td/><td/></tr><tr><td>All (+ related unpaired)</td><td>49.1</td><td>54.3</td><td/><td/><td/></tr></table>

Comparison of several models for unseen activity captioning using the ActivityNet dataset, using METEOR and CIDEr scores. This evaluation was done on 10 unseen classes held out during training. Higher values are better.

Table content: <table><tr><td/><td colspan="2">METEOR CIDEr</td></tr><tr><td>Fixed Text Representation</td><td>3.64</td><td>8.95</td></tr><tr><td>Joint</td><td>4.21</td><td>9.23</td></tr><tr><td>All (paired)</td><td>5.31</td><td>11.21</td></tr><tr><td>All (paired + unpaired)</td><td>6.89</td><td>13.95</td></tr></table>

Comparison of temporal pooling methods for 5 unseen classes in the ActivityNet dataset.

Table content: <table><tr><td/><td>Accuracy</td></tr><tr><td>Max Pooling</td><td>23.4</td></tr><tr><td>Sum Pooling</td><td>24.1</td></tr><tr><td>LSTM</td><td>42.3</td></tr><tr><td>Temporal Attention Filters</td><td>55.2</td></tr></table>

Comparison of different ratios of paired and unpaired data methods for 5 unseen classes in the ActivityNet dataset.

Table content: <table><tr><td colspan="2">Paired/Unpaired Accuracy</td></tr><tr><td>100% / 0%</td><td>74.2</td></tr><tr><td>75% / 25%</td><td>73.2</td></tr><tr><td>50% / 50%</td><td>69.7</td></tr><tr><td>25% / 75%</td><td>62.6</td></tr><tr><td>0% / 100%</td><td>24.5</td></tr><tr><td>A ADDITIONAL EXPERIMENTS</td><td/></tr></table>

Comparison using 40k paired examples and varying amounts of unpaired samples for 5 unseen classes in the ActivityNet dataset.

Table content: <table><tr><td colspan="2">Unpaired Samples Accuracy</td></tr><tr><td>0</td><td>77.1</td></tr><tr><td>10k</td><td>82.4</td></tr><tr><td>20k</td><td>83.9</td></tr><tr><td>40k</td><td>83.6</td></tr><tr><td>60k</td><td>83.5</td></tr></table>

Comparison of several models for standard, seen video captioning using the MLB-YouTube dataset, using Bleu, METEOR and CIDEr scores. Higher values are better.

Table content: <table><tr><td/><td colspan="3">Bleu METEOR CIDEr</td></tr><tr><td colspan="2">Fixed Text Representation 0.12</td><td>0.04</td><td>0.12</td></tr><tr><td>Joint Representation</td><td>0.14</td><td>0.08</td><td>0.15</td></tr><tr><td>Joint + all paired</td><td>0.15</td><td>0.10</td><td>0.18</td></tr><tr><td>Joint + paired + unpaired</td><td>0.10</td><td>0.02</td><td>0.08</td></tr><tr><td>A.3 MLB-YOUTUBE CAPTIONS</td><td/><td/><td/></tr></table>

Comparison of various pronouns on the UCF101 dataset with 50 unseen classes.

Table content: <table><tr><td/><td>Accuracy</td></tr><tr><td>Baseline Sentences</td><td>33.4</td></tr><tr><td>All 'man'</td><td>33.2</td></tr><tr><td>All 'woman'</td><td>33.3</td></tr><tr><td>All 'person'</td><td>33.4</td></tr><tr><td>Random pronoun</td><td>33.4</td></tr></table>

Peer review (iclr2019):
 The paper attempts multimodal representation of video and text through an attention layer that allows weighted temporal pooling. The approach was tested on a collection of datasets including a newly introduced dataset, with the embedding and evaluated on three tasks: zero-shot classification, activity clustering and captioning. The paper is easy to read in general and the approach is scientifically sound. The need for an autoencoder in multimodal embedding has been proven for a variety of modalities including image-text, video-text, image-audio and video-audio. The contribution here is thus focused on temporal pooling through a learnt attention layer. However, the paper has a mix of tasks (3 tasks tested), without a conclusive understanding of the effect of the various loss functions on the learnt space. As the importance of various losses changes per task and dataset, the take-away message from the work is not obvious. Additionally, using unpaired data, proposed through a large-scale dataset is not obvious. The paper concludes that related data is required but how related data can be collected remains unexplored. The evaluation for the unsupervised discovery seems biased – 1NearestNeighbour is used as opposed to the more balanced mAP on ranking all test sequences as opposed to top-1. The collected dataset, which is a contribution of the paper is also poorly explained. The authors collect ‘dense annotations’ but it is not clear how many annotators were used, and what instructions they were given. The paper does not give examples of the collected annotations and how these differ from previous annotations available with the dataset (Fig 4). Appendix 1 concludes with sentences proposed to annotate UCF. These seem to apply per action and it’s not clear how they scale to the different instances, e.g. Action Surfing (85) is assigned to a male caption ‘a man is’, action 100 to a woman and action 96 to groups of people ‘people are riding’. This distinction is not obvious in all the instances of the dataset and such captioning might have significantly biased the results. Overall, there is little explanation of the decisions made to produce the comparative results. The novelty is limited to the attention pooling, which is not evaluated on all the three tasks.