ON THE EFFECTIVENESS OF TASK GRANULARITY FOR TRANSFER LEARNING

1 INTRODUCTION . Common-sense video understanding entails fine-grained recognition of actions , objects , spatial temporal relations , and physical interactions , arguably well beyond the capabilities of current techniques . A general framework will need to discriminate myriad variations of actions and interactions . For example , we need to be able to discriminate similar actions that differ in subtle ways , e.g. , ’ putting a pen beside the cup ’ , ’ putting the pen in the cup ’ , or ’ pretending to put the pen on the table ’ . Similarly , one must be able to cope with diverse actors and object classes . Such generality and fine-grained discrimination are key challenges to video understanding . In contrast to current approaches to action recognition and captioning on relatively small corpora , with coarse-grained actions , this paper considers fine-grained classification and captioning tasks on large-scale video corpora . Training is performed on the Something-Something dataset ( Goyal et al . ( 2017 ) ) , with 174 fine-grained action categories and thousands of different objects , all under significant variations in lighting , viewpoint , background clutter , and occlusion . We describe a DNN architecture comprising a 2-channel convolutional net and an LSTM recurrent network for video encoding . A common encoding is shared for classification and caption generation . The resulting network performs better than baseline action classification results in Goyal et al . ( 2017 ) , and it gives impressive results on an extremely challenging fine-grained captioning task . We also demonstrate the quality of learned features through transfer learning from SomethingSomething features to a kitchen action dataset . The main contributions in the paper include : 1 . Explore the link between label granularity and feature quality : We exploit 3 levels of granularity in sth-sth-v2 , namely , action groups , action categories , and captions . Experiments show that more fine-grained labels yield richer features ( see Table 3 and Fig . 4 ) . 2 . Baselines for captioning on sth-sth-v2 data : We note that the captioning task is new for this dataset ; the original dataset did not provide captions . 3 . Captioning as a source task for transfer learning : We show that models trained to jointly perform classification and captioning learn features that transfer better to other tasks ( e.g. , see Fig . 4 ) . To the best of our knowledge , captioning has , to date , been used as a target task . Our results suggest that captioning is a powerful source task . 4 . 20bn-kitchenware : We introduce a new dataset , ostensibly for video transfer learning . 2 VIDEO DATA . Video action classification and captioning have received significant attention for several years , but progress has been somewhat slow in part because of a lack of large-scale corpora . Using web sources ( e.g. , YouTube , Vimeo and Flickr ) and human annotators , larger datasets have been collected ( e.g. , Kay et al . ( 2017 ) ; Monfort et al . ( 2018 ) ) , but they lack control over pose variations , motion and other scene properties that might be important for learning fine-grained models . More recently , crowd-sourced/crowd-acted data has emerged . This allows targeted video domains , action classes , and control over subtle differences between fine-grained action classes . The first version of Goyal et al . ( 2017 ) has 100 , 000 videos of human-object interactions , comprising 50 coarse-grained action groups , decomposed further into 174 related action categories . The videos exhibit significant diversity in viewing and lighting , objects and backgrounds , and the ways in which actors perform actions . Baseline performance in Goyal et al . ( 2017 ) was a correct action classification rate of 11.5 % , and 36.2 % on action groups . Zhou et al . ( 2017 ) report 42.01 % classification accuracy on SomethingSomething-V1 action categories . With our architecture we obtain validation accuracy of 38.8 % . Something-Something-V2 is larger , with 220 , 847 videos of the same 174 action categories . In addition , each video includes a caption that was authored and uploaded by the crowd actor . These captions incorporate the action class as well as descriptions of the objects involved . That is , the captions mirror the action template , but with the generic placeholder Something replaced by the object ( s ) chosen by the actor . As an example , a video with template action ’ Putting [ something ] on [ something ] ’ , might have a caption ’ Putting a red cup on a blue plastic box ’ . In a nutshell , this dataset provides different levels of granularity : 50 coarse-grained action groups , 174 fine-grained action categories , and even more fine-grained labeling via video captions . 3 VIDEO CLASSIFICATION AND CAPTIONING TASKS . Due to the prevalence of datasets like UCF-101 ( Soomro et al . ( 2012 ) ) , sports1M ( Karpathy et al . ( 2014 ) ) and more recently Kinetics ( Kay et al . ( 2017 ) ) , most research on action classification has focused on models for coarse-grained action classification . In extreme cases , action classes can be can be discriminated from a glimpse of the scene , often encoded in isolated frames ; e.g. , inferring “ soccer play ” from the presence of a green field . Even when motion is essential to the action , many existing approaches do well by encoding rough statistics over velocities , directions , and motion positions . Little work has been devoted to the task of representing details of object interactions or how configurations change over time . Image and video captioning have received increasing attention since the release of captioning data , notably , Microsoft COCO ( Chen et al . ( 2015 ) ) and MSR-VTT ( Xu et al . ( 2016 ) ) . One problem with existing captioning approaches is that many datasets implicitly allow models to “ cheat ” , e.g. , by generating phrases that are grammatically and semantically coherent , but only loosely related to the fine-grained visual structure . It has been shown , for example , that a language model trained on unordered , individual words ( e.g. , object-nouns ) predicted by a separate NN can compete with captioning model trained on the actual task end-to-end ( e.g. , Yao et al . ( 2015 ) ; Heuer et al . ( 2016 ) ) . Similarly , nearest neighbor methods have been surprisingly effective ( Devlin et al . ( 2015 ) ) . Captioning tasks , if designed appropriately , should capture detailed scene properties . Labels with more subtle and fine-grained distinctions would directly expose the ability ( or inability ) of a network to correctly infer the scene properties encoded in the captions . The captions provided with the Something-Something dataset are designed to be sufficiently fine-grained that attention to details is needed to yield high prediction accuracy . For example , to generate correct captions , networks need not only infer the correct actions , but must also learn to recognize different objects and their properties , as well as object interactions . 4 APPROACH . We use a modified encoder-decoder architecture with an action classifier applied to the video encoding ( see Fig . 1 ) . The decoder or classifier can be switched off , leaving pure classification or captioning models . One can also jointly train classification and captioning models . We train our two-channel architecture to solve four different tasks with different granularity levels : - Coarse-grained classification on action groups ( CG actions ) - Fine-grained classification on 174 action categories ( FG actions ) - Captioning on simplified objects ( SO captions ) - Fine-grained Captioning with full object placeholders ( Captions ) Finally , we investigate the effectiveness of the learned features for transfer learning . 4.1 MODIFIED VIDEO ENCODER-DECODER . The video encoder receives the input video v , and maps it to an embedded representation h. Conditioned on h , a caption decoder generates the caption c , and a classifier predicts the action category . The encoder processes the video with a two-channel convolutional architecture ( Fig . 2 ) . A spatial 2D-CNN and a spatiotemporal 3D-CNN are applied in parallel . The 3D-CNN features are used in lieu of a separate module to compute motion ( e.g. , optical flow ) features . The basic building block of each channel is a 3 × 3 × 3 ( 3 × 3 in 2D-CNN channel ) convolution filter with batchnorm and ReLU activation . To aggregate features across time , feature vectors from each channel are combined and fed to a 2-layer bidirectional LSTM . We average these features to get the video encoding , h. The action classifier applies a fully-connected layer to the encoding h , followed by a softmax layer . Training uses a cross-entropy loss over action categories . The caption decoder is a two-layer LSTM . Much like conventional encoder-decoder methods for video captioning ( Venugopalan et al . ( 2014 ) ; Donahue et al . ( 2015 ) ; Kaufman et al . ( 2016 ) and Sutskever et al . ( 2014 ) ) , our decoder generates captions using a softmax over vocabulary words , conditioned on previously generated words . The loss is the usual negative log-probability of the word sequence : losscaptioning = − N−1∑ i=0 log p ( wi+1|w≤i , h ; θ ) . ( 1 ) where wi denotes the ith word of the caption , h is the video encoding , and θ denotes model parameters . To optimize speed and memory usage during training , the length of captions generated by the decoder is fixed at 14 words1 . As is common for encoder-decoder networks , we train with teacher- 1Less than 1 % of Something-Something captions have more than 14 words . forcing ( Williams & Zipser ( 1989 ) ) . At test time , the input to the decoder at each time-step is the token generated at the previous time-step ( i.e. , no teacher forcing ) . The model is trained end-to-end for classification and captioning with a weighted sum of the classification and captioning losses , i.e. , loss = λ · lossclassification + ( 1− λ ) · losscaptioning ( 2 ) With λ = 1 or λ = 0 , we end up with pure classification and captioning tasks respectively . For other values of λ , they are trained jointly . The encoder is shared by the action classifier and the caption decoder . The experiments below also compare this joint training regime with models for which the encoder is trained on the classification loss or the captioning loss alone . 5 EXPERIMENTS . We train four two-channel models that are trained to solve coarse-grained and fine-grained classification and captioning tasks . In what follows we discuss different tasks in more details . 5.1 COARSE- AND FINE-GRAINED CLASSIFICATION . Something-Something provides coarse-grained categories ( action groups ) , each comprising a set of fine-grained actions . We trained a classification model on coarse-grained action groups , using the M ( 256-256 ) architecture , with accuracy of 57.6 % ( see Table 2 ( top-left ) ) . Table 1 reports the performance of our model for fine-grained classification . For the pure classification task ( with λ = 1 ) we consider different numbers of features produced by the 2D ( spatial ) CNN and the 3D ( spatiotemporal ) CNN . The total number of features is 512 in all cases . The results show that the model benefits from 2D and 3D features . The even split M ( 256-256 ) provides a good trade-off between performance and model complexity . We therefore use this model below , unless otherwise specified . Transferring between Coarse- and Fine-Grained Classification We can perform coarse-grained classification by mapping predictions from the fine-grained classifier onto the action groups . To this end we sum the probabilities of fine-grained actions belonging to each action group . Interestingly , the resulting accuracy on coarse-grained action groups increases to 62.5 % . This improvement suggests that fine-trained training provides higher quality features . We also examine to what extent coarse-grained performance accounts for fine-grained accuracy ; i.e , how better fine-grained performs compared to chance when conditioned on coarse-grain predictions . For example , conditioned on a predicted action group , if we select the most frequent action within the action group , finegrained test accuracy would be 23.8 % . One can also fix the coarse-grained model and train a linear classifier on top of its penultimate features . This yields test accuracy of 41.7 % , still 8.7 % below test performance for the corresponding architecture trained on the fine-grained task , further supporting our contention that training on fine-grained details yields richer features .

Summary: This paper describes a multi-task video classification and captioning model applied to a fine-grained object relationship video dataset, for a range of different classification and captioning tasks at different levels of granularity. This paper also creates a new video action dataset around kitchen objects and actions.  Finally, the paper includes an empirical study on both the multi-task performance and transfer learning performance between the two datasets considered.