dataesr/scientific-paragraphs-categorization huggingface dataset:
Imagine a surveillance or ambient assisted living system, where a dataset is already collected for a set of actions from a certain camera. Placing a new camera in the environment from a new viewpoint would require re-annotating data because the appearance of an action is drastically different when performed from different viewpoints  [26, 44, 95]. In fact, we observe that state-of-the-art action recognition networks fail drastically when trained and tested on distinct viewpoints. Specifically, we train the model of  [18]  on videos from a benchmark dataset NTU RGB+D  [69]  where people are facing the camera. When we test this network on other front-view (0 • ) videos, we obtain ∼80% accuracy. When we test with side-view (90 • ) videos, the performance drops to ∼40% (see  Section 4). This motivates us to study action recognition from novel viewpoints.
However, action classification requires certain motion patterns and semantics. It is challenging to generate synthetic data with action labels  [10]. Typical MoCap datasets  [1] , targeted for pose diversity, are not suitable for action recognition due to lack of clean action annotations. Even if one collects a MoCap dataset, it is still limited to pre-defined set of categories. In this work, we propose a new, efficient and scalable approach for generating synthetic videos with action labels from the target set of categories. We employ a 3D human motion estimation method, such as HMMR  [28]  and VIBE  [31] , that automatically extracts the 3D human dynamics from a single-view RGB video. The resulting sequence of SMPL body  [50]  pose parameters are then combined with other randomized generation components (e.
g. viewpoint, clothing) to render diverse complementary training data with action annotations. Figure  1  presents an overview of our pipeline. We demonstrate the advantages of such data when training spatio-temporal CNN models for (i) action recognition from unseen viewpoints and (ii) training with one-shot real data. We boost performance on unseen viewpoints from 53. 6% to 69. 0% on NTU, and from 49. 4% to 66. 4% on UESTC dataset by augmenting limited real training data with our proposed SURREACT dataset. Furthermore, we present an in-depth analysis about the importance of action relevant augmentations such as diversity of motions and viewpoints, as well as our non-uniform frame sampling strategy which substantially improves the action recognition performance. Our code and data will be available at the project page 1.
Synthetic datasets built for these tasks, such as the recent SURREAL dataset  [83] , however, do not provide action labels. Among previous works that focus on synthetic human data, very few tackle action recognition  [10, 43, 66]. Synthetic 2D human pose sequences  [53]  and synthetic point trajectories  [65, 67, 25]  have been used for viewinvariant action recognition. However, RGB-based synthetic training for action recognition is relatively new, with  [10]  being one of the first attempts. De Souza et al. [10]  manually define 35 action classes and jointly estimate real categories and synthetic categories in a multitask setting. However, their categories are not easily scalable and do not necessarily relate to the target set of classes. Unlike  [10] , we automatically extract motion sequences from real data, making the method flexible for new categories.
Recently,  [62]  has generated the Vir-tualHome dataset, a simulation environment with programmatically defined synthetic activities using crowdsourcing. Different than our work, the focus of  [62]  is not generalization to real data. Most relevant to ours,  [43]  generates synthetic training images to achieve better performance on unseen viewpoints. The work of Liu et al. [43]  is an extension of  [66]  by using RGB-D as input instead of depth only. Both works formulate a frame-based pose classification problem on their synthetic data, which they then use as features for action recognition. These features are not necessarily discriminative for the target action categories. Different than this direction, we explicitly assign an action label to synthetic videos and define the supervision directly on action classification.
Cross-view action recognition. Due to the difficulty of building multi-view action recognition datasets, the standard benchmarks have been recorded in controlled environments. RGB-D datasets such as IXMAS  [87] , UWA3D II  [64]  and N-UCLA  [85]  were state of the art until the availability of the large-scale NTU RGB+D dataset  [69]. The size of NTU allows training deep neural networks unlike previous datasets. Very recently, Ji et al. [24]  collected the first large-scale dataset, UESTC, that has a 360 • coverage around the performer, although still in a lab setting. Since multi-view action datasets are typically captured with depth sensing devices, such as Kinect, they also provide an accurate estimate of the 3D skeleton. Skeleton-based cross-view action recognition therefore received a lot of attention in the past decade  [30, 45, 46, 47, 93].
3D human motion estimation In order to generate a synthetic video with graphics techniques, we need to have a sequence of articulated 3D human body models. We employ the parametric body model SMPL  [50] , which is a statistical model, learned over thousands of 3D scans. SMPL generates the mesh of a person given the disentangled pose and shape parameters. The pose parameters (R 72 ) control the kinematic deformations due to skeletal posture, while the shape parameters (R 10 ) control identity-specific deformations such as the person height. We hypothesize that a human action can be captured by the sequence of pose parameters, and that the shape parameters are largely irrelevant (note, this may not necessarily be true for human-object interaction categories). Given reliable 3D pose sequences from action recognition video datasets, we can transfer the associated action labels to synthetic videos.
SURREACT dataset components In this section, we give details on our synthetic dataset, SURREACT (Synthetic hUmans foR REal ACTions). We follow  [83]  and render 3D SMPL sequences with randomized cloth textures, lighting, and body shapes. We animate the body model with our automatically extracted pose dynamics as described in the previous section. We explore various motion augmentation techniques to increase intra-class diversity in our training videos. We incorporate multi-person videos which are especially important for two-people interaction categories. We also systematically sample from 8 viewpoints around a circle to perform controlled experiments. Different augmentations are illustrated in Figure  2  for a sample synthetic frame. Visualizations from SURRE-ACT are further provided in Figure  3.
Each generated video has automatic ground truth for 3D joint locations, part segmentation, optical flow, and SMPL body  [50]  parameters, as well as an action label, which we use for training a video-based 3D CNN for action classification. We use other ground truth modalities as input to action recognition as oracle experiments (see Table  A. 5). We further use the optical flow ground We modify the joint angles with additive noise on the pose parameters for motion augmentation. We systematically change the camera position to create viewpoint diversity. We sample from a large set of body shape parameters, backgrounds, and clothing to randomize appearances. truth to train a flow estimator and use the segmentation to randomly augment the background pixels in some experiments. Our new SURREACT dataset differs from the SUR-REAL dataset  [83]  mainly by providing action labels, exploring motion augmentation, and by using automatically extracted motion sequences instead of MoCap recordings  [1].
Moreover, Varol et al. [83]  do not exploit the temporal aspect of their dataset, but only train CNNs with single-image input. We further employ multi-person videos and a systematic viewpoint distribution. Motion augmentation. Automatic extraction of 3D sequences from 2D videos poses an additional challenge in our dataset compared to clean high-quality MoCap sequences. To reduce the jitter, we temporally smooth the estimated SMPL pose parameters by weighted linear averaging. SMPL poses are represented as axisangle rotations between joints. We convert them into quaternions when we apply linear operations, then normalize each quaternion to have a unit norm, before converting back to axis-angles. Even with this processing, the motions may remain noisy, which is inevitable given that monocular 3D motion estimation is a difficult task on its own.
This can be adjusted according to the target test setting. Backgrounds. Since we have access to the target real dataset where we run pose estimation methods, we can extract background pixels directly from the training set of this dataset. We crop from regions without the person to obtain static backgrounds for the NTU and UESTC datasets. We experimentally show the benefits of using the target dataset backgrounds in the Appendix (see Table  A. 6). For Kinetics experiments, we render human bodies on top of unconstrained videos from nonoverlapping action classes and show benefits over static backgrounds. Note that these background videos might also include human pixels.
This can be seen as a data augmentation technique, as well as a way to capture long-term cues. At test time, we sample several 16-frame clips and average the softmax scores. If we test the uniform case, we sample non-overlapping consecutive clips with sliding window. For the non-uniform case, we randomly sample as many non-uniform clips as the number of sliding windows for the uniform case. In other words, the number of sampled clips is proportional to the video length. More precisely, let T be the number of frames in the entire test video, F be the number of input frames per clip, and S be the stride parameter. We sample N clips where N = (T -F )/S) + 1. In our case F = 16, S = 16. We apply sliding window for the uniform case. For the non-uniform case, we sample N clips, where each clip is an ordered random (without replacement) 16-frame subset from T.
We observe that it is important to train and test with the same sampling scheme, and keeping the temporal order is important. More details can be found in Appendix B. 5. Synth+Real. Since each real video is augmented multiple times (e. g. 8 times for 8 views), we have more synthetic data than real. When we add synthetic data to training, we balance the real and synthetic datasets such that at each epoch we randomly subsample from the synthetic videos to have equal number for both real and synthetic. We minimize the cross-entropy loss using RMSprop  [78]  with mini-batches of size 10 and an initial learning rate of 10 -3 with a fixed schedule. Color augmentation is used for the RGB stream. Other implementation details are given in Appendix A.
Datasets and evaluation protocols We briefly present the datasets used in this work, as well as the evaluation protocols employed. NTU RGB+D dataset (NTU). This dataset  [69]  captures 60 actions with 3 synchronous cameras (see Figure  5 ). The large scale (56K videos) of the dataset allows training deep neural networks. Each sequence has 84 frames on average. The standard protocols  [69]  report accuracy for cross-view and cross-subject splits. The cross-view (CV) split considers 0 • and 90 views as training and 45 • view as test, and the same subjects appear both in training and test. For the cross-subject (CS) setting, 20 subjects are used for training, the remaining 20 for test, and all 3 views are seen at both training and test. We report on the standard protocols to be able to compare to the state of the art (see Table  8 ).
However, we introduce a new protocol to make the task more challenging. From the cross-subject training split that has all 3 views, we take only 0 • viewpoint for training, and we test on the 0 • , 45 • , 90 • views of the cross-subject test split. We call this protocol crossview-subject (CVS). Our focus is mainly to improve for the unseen and distinct view of 90 •. UESTC RGB-D varying-view 3D action dataset (UESTC). UESTC is a recent dataset  [24]  that systematically collects 8 equally separated viewpoints that cover 360 • around a person (see Figure  5 ). In total, the dataset has 118 subjects, 40 actions categories, and 26500 videos of more than 200 frames each. This dataset allows studying actions from unusual views such as behind the person. We use the official protocol Cross View I (CV-I), suitable for our task, which trains with 1 viewpoint and tests with all other 7 for each view.
The final performance is evaluated as the average across all tests. For completeness, we also report the Cross View II (CV-II) protocol that concentrates on multi-view training, i. e. , training with even viewpoints (FV, V2, V4, V6) and testing with odd viewpoints (V1, V3, V5, V7), and vice versa. One-shot Kinetics-15 dataset (Kinetics-15). Since we wish to formulate a one-shot scenario from in-thewild Kinetics  [29]  videos, we need a pre-trained model to serve as feature extractor. We use a model pre-trained on Mini-Kinetics-200  [88] , a subset of Kinetics-400. We define the novel classes from the remaining 200 categories which can be described by body motions. This procedure resulted in a 15-class subset of Kinetics-400: bending back, clapping, climbing a rope, exercising arm, hugging, jogging, jumpstyle dancing, krumping, push up, shaking hands, skipping rope, stretching arm, swinging legs, sweeping floor, wrestling.
In particular, we use a ResNet-18 architecture  [20]. We experiment with both HMMR and VIBE to use SMPL parameters as input, as well as Kinect joints provided by the NTU dataset  for comparison. Figure  6  reports the results of various pose representations against the performance of synthetic renderings for three test views. We make several observations: (i) Removing viewpoint-dependent factors, e. g. , pose parameters over joints, degrades performance on seen viewpoint, but consistently improves on unseen viewpoints; (ii) Synthetic video renderings from all viewpoints significantly improve over raw motion parameters for the challenging unseen viewpoint; (iii) VIBE outperforms HMMR; (iv) Both RGB-based motion estimation methods are competitive with the depth-based Kinect joints. We note the significant boost with renderings (45.
For this experiment, we use VIBE pose estimates and compare ST-GCN with the 2D ResNet architecture in Table  3. Although we observe improvements with using ST-GCN (29. 0% vs 36. 2%), the synthetic renderings provide significantly better generalization to the unseen 90 • view (45. 3%). Amount of data. In the NTU CVS training split, we have about 220 sequences per action. We take subsets with {10, 30, 60, 100} sequences per action, and train the three scenarios: Real, Synth, Synth+Real, for each subset. Figure  7  plots the performance versus the amount of data for these scenarios, for both RGB and Flow streams. We observe the consistent improvement of complementary synthetic training, especially for unseen viewpoints. We also see that it is more effective to use synthetic data at a given number of sequences per action.
For example, on the 90 • viewpoint, increasing the number of sequences from 100 to 220 in the real data results only in 4. 6% improvement (49. 0% vs 53. 6%, Real), while one can synthetically augment the existing 100 sequences per action and obtain 64. 7% (Synth+Real) accuracy without spending extra annotation effort. View diversity. We wish to confirm that the improvements presented so far are mainly due to the viewpoint variation in synthetic data. The "Synth(v=0 • ) + Real" plot in Figure  7  indicates that only the 0 • viewpoint from synthetic data is used. In this case, we observe that the improvement is not consistent. Therefore, it is important to augment viewpoints to obtain improvements. Moreover, we experiment with having only ±45 • or ±90 • views in the synthetic-only training for 60 sequences per action.
Comparison with the state of the art In the following, we employ the standard protocols for UESTC and NTU datasets, and compare our performance with other works. Tables  6  and 7  compare our results to the state-of-the-art methods reported by  [24]  on the recently released UESTC dataset, on CV-I and CV-II protocols. To augment the UESTC dataset, we use the VIBE motion estimation method. We outperform the RGB-based methods JOULE  [23]  and 3D ResNeXt-101  [18]  by a large margin even though we use a less deep 3D ResNet-50 architecture. We note that we have trained the ResNeXt-101 architecture  [18]  with our implementation and obtained better results than our ResNet-50 architecture (45. 2% vs 36. 1% on CV-I, 82. 5% vs 76. 1% on CV-II). This contradicts the results reported in  [24]. We note that a first improvement can be attributed to our non-uniform frame sampling strategy.
Therefore, we report our uniform real baseline as well. A significant performance boost is later obtained by having a mixture of synthetic and real training data. Using only RGB input, we obtain 17. 0% improvement on the challenging CV-I protocol over real data (66. 4 vs 49. 4). Using both RGB and flow, we obtain 44. 1% improvement over the state of the art (76. 1 vs 32. 0). We also report on the even/odd test splits of the CV-II protocol that have access to multi-view training data. The synthetic data again shows benefits over the real baselines. Compared to NTU, which contains object interactions that we do not simulate, the UESTC dataset focuses more on the anatomic movements, such as body  and vice versa (right). We present the results on both splits and their average for the RGB and Flow streams, as well as the RGB+Flow late fusion.
Real+Synth training consistently outperforms the Real baseline. push up stretching arm swinging legs krumping jumpstyle dancing jogging exercising arms climbing a rope clapping Fig. 8 : Sample video frames from the one-shot Kinetics-15 dataset. We provide side-by-side illustrations for real frames and their synthetically augmented versions from the original viewpoint. Note that we render the synthetic body on a static background for computational efficiency, but augment it during training with random real videos by using the segmentation mask. exercises. We believe that these results convincingly demonstrate the generalization capability of our efficient synthetic data generation method to real body motion videos. In Table  8 , we compare our results to the state-ofthe-art methods on standard NTU splits.
One-shot training We test the limits of our approach on unconstrained videos of the Kinetics-15 dataset. These videos are chal- lenging for several reasons. First, the 3D human motion estimation fails often due to complex conditions such as motion blur, low-resolution, occlusion, crowded scenes, and fast motion. Second, there exist cues about the action context that are difficult to simulate, such as object interactions, bias towards certain clothing or environments for certain actions. Assuming that body motions alone, even when noisy, provide discriminative information for actions, we augment the 15 training videos of one-shot Kinetics-15 subset synthetically using HMMR (see Figure  8 ) by rendering at 5 viewpoints (0 • , 30 • , 45 • , 315 • , 330 • ). We use a pre-trained feature extractor model and only train a linear layer from the features to the 15 classes.
We observe over-fitting with higher-capacity models due to limited one-shot training data. We experiment with two pre-trained models, obtained from  [9] : RGB and flow. The models follow the 3D ResNeXt-101 architecture from  [18]  and are pre-trained on Mini-Kinetics-200 categories with 16 × 112 × 112 resolution with consecutive frame sampling. In Table  9  (top), we first provide simple baselines: nearest neighbor with pre-trained features is slightly above random chance (8. 6% vs 6. 7% for RGB). Table  9  (bottom) shows training linear layers. Using only synthetic data obtains poor performance (9. 4%). Training only with real data on the other hand obtains 26. 2%, which is our baseline performance. We obtain ∼6% improvement by adding synthetic data. We also experiment with static background images from the LSUN dataset  [90]  and note the importance of realistic noisy backgrounds for generalization to in-the-wild videos.
Conclusions We presented an effective methodology for automatically augmenting action recognition datasets with synthetic videos. We explored the importance of different variations in the synthetic data, such as viewpoints and motions. Our analysis emphasizes the question on how to diversify motions within an action category. We obtain significant improvements for action recognition from unseen viewpoints and one-shot training. However, our approach is limited by the performance of the 3D pose estimation, which can fail in cluttered scenes. Possible future directions include action-conditioned generative models for motion sequences and simulation of contextual cues for action recognition.
A Additional details SURREACT rendering. We build on the implementation of  [83]  and use the Blender software. We add support for multi-person images, for using estimated motion inputs, for systematic viewpoint rendering, and different sources for background images. We use the cloth textures released by  [83] , i. e. , 361/90 female, 382/96 male textures for training/test splits, respectively. The resolution of the video frames is similarly 320x240 pixels. For background images, we used 21567/8790 train/test images extracted from NTU videos, and 23034/23038 train/test images extracted from UESTC videos, by sampling a region outside of the person bounding boxes. The rendering code takes approximately 6 seconds per frame, for saving RGB, body-part segmentation and optical flow data. We parallelize the rendering over hundreds of CPUs to accelerate the data generation.
We validate this empirically in Table  A. 1. We render multiple versions of the synthetic dataset with 10 motion sequences per action, each rendered from 8 viewpoints. We train only with this synthetic data and evaluate on the real NTU CVS protocol. Including multiple people improves performance (first and second rows), mainly because 11 out of 60 action categories in NTU are two-person interactions. Figure A. 6 also shows the confusion matrix of training only with single-person, resulting in the confusion of the interaction categories. Dropping the z component from the translation further improves (second and third rows). We also experiment with no translation if there is a single person, and xy translation only for the multi-person case (fourth row), which has the best generalization performance.
B Additional analyses We analyze further the synthetic-only training (Section B. 1) and synthetic+real training (Section B. 2). We define a syn- Effect of backgrounds. As explained in Section 3. 2 of the main paper, we use 2D background images from the target action recognition domain in our synthetic dataset. We perform an experiment whether this helps on the NTU CVS setup. The NTU dataset is recorded in a lab environment, therefore has specific background statistics. We train models by replacing the background pixels of our synthetic videos randomly by LSUN  [83, 90]  images or the original NTU images outside the person bounding boxes. Table  A. 6 summarizes the results. Using random NTU backgrounds outperform using random LSUN backgrounds. However, we note that the process of using the segmentation mask creates some unrealistic artifacts around the person, which might contribute to the performance degradation.