Is end-to-end learning enough for fitness activity recognition?
Authors: Antoine Mercier
Guillaume Berger
Sunny Panchal
Florian Dietrichkeit
Cornelius Böhm
Ingo Bax
Roland Memisevic
Venue: Submitted to ICLR 2023 (2023)
Keywords: end-to-end learning
action recognition
3D convolution
video
fitness

Abstract: End-to-end learning has taken hold of many computer vision tasks, in particular, related to still images, with task-specific optimization yielding very strong performance. Nevertheless, human-centric action recognition is still largely dominated by hand-crafted pipelines, and only individual components are replaced by neural networks that typically operate on individual frames. As a testbed to study the relevance of such pipelines, we present a new fully annotated video dataset of fitness activities. Any recognition capabilities in this domain are almost exclusively a function of human poses and their temporal dynamics, so pose-based solutions should perform well. We show that, with this labelled data, end-to-end learning on raw pixels can compete with state-of-the-art action recognition pipelines based on pose estimation. We also show that end-to-end learning can support temporally fine-grained tasks such as real-time repetition counting.

Under review as a conference paper at ICLR 2023
Is END-TO-END LEARNING ENOUGH FOR FITNESS ACTIVITY RECOGNITION?
Anonymous authors Paper under double-blind review
ABSTRACT
End-to-end learning has taken hold of many computer vision tasks, in particular, related to still images, with task-specific optimization yielding very strong perfor- mance. Nevertheless, human-centric action recognition is still largely dominated by hand-crafted pipelines, and only individual components are replaced by neural networks that typically operate on individual frames. As a testbed study the relevance of such pipelines, we present a new fully annotated video dataset of fit- ness activities. Any recognition capabilities in this domain are almost exclusively a function of human poses and their temporal dynamics, should perform well. We show that, with this labelled data, end-to-end learning pixels can compete with state-of- arr action recognition pipelines based on pose estimation. We also show that end-to-end learning can support temporally fine- fine-grained tasks such real-time repetition counting.
pose-based solutions
INTRODUCTION
Action recognition in videos has slowly been transitioning to real-world applications following ex- tensive advancements in feature representation and deep learning- based architectures. In many ap- plications, models need to extract detailed information of the underlying spatio-temporal dynamics. Towards this, end-to-end learning has recently had a lot of success on generic action recognition datasets comprised of varied everyday activities (Carreira & Zisserman [017]] Goyal et al. 2017]
Materzynska et al. 2019). However, pose-based pipelines seem to remain the preferred solution when the task is strongly related to analyzing body motions (Bazarevsky et al.] 2020 [Shahooudy
et al. 2016 Liu et al. 2020albb Yan al., (2088), such in the rapidly growing application domain of virtual fitness, where an Al system can be used to deliver real-time form feedback and count exercise repetitions.
In this paper, we present a new fitness action recognition dataset with granular intra-exercic labels and compare few-shot learning abilities of pose estimation- pipelines with end-to-end learning from raw pixels. We also compare the influence of using different pre-training datasets on the chosen models and additionally train them for repetition counting.
Common approaches to generic video understanding based on end-to-end learning include com- binations of 2D-CNNs for spatial feature extraction followed by an LSTM module for learning temporal dynamics (Donahue et al. 22017 Ng et al. 2015), directly learning spatio-temporal dy- namics with 3D-CNN (Ji et (013), or combining 3D-CNN with an LSTM (Molchanov al. 2016). The temporal understanding can be further improved in a two-stream approach with a second CNN-based stream trained on optical flow (Carreira & Zisserman 2017: Feichtenhofer al. 2016
Simonyan & Zisserman, 2014). The large parameter space of CN-NNS can be prohibitive and ef- forts to reduce this include dual-pathway approaches to low/high frame-rate (Feichtenhofer et al. 2019) and resolution (Fan et al., 2019), temporally shifting frames in 2D-CNN (Lin et al. (219), and non-uni con-uniformly aggregating features temporally (Li et al. 2020). Using a multi-task approach, an end-to-end model jointly trained for pose estimation and subsequent action classification was shown to improve performance of individual components (Li et al. 2017) but pose information is still needed for training.
base-baseddae solutions for action recognition have two main stages: pose extraction and action classi- fication. While bottom-up pose estimation approaches extract skeletons in one step (Cao al. 2016

---

Under review as a conference paper at ICLR 2023
Datasets
Exercise Videos Dainset
NTU RGBD
FineClym
Jester
Smih-smit
Charades
Kineties
MomentsIT
Focus body motions
Fine-grained labels
x
Controlled environment
"In the wild" Large-seale
Table 1: Side-by-side comparison of the Exercise Videos Dataset (ours) versus common video datasets including NTU RGBD+D (Liu et al., 2020a), FineGym (Shao al. 2020), Jester Materzynska et al. 2019). sooething-something Goyal et al. (2017), Charades (Sigurdsson et al. 2016]. Kinetics (Kay al., 2017) and Moments (Monfort al., 2019) based on five criteria: a) focus on body motion, b) fine-grained label taxonomy (e.g. presence of intra-activity variations), c) controlled environment (e.g. fixed camera angle in home environment), d) "in the wild" (as opposed to e.g. recorded in a lab), and e) dataset size sufficient for stand-alone pre-training.
Cheng et al. 2019: Newell & Deng 2016; Geng et al., 2021), top-down methods split pose estima-
tion into first localization and then pose extraction (Bazarevsky et al.) 2020 Newell et al. 2016 Sun al. 2019: Xiao et al., 2018). The classification stage then optimized independently. with no end-to-end finetuning of the whole pipeline. Posc-based action classifiers typically use either hand-crafted features (Ofli et al.] [014 [Wan ct al. Vemulapali et al. (2014) or, increasingly, deep Searning-bssed modules. Recent approaches have employed CNNs (Ke al., 2017: Kim & Reiter 2017 Li et al. 2017). LSTMs (Liu et al.
2016 Zhu et al.] [016 Shahroudy et al 2016]
Zhang al. 2017), Graph CNNs (Yan al. 2018 Thakkar & Narayanan
2019 Si et al. (2019). or
3D-CNNs on top of pose heatmaps (Duan et al. 2021).
In addition to an appropriate model architecture, a dataset with a fine-grained action taxonomy is crucial to learning robust action representations. Existing RGBBbaaed video datasets such as Kinet- ics Kay et al. (2017). Moments in Time Monfort et al. (2019) and Sports- 1M Karpathy et (2014) are based on a high-level taxonomy and further, possess correlated scene-action pairings resulting in pronounced representation bias Choi al. (2019): [i et al. (018). These concerns can be mitigated through crowd-sourced collections of predehned labels where the same action can be collected from multiple workers such as in the Something -omnething-Smethingg Goyal et al. (0177). and Charades Siiuuds- son et al. (2016) datasets. However, the "everyday general human actions" within these datasets are loosely specified and left to the worker's interpretation resulting in a high inter-worker action variance. On the other hand, FineGym [hhhhooudy et al.] (2016) focuses on specific fine- ane-grained body motions but includes variability in camera position resulting in lower overall action salience. In contrast, gesture recognition datasets such as Jester Materzynska et al. (2019) control camera and worker positioning and additionally, constrain human motion to appropriately specified hand ges- tures. A similarly constrained dataset for exact human body movement, that also controls camera motion, does not exist and we believe home fitness is the perfect domain in which to create one as workers can be instructed move very specific ways perform exercises.
Pose-specific datasets contain an additional layer of annotated skeletal joints obtained either through annotation of scraped video datasets (either manually or using a pose estimation model Liu et al. (020b)) or a sensor-derived approach in constrained lab settings (Sharoouyy et al. (2016); Liu et al. 2020a).
We present new crowd-sourced benchmark dataset to fill a gap in the dataset landscape (see Table [] videos of fitness exercises in a home setting are recorded in the wild providing challenging scene variety while also following fine-grained label taxonomy. We compare end-to-end action classification models with state-of-the-art pose estimation-based action classifiers and show that the end-to-end approaches models are pre-trained on a large and granular labelled video corpus. We also show that the pose estimation models themselves can greatly benefit from pre-training on the large labelled dataset.
outperform the posc iidimattion-bseed alternatives, if the end-to-end
THE Exercise Videos Datase A NEW BENCHMARK DATASET
Fitness activities are defined by a well-constrained set of body movements outside of which an indi- vidual risks injury or ineffectiveness. There an opportunity for AI systems to detect mistakes and
The name is temporary placeholder due double blind submission.
2

---

Under review as a conference paper at ICLR 2023
Figure 1:
Samples from the Exercise Videos Dataset for all four exercises. From top to bottom:
Dead bug, inchworm, alternating lateral lunges, spiderman pushups. Best viewed on screen.
Train
Validation
Test
Overall
Number of videos
4000
711
800
5511
Number of unique workers
129
20
165
314
Table 2: Dataset statistics: Number of videos and unique crowd workers in each split
provide real-time form-correcting feedback. To this end, present the Exercise Videos Dataset comprised of granular video-level activity classes capturing subtle variations, including common mistakes. The dataset spans four fitness exercises recorded home environment by crowd work- ers:
The user lies on the back with arms and legs raised and moves them back and forth asynchronously.
Dead bug:
Inchworm:
From standing position, the user touches the floor with both hands, walks them forwards, and then back again.
Alternating lateral lunges: The user performs a lunge step in sideways direction, alternating in both directions.
Spiderman pushups: A pushup variation where one leg is moving to touch knee and elbow.
Example frames from the dataset for each exercise can be recorded with deliberate variations such as increased pace or incorrect execution of different aspects of each exercise, some which visible from a static frame (foot touching the floor), and others which are only apparent across multiple consecutive frames (being too fast or too slow). In total, fine-grained taxonomy of 40 video-level classes is available trigger direct feedback.
in figure [ Each exercise was
Each of the 40 classes contains between 130 and 140 videos, with each video lasting between 5 and 8 seconds. The dataset is split into train, validation and test sets with no worker overlap between them. All videos are provided in MP4 format at a frame rate of 30 fps. The dataset contains 5511 videos in total across all splits (see Table 2 for details on the data split). For few-shot experiments, we prepared different versions of the train splits, containing fewer examples per class. We release splits that contain 5, 10, 20, and 100 samples per class.
In addition to delivering form feedback in real-time, another challenging task for fitness Al applica- tions repetition counting. relies on precisely parsing the temporal extent of an action segment within activity and, such, benefits greatly from the availability of temporal annotations. To this end, addition to providing video-level labels, we tagged a subset of each exercise within the Exercise Videos Dataset with frame-level classes, thus making it possible to benchmark models repetition counting. More details will be provided in Section 3.6
The data has been collected in the wild by individual crowd workers who performed the actions following instructions from an example video. To match the desired viewing angle of phone placed the floor (fitness app scenario), the workers recorded themselves using camera
3

---

Under review as a conference paper at ICLR 2023
Figure 2:
The videos in the dataset provide a wide range of lighting and scene settings. From left to right: cluttered background, textured background; high contrast, low contrast. Best viewed screen.
low position. All recorded videos were reviewed confirm the execution was performed correctly. Because of the distributed nature of the data collection, the recorded samples show large variety of scene settings, backgrounds and illumination (see figure 2 Each worker recorded videos for multiple action classes, so that the performed action cannot be learned by the visible video setting, but only by learning feature representations the actual body motion.
The Exercise Videos Dalaset has been collected for the purpose of discerning fine variations of exercise execution performed by worker. order to create the label taxonomy and recording instructions. the domain knowledge of several fitness experts had been consulted to collect list of common mistakes and frequent variations of the individual exercises. Some examples subtle variations are:
Dead bug: A foot touches the floor; arms are not moving; the wrong leg is moving; execu- tion is too fast
Inchworm: Feet are too narrow or too wide; hands are too far from the body in the initial position; hands stepping too far forward with each step
Alternating lateral lunges: Bending the wrong leg; low range of motion; execution is too
fast
Spiderman pushups: Execution is too fast or too slow; leg movement is not in sync with pushup (three different error variations are labeled); pushup too shallow
The full label taxonomy can be found in the supplementary materials. We plan to release the dataset under con-commercial license, which permits non non-profit research only.
EXPERIMENTS
All models were trained on subsets of the Exercise Videos Dataset training split, with 5, 10, 20, 50, and 100 samples per class, evaluate few-shot behavior. Different initialization approaches were tested for each model. including training from scratch, starting from a pre-trained model and fine- tuning the final classification layer, all layers, a subset of the layers. The approaches are described in more detail in section 3.4
3.1 ARCHITECTURES
Three end-to-end and two pose estimation-bases architectures are compared in our experiments. End-to-end architectures include I3D (Carreira & Zisserman 2017). SI-EN (ours) and S1-BlazePose
(ours). For the pose-based pipelines, we use BlazePose (Bazarevsky et al. 2020) to localize and extract human poses followed by one of two satte-of-the-art graph-based classifiers: ST-GCN (Yan selected BlazePose for the pose extraction part be- cause it is optimized for real-time fitness applications and comparable to the end-to-end architectures in terms of FLOPs and model size (see section 3.1.3 for more details).
jet al., 2018) and MS-G3D (Liu et al. (202bb).
3.1.1 END-TO-END
I3D. an end-to-end baseline model for video action recognition, we used the 3D-CNN archi- tecture, I3D-RGB, proposed in Carreira & Zisserman (007)).
Strided-Inflated EfficientNet (SI-EN). We present SI-EN, which uses EfficientNet-Lite (Tan &
Le 2019), 2D-CNN, as backbone, with few modifications some of the inverted residual
4

---

Under review as a conference paper at ICLR 2023
ST-GCN
MS-G3D
80
80
Test accuracy
60
Test set accuracy
60
50
30
30
20
20
10
10
50
100
5
20
100
Number of training samples per class
Number of training samples class
(a)
from scratch
pretrained on Kinetics
pretrained on BigFitness
SI 9-EN-Big Bi-iiiFitness-11
Figure Effect of pre-training ST-GCN and MS-G3D on Kineties and BigFitness.
blocks. Specifically, we inflate of the blocks in the temporal dimension (blocks 7, 11, 17, 20, 23 and 25), using temporal kernel of 3, effectively turning them into 3D convolutional modules taking inspiration from (Carreira Zisserman 2017). More precisely, it only the first point-wise convolution in the inverted residual block that is inflated. Two of the inflated convolutions (blocks 7 and 14) are implemented with a stride of 2, enabling a lower footprint output of 4 fps from 16 fps input stream.
SI-BlazePose. As method back-propagate through pose feature bottleneck during end- to-end classification task, we propose the following architecture which we call ST-BlazePose. It is based on the BlazePose model (Bazarevsky al. 2020) using inflation extend it in the temporal dimension. We inflate the last 8 point-wise convolutions with temporal kernel of size 3, adding temporal stride of the 2nd and 4th one. We freeze all layers before the first inflated layer. We use the full image as input, and resize it to 256 256 preserving the aspect ratio. We did not crop around the person as a first step, in contrast to what is done within MediaPipe Since the Exercise Videos Dataset is classification dataset, replace BlazePose's body part regression head with softmax layer.
3.1.2 POSE-BASED CLASSIFIERS
ST-GCN. Spatial-temporal graph convolution networks (ST-GCN) use graph convolutions across spatial joint connections and temporal connections from frame to frame (Yan et al. 2018). Following the original authors' approach, included their suggested edge importance weighting method with a spatial partitioning strategy. As our results did not benefit from dropout regularization, we disabled it.
MS-G3D, Multi-scale graph convolutional networks (MS-G3D) adjust the node weighting in the graph for improved multi-scale aggregation and introduce skip connections to the graph for better modeling of spatio-temporal dependencies across longer distances.
As these models are able to work on generic graph layouts, we added support for the BlazePose layout by providing the adjacency matrix the 33 graph nodes.
https::/google.gittte

---

Under review as a conference paper at ICLR 2023
80
70
60
Test set accuracy
50
40
30
20
10
10 Number training samples per class
20
50
100
Si-EN--mageNet-all
Sl-EN--BigFinness-11 10
IBD-Kinetics-all
SI-BlazePose-9
ST-GCN-Scratch
MS-G3D- G3D-Scratch
Figure 4: Exercise Videos Datasel top-1 accuracy of selected existing architectures, pretrained on various datasets. We report results using 5, 10, 20, 50, and 100 training samples per class. For each model, we use the following convention: (rppifecttre)--pptttaininng dataset)-(optionall number of finetuned layers).
3.1.3 A NOTE ON COMPUTATIONAL. EFFICIENCY
pipeline based on pose-estimation typically consists of 3 components: a detection network pro- ducing rough person positions, a pose estimation network producing skeletons for each person (BlazePose in our case), and classifier mapping sequence of skeletons to an activity label (STGCN or MSG3D in our case). The first two components are image-based while the action classifier is video-based in the sense that it needs a sequence of skeletons. While the detection network can run least, the person not moving their position much), the framerate which the pose estimation component needs to run is determined by the temporal granularity required by the action classifier to obtain high accuracy. An end-to-end neural network on the other hand pro- vides variety of flexible ways to reduce computational footprint, e.g. by using temporally strided convolutions which reduces the framerate of subsequent layers and outputs. SI-EN specifically ex- ploits this by introducing two 3D convolutions with a temporal stride of 2 early in the architecture. As a result, most of the SI-EN layers only need to run at 4fps rather than the 16fps input framer- ate, greatly reducing the computational footprint of our end-to-end solution. At an input framerate of 16, SI-EN only requires 4.0 GMACs/s, whereas running BlazePose alone (i.e. without counting localization and action classification) already amounts 6.7 GMACs/s.
fairly infrequently
3.2 DATASETS USED FOR PRE-TRAINING
In addition to the dataset we are releasing along with this paper, we use larger internal video dataset, which refer to BigFitness, for pre-training in some experiments. This dataset consists of around 300, 000 videos of fitness exercises with a fine-grained label taxonomy across 536 classes that disjoint from the data in the Exercise Videos Dataset.
In addition to this internal dataset, we also made use of Kinetics (Kay et al. 2017) and ImageNet (Russakovsky al., 2015) for pre-training, as will be described in the results section. For more information about the relationship between the pre-training datasets used in our experiments and the Exercise Videos Dataset, please refer to Table 6 in the Appendix.

---

Under review as a conference paper at ICLR 2023
Number of samples per class:
5
10
20
50
100
End-to-end
SI-EN-ImageNe
8.9
15.5
23.9
33.5
38.1
SI-BlazePose
25.3
31.1
39.9
47.1
52.9
JSD-Kinetics-1
12.2
17.1
22.5
25.9
28.4
35D-Kinetics-4
18.9
28.6
39.8
51.5
56.1
BD--iinetics-all
19.4
28.7
43.3
53.9
60.9
SI-EN-BigFitnn
38.1
44.4
49.5
56.0
58.9
SI-EN-Bipe BEN.BiiFitness-I 10
45.2
50.7
56.0
63.5
66.8
J.EN-BigFitness
36.2
43.6
51.5
60.8
Pose-based pipeline
ST-GCN-Scratee
26.7
39.4
45.4
53.7
57.4
MS-G3D-Seratch
25.5
32.3
44.7
57.6
62.1
ST-GCN- Kineticss
30.8
39.1
49.9
58.0
60.7
ASS-33D-inettin
38.9
47.2
53.5
62.2
65.6
ST-GCN-Big
38.7
49.1
53.6
59.7
63.0
MS-G3D-Bigg MSS-GeD-BigFitness
41.9
51.6
56.3
62.2
65.5
Table Results across all experiments. We report the test set accuracy percentage on the Exercise Videos Dataset.
3.3
IMPLEMENTATION DETAILS
3.3.1 END-TO-END
End-to-end models were trained on raw pixels from the Exercise Videos Dataset videos. The native resolution was down-scaled to resolution of 256 x 256 pixels. To keep the original aspect ratio, frames padded with black pixels be in square format before downscaling. Videos were subsampled to 16 fps which showed improved performance over the native 30 fps in preliminary experiments. For training, we took random crops of 63 frames from each video, which corresponds to roughly 4 second long video clips. 63 was chosen because of memory constraints. For evaluation, all frames of a video were passed to the model. As additional augmentation, we applied random color jittering to the 3 input channels. RGB values were scaled to the range from 0 to 1.
3.3.2 POSE-BASED
To pre-train pose-basee based models in a way that is comparable the end-to-end models, we extracted pose features from BigFitness using BlazePose (Bazarevsky et al.) (020) as provided by the Medi- aPipe library 3] The same method was used to extract pose features to train on the Exercise Videos Dataset. In our experiments, we used all 33 joints and 3 input channels per joint: x position, posi- tion and confidence score. The resulting pose sequences were created at 16 fps, because preliminary experiments showed better results than using the raw 30 fps (just like in the end-to-end experiments). For training, we took random crops of 90 consecutive poses. For evalutation, we passed in the full pose sequence of each sample. Following (Yan et al. 2018), used simulated camera movement on top of keypoint coordinates as a data augmentation technique during training.
The Kinetics-Skeletae dataset Yan al. 2018), that uses the OpenPose (Cao et al., 2019) layout, which has fewer key points than the BlazePose layout (18 instead of 33). In our experiments, we mapped BlazePose keypoints the OpenPose format with the neck position being defined as the center between the two shoulder joints.
for pre-training some of the models,
3.4 RESULTS
The performance the Exercise Videos Dataset across architectures is reported in Figure For each model, we have tried multiple fine-tuning strategies (e.g. freezing all layers, fine-tuning a subset of the layers, fine-tuning the whole network). Figure 4 only reports the approach that worked the best for each model. Results obtained using the other strategies can be found Table 3 Regarding pose-hased baselines, the best of our knowledge, there
versions of MS-G31 and ST-GCN
ptttps:://medappiie
Note that use the GHUM Full version of BlazePose in all our exper-
iments.

---

Under review as a conference paper at ICLR 2023
Temporal annotations schemes
pushup
dead bug
lateral lunges
inchworm
SI-EN
(1) within-repe cebhin-rpptition vs
25.9
39.3
33.3
109.0
(2) within vs middle-of vs end-of-repetitoon
17.1
22.3
13.4
49.2
(3) first half second half
4.6
7.2
2.2
21.5
MSG3D
(1) within-repetition vs. end-of-repetition
22.2
40.1
38.4
102.0
(2) within vs middle-of vs end-of-repetitien
10.6
27.5
9.0
51.0
(3) first half second half
4.9
8.5
4.2
17.2
STGCN
(I) within-repetition vs. end-of-repetition
37.3
81.8
66.3
144.0
(2) within vs middle-of vs end-of and-of-reettttonn
11.9
12.8
7.0
46.5
(3) first half second half
6.0
13.7
3.6
22.0
Table 4: Repetition counting results across all experiments (mean absolute percentage error).
pre-trained on the 33 joints returned by BlazePose and we therefore train the two graph CNNs from scratch in this experiment. We investigate the effect of pre-training MSG-3D and ST-GCN in the next section. Interesting findings from Figure be summarized as follows:
Best performance is obtained by an end-to-end network. Fiitesss tops all other ap- proaches with a significant margin, including pose-based solutions that use graph CNN initialized from random weights. The gap with pose-based pipelines is higher when training data is scarce (45.2% vs 26.7% for ST-GCN-Scrater in the 5-shots case) but shrinks more training samples are available (66.8% vs 62. for MS-G3D-Scratch when the full trainset used).
Pre-training large video dataset is key. Unsurprisingly, the type of data used to pre- pre-train each baseline plays an important role in downstream performance. Best results are obtained by the model that pre- tre-trained on Bignessss, which is by far the most granular pre-training dataset considered in this experiment. The exact same SI-EN architecture pre-trained on ImageNer performs poorly. The Kinetics baseline, 13D, is roughly par with pose-based based pipelines. On the other hand, the inflated pose 2D CNN, SI-BlazePose-9 obtains decent results when few samples are available but gets significantly outperformed as more samples available.
MS-G3D seems more prone to overfilling than ST-GCN. While MS-G3D outperforms ST-GCN when more than 50 training samples are available, ST-GCN gets better results in the 5, 10 and 20-shot cases.
3.5 CLOSING THE GAP BETWEEN POSE-BASED AND END-TO-END APPROACHES
In this section, we investigate the effect of pre-training the graph CNN component of pose-based pipelines. Pre-training is performed with two datasets: Kinetics and BigFimess. Results can be found in Figure B
Figure 3 shows that, even for pose-based pipeline, pre-training large vidco dataset boost classification accuracy. While an accurate frame-level pose representation alone obtains decent results, the overall solution greatly benefits from pre-training videos. This suggests that training data that provides some understanding of the temporal aspects human body motions is highly beneficial, even for pose-based models. While pre-training on Kinetics produces good downstream performance, pre-training on a more granular dataset such as BigFitness works better overall. When is pre-trained on BigFimess, the MS-G3D-baser pipeline is on par with the end-to-end baseline, and the ad advantage that ST-GCN has over MS-G3D in the lower data regimes vanishes. Additional metries ge.g. confusion matrices, f-measures) can be found in the supplementary material.
3.6 LEARNING TO COUNT
To explore temporally fine-grained recognition task, also experiment with end-to-end repetition counting ("how many times has a given exercise been performed?"). This a common task, in particular, in many fitness applications.
Repetition counting is an inherently temporal prediction task. To train the networks on this task, we temporally annotated subset of videos with frame-by-frame labels describing which phase of the
8

---

Under review as a conference paper at ICLR 2023
exercise the subject is at any moment in time. We use the same frame-rates as before (16 fps input, fps output) and annotated 100 videos for each of the exercises in the training set. We use the same train/test split above for evaluation. We experiment with various temporal annotation schemes that can be turned into counts after training: (1) marking frames as within-repetitionn vs. end-of- repetition, (2) marking frames within-repetition
end-of-repetition vs. micdlee-of-reeeiiion,, (3) using a different encoding of the 3-way annotations in (2), by marking frames as first-half of a repetition (between end-of-repetition and middle-of-repetition) vs. second-half of a repetition (between oaddle-of-repetitinn and end-of- rod-tt-repetition).
We train the models by treating these annotations as a simple temporal classification task. For training. we concatenate the videos within a mini-batch along the temporal axis rather than stacking videos on the batch-axis. Since annotation schemes (1) and (2) are highly imbalanced, we weight the classification cost by 0.2 for the under-representted class "within-repetition" during training. For SI-EN, we only train the 10 final layers (as for the best model above).
We turn temporal classifications into counts at inference time by incrementing the count when the end a repetition is detected, For annotation schemes (1) and (2), we only increment the count if an end-of-repetition event is followed by at least one middle-of-repetttion event to avoid over- counting. Table 4 shows the performance of the models in terms of mean absolute percentage error (MAE) (Rumia et al., 2018 Levy & Wolf 2015). It shows that accurate counting performance can be obtained from the relatively small number of annotated videos. While performance is comparable across models, interestingly, par with or better than the other approaches in most cases, In fact, it shows the best performance in all exercises except for "Inchworm" which unlike the other exercises, has much smaller number of repetitions per video and yields overall lower accuracy. Note that only SI-EN can make predic- tions on-line. Overall, while a deeper analysis and comparison with other counting approaches is beyond the scope of this paper. we find that it is possible to obtain very accurate repetition counts entirely end-to-end. We also find that accuracy depends strongly on how temporal annotations are represented during training.
in this setup, the end-to-end approach SI-EN performs roughly on
4 CONCLUSION
In conclusion, our experiments show that end-to-end training on large-scale labeled video datasets without any form of frame-by-frame intermediate representation can compete with pose-based ap- proaches, even in the context of fitness activity recognition where one could assume that an accurate pose representation is all you need. More importantly, regardless of the selected approach, pre- training on large and granular video dataset is a key ingredient to achieving good downstream performance. In fact, our experiments show that good performance in action recognition tasks is mostly function of dataset size and label granularity and less of the choice of model.
Limitations and broader impact. The introduced dataset subserves research on end-to-end rea- soning about human activities using an RGB camera. It can be used to study and benchmark model architectures and to rethink workflows in the development of end-to-end neural networks. How- ever, the dataset in its current size and form may contain biases. Training this dataset alone may, for example, lead to models whose behaviors could depend on a subject's age, gender, eth- nie background, etc. As such, the dataset as defined is suitable only for performing the research needs described above. In addition, model behavior will be a function of camera angle, lighting, and possibly other random aspects of the scene, camera, camera- angle and the subject interacting with the model. As for positive impact, research towards enabling quantitative assessment of health and fitness-related activities with just camera can democratize access to and can greatly improve individuals" understanding of such activities and help unlock their benefits.
REFERENCES
Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. BlazePose; On-device Real-time Body Pose tracking. arXiv, 2020.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi person 2d pose estimation using part affinity fields, 2016. httttp/a///ax.iooog//////////// 08050

---

Under review as a conference paper at ICLR 2023
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih En Wei, and Yaser Sheikh. OpenPose: Realtime Multi- Person 2D Pose Estimation Using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. ISSN 19393539. doi: 10.109/TPAM.22011222995...
João Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinet- ics Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Bowen Cheng, Bin Xiao, Jingdong Wang. Honghui Shi, Thomas S. Huang, and Lei Zhang. Bottom- up higher-resolution networks for multi-person pose estimation. CoRR, abs/1908. 10357, 2019.
URL http: org/ abs 1908 10357
Jinwoo Choi, Chen Gao, Joseph C. Messou, and Jia-Bin Huang. Why Can't Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition. 12 2019. URL http::ttt///...orggggg/
abs/19122 05534
Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadar- rama, Kate Saenko, and Trevor Darrell. Long-Tere Term Recurrent Convolutional Networks for Visual Recognition and Description. JEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4), 2017. ISSN 01628828. doi: 10.11009PAMM2.21622599177
Haodong Duan, Yue Zhao, Kai Chen, Dian Shao, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. CoRR, abs/2104. 13586, 2021.
13586
Quanfu Fan, Chun Fu Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More less: Learn- ing efficient video representations by big-little network and depthwise temporal aggregation. In Advances in Neural Information Processing Systems, volume 32, 2019.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional Two-Stream Net- work Fusion for Video Action Recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-December 2016. doi: N0.11099/VPPR.2066.21.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE International Conference on Computer Vision, volume 2019-October, 2019. doi: 10.1 10.100//CC..2211.00066
Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. Bottom-up human pose estimation via disentangled keypoint regression. CoRR, abs/2 10s.21044,2300, 2021. URL https:
//arxiv.orq/abttttt abs/ 2204.11100.000 02300
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5842-5850, 2017.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3D Convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 2013. ISSN 01628828. doi: 10.1 10.1109//AM..202.5555
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The Kinetics human action video dataset, 2017. ISSN 23318422.
Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, and Farid Boussaid. A new representation of skeleton sequences for 3D action recognition. In Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, volume 2017-January, 2017. doi: 10. 10.1109/CPP..217..0
10

---

Under review as a conference paper at ICLR 2023
Tae Soo Kim and Austin Reiter. Interpretable 3D Human Action Analysis with Temporal Con- volutional Networks. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, volume 2017-July, 2017. doi: 10.1 10.1009/VVR..2211.227.
Ofir Levy and Lior Wolf. Live repetition counting. In Proceedings of the IEEE International Con- ference on Computer Vision (ICCV), December 2015.
Chao Li, Qiaoyong Zhong. Di Xie, and Shiliang Pu. Skeleton-based action recognition with con- volutional neural networks. In 2017 IEEE International Conference on Multimedia and Expo doi: 10.109/CCMEE2201.8822222
Workshops, ICMEW 2017, 2017.
Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Hao Chen, and Joseph Tighe. NUTA: Non-uniform temporal aggregation for action recognition, 2020. ISSN 2331 23318422.
Yingwei Li, Yi Li, and Nuno Vasconcelos. RESOUND: Towards action recognition without repre- sentation bias. Proceedings of the European Conference on Computer Vision (ECCV), volume 11210 LNCS, pp. 513-528, 2018. doi: 10. 3-000--0000000--111111
Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference on Computer Vision, volume 1019-October, 2019. doi: 10. 1..1109/CCCV2221..00777
Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal LSTM with trust gates for 3D human action recognition. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes Bioinformatics), volume 9907 LNCS, 2016.
doi:
10.10/79 3-11--331199--99 31--477777--9 }:50.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling Yu Duan, and Alex C. Kot. NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding, JEEE Trans- actions on Pattern Analysis and Machine Intelligence, 42(10), 2020a. ISSN 19393539. doi: 10.1109/TPAMI.20192222 2966733....
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang, Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp- 140-149, 2020b. URLhttp:://aaett wttp://wr.g.abbb...aaaa.
Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The jester dataset: A large-scale video dataset of human gestures. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D) Convolutional Neural Networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume Dece-eee,,,,r, 2016. doi: 10.11099CVPR22211.4466
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1 1-8, 2019. ISSN 0162-8828. doi: 10. 10.1109/PAAM..201999900
Alejandro Newell and Jia Deng. Associative embedding: End-to-end for joint detection and grouping. CoRR, abs/161 105424, 2016. URL TRLL:ttt::/aaxxivcrrr//bbb////// 1ttp:/11111.....111111111
Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks human pose estima- tion. CoRR, abs/1603.0699) 06937, 2016. UBl.iittp///amnii.orr///////11./ 06937
Joe Yue Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Pro- ceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 07-12-June-20155 2015. doi: 10. 10.1099/VPR.2055.29910

---

Under review as a conference paper at ICLR 2023
Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, and Ruzena Bajesy. Sequence of the most informative joints (SMJJy A new representation for human skeletal action recognition Jour- nal of Visual Communication and Image Representation, 25(1):24-38, 2014. ISSN 10473203.
doi: 10. v/v./// 1..00.00....
Tom F. H. Runia, Cees G. M. Snock, and Amoll W. M. Smeulders. Real-world repetition estimation by div, grad and curl. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3), 2015. ISSN 15731405. doi:
Amir Shahroudy, Jun Liu, Tian Tsong Ng, and Gang Wang. NTU RGB+D: large scale dataset for 3D human activity analysis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016 D0eembeee,err, 2016. doi: 10.1109/CVPR. 2016.115.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: hierarchical video dataset for fine- grained action understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan. An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1227-1236, 2019.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood homes: Crowdsoureing data collection for activity understanding. In European Conference Computer Vision, pp. 510-526. Springer, 2016.
Karen Simonyan and Andrew Zisserman. Two-Stream Convolutional Networks for Action Recog- nition in Videos. In Proceedings of the 27th International Conference on Neural Information Pro- cexving Systems- Systems-Volume 1, pp. 568-576, 2014. URL
5353 1353-tto--a sttremm-convoluute hrg.aaaaaaaattttttoooooooooooooe pdf
videos.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. CoRR, abs/1902.09212, 2019. URL
19C2. 09212
Mingxing Tan and Quoe Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In International Conference on Machine Learning, pp. 6105-6114, 14, 2019.
Kalpit Thakkar and P. J. Narayanan. Part-based graph convolutional network for action recognition. In British Machine Vision Conference 2018, BMVC 2018, 2019.
Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by repre- senting 3D skeletons as points in lie group. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014.
doi: 10. 10.100///2..2
Jiang Wang, Zicheng Liu, Ying Wu, and Junsong Yuan. Mining actionlet ensemble for action recog- nition with depth cameras. In Proceedings of the IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition, 2012. doi: 10.1100//VVRR.202224444
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. CoRR, abs/1 804.06208, 2018. URL
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-bard based action recognition. In 32nd AAAI Conference on Artificial Intelligence. AAAI 2018, 2018.
12

---

Under review as a conference paper at ICLR 2023
Songyang Zhang, Xiaoming Liu, and Jun Xiao. On Geometric Features for Skeleton-Ban Based Action Recognition using Multilayer LSTM Networks. In IEEE Winter Conference on Applications of Computer Vision (WACV), pp.
Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, and Xiaohui Xie. Co- Occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016, 2016.
13

---

Under review as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIAL
Fitness exercises
Video-level classes
Frame-level classes
knee over toe
left leg bent
low range of motion
right leg bent
no obvious mistakes
end-of-repetition
no stepping
not alternating
alternating lateral lunges
stepping foot pointing away too fast
torso bent forward
torso bent sideways
wrong knee bent
foot touching the floor
middle-of-repetiionn of-repetition end-of-repetition
moving opposite leg
moving same side
dead bug
not moving arms
opposite knee too bent too close to chest
too fast
arms narrow
plank pose
arms too wide
end-of-repetition
excessively short
feet too narrow
feet too wide
getting into position
inchworm
getting into position hands too far
good form
head up
hips too low
not far out enough
stepping too big too Fast
arms too narrow
low pushup position end-of-repetition
arms too wide
good form
no pushup
not alternating
spiderman pushups
not synced (down leg in up leg out)
not synced (down - leg up)
synced (down up leg)
shallow
too fast
too slow
Table 5: Label taxonomy of the Exercise Videos Dataset
14

---

Under review as a conference paper at ICLR 2023
Exercise Videos Dataset exercises
Closest classes
Closest Kinetics classes
pushups sloppy burpee upright position burpee no jump
push up crawling baby headbanging
spiderman pushups
bicycle crunches small torso rotation bicycle crunches medium torso rotation bicycle crunches head down
situp
dead bug
knitting
unboxing
skaters single jump (right to left) grabbing off-screen towel skaters slow
lunge
alternating lateral lunges
side kick squat
burpee (no pushup)- stepping feet forward burpee (no pushup) stepping feet back roll down
dribbling basketball deadlifting push up
inchworm
Table 6: Comparing dataset similarity: for each the Exercise Videos Dataset exercise (column 1), we compute a prototypical feature vector and show closest class centroids in feature space within BigFitness (column 2) and Kinetics (column 3). It shows that BigFitness has multiple labels that are conceptually similar, which to be expected as contains fitness actions with a disjoint, but also fine- ane-grained, label taxonomy. More general action recognition datasets like Kinetics have some fitness actions, such as push up and lunge, which resemble the exercises from the Exercise Videos Dataset. However, because of the more coarse label taxonomy, the next nearest neighbors can be very different (such as labels "head banging" or "unboxing").
15

---

Under review as a conference paper at ICLR 2023
ADDITIONAL RESULTS
BigFiinesss-se 10
abemattng
atrmating aleral knnee chilcnn matske
atemebrng Worli linges not alenaaiin lunges udtrel loraes
atemating
stomating idteral lunge taro fureard stemmiing useral Linges wrin kree bent
moverg uppcodic sitt
deacbugs
dcacbuga oppositc lonce too bont boo desc chest
hermy
inchaorm arees too
inclworm fect loo harse
inchwerm genting poution hands
inchoom-
nccworm. head inchwemm
not enoup richworm Soepprr
speder man puataps arres loo surce
raider man tuehups good
speder man pualups not symoed idown-leg mup log souder rean pushupe
syoted (cown leg
upttor sptter man pantacts dhe
ou adugrd weu Weuth
fledrree Sifuni plom fla feshing
sep S0N IN BIRMIO
da
be useil
safuny mann Dualuss
2001 stoggeo
2ahocrd pouds
Predicted label
Figure 5: Confusion matrix for the SI-EN-BigFitness- 10 model the FinestFimess test set, aver- aged over five training runs.
Exercise
SJ-EN-BigFitmess110
MS-G3D-BigFitnes
alternating lateral lunges
0.71
0.75
deadbugs
0.73
0.79
inchworm
0.55
0.48
spider man pushups
0.62
0.60
Table 7: Aggregated f-measures per exercise and model as an indicator of the intra-exercise perfor- mance, computed by averaging across all class-wise f-measures that belong to the same exercise. Each of the two examined models performs better on two out of the four exercises.
16

---

Under review as a conference paper at ICLR 2023
Bigt itness
linges pert ahemanro beech lungss magg motio stemabing literal lungas
20.0
lunges dregeng
atemaatin lateral aPemating intral lunges too nott
373
a stomating lateral layes tone sete soowass neminnnng uteall Lings
23.0
deactupg moving oppostte log deadb.pp not mexing amrs
deachuge
Scodaags inchwomm too narres
12.5
Inthoom excesivess
The
nncharmm geming poction
32.0
getting
inchacrm good larn
75
inchworm not out anough spider man rchwoie too Ibs aoader pustisg tco
5.0
spater
spider man DUSTLgT altemating pode
Tann Caston not ppcce from
25
ceRo raa con ligh lao tont
tha Buschnn
for us resiidl farnA dnitnn
wis Bivon dh Dova -Shoppp
sni Bannii (UR0er
any autaapps
ui mus eomund und Heees
ui be uneil gouut
au whiv unu ingatt
sarood oou woods
Presitted
Figure 6: Confusion matrix for the MS-G3D-BigFitnes model on the FinestFitness test set, aver- aged over five training runs.
Bighitness-
BigFitness
250
alterneting lnteral
200
00
00
0.0
alternating lateral
200.0
0.0
0.
0.0
label
dbadbugs
0.0
120.0
00
0.0
dcadbugs
0.0
119.0
06
0.4
150
incheorm
0.0
0.
259.0
0.2
100
inchworm
0.0
0.0
256.4
3.6
100
0.0
0.0
0.4
219.6
spider
02
02
3.4
216.2
mnvemating lunges
Predicted label
Predicted label
Figure 7: Confusion matrices from Figure and Figure 6 aggregated into exercise- wide scores. Results have been obtained by summing up the scores for all classes belonging to the same exercise.
17

---

Under review as a conference paper at ICLR 2023
atterating deadiugs moving arms
altemating lateral lunges wrong knee bent
witemating lateral linges strpp ng
spider pushups arms nerrow
spider man pushups synces (down-up-les
intwerrm tno fest
deadhags lout touching floor
spider man pushups- arms wide-
abernating
toot pointing deadbugs too fast
alteraaiigg lateral al langes too fast
spider pushuss no pushup
tnchworm
inchworm feet ende-
spider man pushups synced (down leg
deedbugs moving same Side
getting postion
alternating
incworm- hips too
inchworm getting into postoon hands too far
afternating lateral lunges tonn bert forward
attemating
spider man pushups too slow
deadbugs apposote knee too bent or luo close to chesl
atteratinng lunges absious
dearbrugs moving oppostte leg
spider
(down
spider man pustupp shallw
inchworm. head up
incworm-
incworm- excesaively shert
spider mann pushups not ahtrrating
atteratting lateral lunges knee over toe
good
form
incworm- not far out enough
S-EN-B 9-EN-B Fitnes-110
G3D inees
0.0
0.
04
06
08
10
Measure
Figure 8: F-measures per class for -igEnsBig 10 and GS-GGD-Big,innes,, obtained over runs on the FinestFineess test set and sorted according to SI-EN-BigFitnes 10 results.
spider man pushups- arms too wide
spider man pushups not synced (down up leg)
inchworm arms narrow
good form
atteratinng loteral inchworm getting into position hands too for after nating lateral lunges- low range motio spider man pushups
atternating
synced (down-leg spider pustups not synced (deen ietwoorm excensively shert incworrm getting into postion
incworm- feet too narrow
spider puschups amms too namrow sttemaaiig laterel lunges no stepp ng deediuugs moving arms stepping
deadbugs foot
Si-EN- S--N--igiiness- better
altemating lateral lunges wrong knee bent
MS G3D digl iitees better
deadbugs too fest
incworrm
spider
inchworm...
wide
spider man pushups too sinw spider man pushups no pushup spider man pushups good form deadinugs moving opposiee atermating laterall lunges. alvicus mastaki afternating lateral lunges- lono bert forward
afteraainng lateral lunges. stepping foot pointing away inchworm
up
altemating lateral lunges torso bent sideways deadlugs- opposite knee too bent or too close to chest
spider man puships too dredbugs-
attemating lateral lunges. knee over toe
000
005
0.0
0.15
0.00
0.5
0.30
0.35
0.40
Absolute difference of 1-Meesures
Figure 9: Absolute differences of f-measures for the two models from figure sorted by decreasing performance of SI-EN- BEN--igFitness-10 10.
18


Reviews:
[{"review_id": "pN2cmVKWEYQ", "reviewer": "Reviewer_66o9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work targets studying end-to-end learning for fitness activity recognition.  A new fully annotated video dataset of fitness activities is established to evaluate different action recognition methods.  And they show that end-to-end learning could perform similarly to SoTA action recognition pipelines based on pose estimation methods.", "summary_of_the_review": "This work seems to be an interesting study, but I did not recognize sufficient significant contributions.  As a result, I could suggest its acceptance.", "strength_and_weaknesses": "Strength:\n1. It is good to study end-to-end training for fitness action recognition, and it is also good that it shows competitive performance with previous SoTA methods.\n2. It could facilitate future research by establishing the fully-annotated action recognition video dataset.\n\nWeaknesses:\n1. The writing is not good, and the paper could be better organized.  As for the current manuscript, it is not easy for readers to get the research gaps, the significance of the achievements, and the main contributions of this work. \n2. The proposed dataset seems to be very small, which makes the conclusion less convincing. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TPxfM2TtOz", "reviewer": "Reviewer_a8ca", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper collects a new dataset for action recognition, which focuses on fitness activities. Based on this newly proposed dataset, the authors explore the influence of end-to-end training, pretrained dataset, and few-shot learning. This paper also compares these settings on a repetition counting task.", "summary_of_the_review": "Despite I deeply appreciate the investigation in the new dataset, I find it difficult to justify the significance of the finding.", "strength_and_weaknesses": "Strengths:\n\n1. The proposed dataset contains fine-grained labels, including fine-grained video-level classes and frame-level classes.\n\n2. The authors investigate models' performance in several aspects and evaluate them on an interesting repetition counting task.\n\n3. The paper is clearly written and easy to follow.\n\nWeaknesses:\n\n1. The claims in this paper are all based on a very constrained scenery, the proposed Exercise Videos Dataset. I think the conclusions from existing well-known datasets are more convincing to me. Besides, the necessity of this new dataset is not well illustrated. Authors only claim that \"everyday general human actions\" are essential. The contribution does not meet the bar of ICLR. \n\n2. Some conclusions are not novel.\n\n    - Many state-of-the-art methods in action recognition opts the end-to-end training scheme. For example, DirecFormer\\[1\\] achieves the state-of-the-art performance on the Jester Dataset\\[2\\]. The paper of FineGym\\[3\\] compares a large range of methods with different input modalities. The methods with end-to-end training outperform the skeleton-based methods.\n\n    - Pretraining is a well-known trick in both end-to-end networks\\[4\\] and pose-based action recognition\\[5\\]. \n\n\n\n\\[1\\] DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition\n\n\\[2\\] The Jester Dataset: A Large-scale Video Dataset of Human Gestures\n\n\\[3\\] FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding\n\n\\[4\\] Omni-sourced Webly-supervised Learning for Video Recognition\n\n\\[5\\] Revisiting Skeleton-based Action Recognition\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PQTiCJLpgiq", "reviewer": "Reviewer_zfp8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts comparison experiments showing that end-to-end training on large-scale labeled video datasets without any form of frame-by-frame intermediate representation can compete with pose-based approaches. In addition, pre-training on a large can help achieve good performance in action recognition tasks.", "summary_of_the_review": "I have concerns about the uniqueness of the dataset and there are not enough evaluation tasks to support the claim. Addressing these issues will make this paper much stronger.", "strength_and_weaknesses": "Strength:\n1. The paper is written in a clear way and is easy to follow.\n2. The effort of proposing a new public-available dataset is great.\n\nWeakness\n1. There are only four activities, which is really a small number compared to other datasets.\n2. Datasets that focus on both human body movement and controlling camera motion have already existed. For example, PKU Multi-Modality Dataset is a large-scale multi-modalities action detection dataset containing 51 action categories in 3 camera views.\n2. In table 3, why SI-EN-BigFitness-10 is superior to SI-EN-BigFitness-all? And in the meanwhile, I3D-Kinetics-4 is worse than I3D-Kinetics-all.\n3. A minor suggestion: The author noted that end-to-end networks can reduce the runtime than that of pose-based networks. To make it a more straightforward comparison, can the author provide a running time comparison in the table?\n4. In table 4, it’s better to add the average error among the four categories.  \n5. Downstream task is too simple. More evaluation tasks such as activity recognition and action localization are expected. Only conducting experiments on counting is not enough to claim the performance on action recognition tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MabVxRsiPa", "reviewer": "Reviewer_ZAXq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dataset for fitness action recognition. The dataset consists of 4 classes of fitness exercises with a total of 5511 videos. Fine-grained labels of 40 classes are provided to assess the quality of the actions. The videos are 5-8 seconds long so there is no need for temporal localization. The authors compare end-to-end models and pose-based models on the proposed dataset.", "summary_of_the_review": "The dataset is small and there is no new method proposed. Some key related works are missing in the discussion.\n", "strength_and_weaknesses": "Strength\n1. The writing is clear and the proposed dataset is useful for fitness-related applications.\n\nWeaknesses\n1. Lack of novelty. This paper proposes a new dataset without additional innovation in terms of methods for this particular problem.\n2. Missing references to recent fitness/repetitive action datasets [1*, 2*]. They should be compared and a discussion of how the proposed dataset is different would be useful.\n3. Limitation due to fixed action classes. For the fitness action specifically, I think the authors should think about how the proposed dataset can help researchers scale to other types of fitness actions. Maybe class-agnostic is the way to go (like [1*])? The authors mention an internal 300k fitness dataset with 1.5k classes. Is the much larger dataset more useful in real-world applications?\n\n[1*] Counting Out Time: Class Agnostic Video Repetition Counting in the Wild, CVPR 2020\n[2*] TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting, CVPR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IazgT5loyPC", "reviewer": "Reviewer_GHiN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper contains two main contributions. Firstly, the authors have collected a new dataset for fitness activity recognition in a controlled environment (the setting of the videos is controlled by the authors). The dataset contains 5511 videos with 40 fine-grained activity labels. They then additionally collected a much larger, private dataset of 300 000 videos over 1536 classes and demonstrated that end-to-end activity recognition architectures (I3D or inflated 2D ConvNets) can perform on par or even outperform more traditional, pose-based approaches when pre-trained on a large amount of labeled data.", "summary_of_the_review": "This paper is more of an empirical study which is only interesting in the narrow context of fitness activity recognition (no conclusions that would be interesting to the broader community). Moreover, the experiments are not reproducible since the authors are not planning to release the corresponding dataset. I find that this paper would be more suitable for a workshop on fitness activity recognition.", "strength_and_weaknesses": "Strengths: \n\nThe paper is readable and is relatively easy to follow.\n\nExperimental evaluation seems sound.\n\nThe conclusions that end-to-end training outperforms handcrafted features given enough data is accurate, though highly not novel.\n\n\nWeaknesses:\n\nThe collected dataset does not seem to be valuable to the community since it is relatively small in scale and the difference to existing datasets is not significant. In particular, the authors argue that having a controlled environment (the only difference to FineGym), is a benefit. This argument requires a stronger justification, since in the other fields it has been consistently observed that the more diverse the dataset is the more robust the models trained on it are (e.g. COCO). The large scale version of the dataset would have been a significant contribution but the authors have no plans of releasing it.\n\nRelated work overview on action recognition is incomplete, essentially missing the entire body of work over the last 2 years. In particular, the transformer-based approaches are not discussed.\n\nThe main conclusion of the paper that pre-training deep network on large amounts of labeled data allows to outperform handcrafted features hasn't been novel since 2014. It is true that this has not been demonstrated for fitness activity recognition yet, but it has been for virtually every other task. \n\nExperimental evaluation is incomplete. The choice of the architectures is not well justified (they vary from dated to extremely dated) and even some obvious variants are not reported (e.g. what if you pre-train I3D on your private data? would it outperform the inflated 2D convolutions?). In additions, the authors need to evaluate their approach on existing fitness recognition datasets to confirm that their conclusions hold (NTU, FineGym).\n\nThe presentation still needs improvement. The captions are not sufficient to parse the figures/tables, and some important details are missing (e.g. how were the 4 exercises selected? why is there no positive label for dead bug?).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9yO7K-TIqjU", "reviewer": "Reviewer_vBip", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new large human-centric action recogntiion dataset. From empirical results testing all the SOTA models on this dataset, the paper shows that end-to-end learning is more effective and it can support temporally fine-grained tasks such as real-time repetition counting.", "summary_of_the_review": "Limited novelty, no new insights from the curated dataset, availability of the dataset is unknown,\nA reject", "strength_and_weaknesses": "[Strength]\nA new large human-centric dataset is proposed.\nPaper is well-written\n\n[Weakness]\nThe contributions of this paper are somewhat limited.\nA new dataset in action recognition is proposed; however, there are no new insights that we can learn from. The empirical results from the paper align with many existing works of literature where these papers have already discovered and discussed about these observations. \n\nThe authors are strongly recommended to re-submit this paper to other venues which focus on dataset creation.\n\nFor improvements, I list the following suggestions:\n\n[ethics] How would the authors collect such datasets? Have the authors considered privacy issues (such as indoor home settings and faces)? Has IRB been sought and approved before data collection?\n\n[dataset availability] Would the authors make the dataset publicly available? If so, pls make such a statement in the paper.\n\n[model development] Counting repeated actions in a video is interesting. The authors can specifically focus on this task and conduct more rigorous experiments and design new models.\n\n[novelty] After training and comparing all the SOTA models on this newly curated dataset, what new conclusions can we draw from these? Any new insights? Any model behavioral analysis, such as attention?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "6DJQD9tDjxW", "reviewer": "Reviewer_Pjes", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper collected a small-scale exercise video dataset with fine-grained labeling. Some baselines such as I3D, SI-EN, and GCN-based models (e.g., ST-GCN and MS-G3D) are provided for evaluation in the settings of few-shot learning. The baselines do not represent state-of-the-art models in their corresponding modalities, RGB video and Pose, respectively. Regarding the titled question, there are already plenty of results in the existing literature on datasets such as NTU RGB+D, which can draw the paper's conclusion. ", "summary_of_the_review": "Unclear writing, lack of justification for the proposed fine-grained labeling for the dataset, outdated baselines, and insufficient discussion on the experimental results make this work not ready for publication in its current form.", "strength_and_weaknesses": "Strength\n\n1.\tThe varied scene and lighting settings are interesting. It would be good to provide an analysis of its effect, e.g., which exercises or classes can be highly affected by poor illumination conditions.\n\n2.\tEnd-to-end methods are compared with pose-based methods, where the former can occasionally be better than pose-based ones.\n\n\nWeakness\n\n1.\tGiven the poor performance of few-shot experiments, why not also report the fully supervised setting? There is no training from scratch results for end-to-end methods, while results of ST-GCN- and MS-G3D-Scratch can be based on a consistent number of joints with other pre-training-based results. Why some results in Table 4 are large than 100? You can report the ground-truth counting and the missed counting. \n\n2.\tThe writing is not clear and some descriptions are not proper.\na.\t“As a testbed to study the relevance of such pipelines”, what is the relevance?\nb.\tThe dataset is collected in an indoor setting, referring to it as “In the wild” is not proper.\nc.\tWhat is GMACs?\nd.\tSometimes in Table 3, Pose-based pipeline can be better than end-to-end methods. Why?\n\n3.\tLack of discussion on some results, Table 3, the SI-EN-BigFitness-all underperformed than SI-EN-BigFitness-10. Why more pre-training data will lead to poor performance?\n\n4.\tThere are no clear standard evaluation settings for few-shot learning and repetition counting. “Repetition counting is an inherently temporal prediction task”, there are many segmentation works that can be expanded as a counting task. The exact repetition counting setting is not clear and lacks comparison with existing works. What is the under-represented class ”within-repetition””?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4INm6TcpkHj", "reviewer": "Reviewer_v9Hi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author asked and answered one question of action recognition, whether end to end learning is enough for the activity recognition. In order to answer this question, the author created a new labeled dataset, Exercise Video Dataset. With the labelled data, end-to-end learning on raw pixels can compete with state-of-the-art action recognition pipelines based on pose estimation.", "summary_of_the_review": "Overall, I think the current paper did not provide a clear motivation and ideas to the community. In my opinion, it is more like a technical report for now while the experimental results are not convincing. ", "strength_and_weaknesses": "The strengths of the paper are: \n+ Interesting perspective\n+ good presentation \n+ comprehensive experiments \n\nThe weaknesses of the paper are:\n- Actually, I am not sure about the contributions of this paper, is it the dataset the authors created or the comparison experimental results the authors tried to show? The proposed and published dataset is Exercise Video Dataset, however it severed as a testing/evaluation dataset in the paper, while the BigFitness is the dataset experimented in the paper. I did not quite get the idea from the authors. \n- The paper proposed that \"end-to-end learning on raw pixels can compete with state-of-the-art action recognition pipelines based on pose estimation\", however the testing dataset had the same domain as BigFitness, it is difficult to say the domain gap between the test/evaluation dataset or the pose estimation/pose oriented dataset provide the benefits. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "18J2buk6oq", "reviewer": "Reviewer_2dWa", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes a new video dataset on action fitness. The dataset covers a fine-grained taxonomy with a balanced distribution of videos per class. The paper proposes two new end-to-end architectures. An analysis of end-to-end vs pose-based architectures and their performance on the proposed dataset on two tasks - classification accuracy and repetition counting has been shown. ", "summary_of_the_review": "The paper lacks a detailed analysis both in terms of architecture setup and different viewpoints on which an analysis is shown on an action dataset (dataset stats, temporal analysis, more in-depth analysis of action classes, etc.).There’s no new problem that the proposed dataset is trying to solve and no new metric proposed in the paper for the proposed dataset. Due to these reasons, I tend to reject the paper.", "strength_and_weaknesses": "Strengths\n\nSince the dataset is collected in the wild, the dataset covers a wide range of lighting variations and scene settings. The proposed end-to-end architecture in general outperforms other architectures shown in the paper and is more computationally efficient than the pose-based architectures.\n\nWeakness\n\n1. The Paper lacks a lot of analysis.\n    a ) Activity Selection - How these specific activities have been shortlisted? Do these four activities cover all types of variations we usually see in an Action Dataset? \n    b) Architecture selection - How were the architectures selected for End-to-End training? The paper does not have an in-depth analysis to make a conclusion. There are multiple conventional 2D, 3D CNN, and transformer-based architectures that need to be tested on the dataset. There are also multiple architectures for pose-based that needs to be coveted for thorough analysis. Why was an efficient net selected as a backbone for SI-EN?\n\n2. A table that compares the exact stats of the proposed dataset vs other action recognition datasets will help in the scale of the dataset and analysis of the variation of the dataset.\n\n3. Missing details\n  a) The number of epochs trained on pre-training datasets and the proposed dataset. \n  b) The average duration of each activity in the dataset?\n  c) Is the dataset trimmed or untrimmed?\n  d) What is the original resolution of the video collected? How the quality of videos has been assessed?\n  e) Is the dataset trained with 4 classes or 40 classes for the action classification task?\n  f) Performance on the whole dataset training.\n  g) In Table 3, what do 1, 4, 10, and all means? Why this setup is only for end-to-end and not for the pose-based pipelines?\n\n4. Pre-training - How much overlap the internal training dataset has with the proposed dataset in terms of environment setup, viewpoint variation, and action length? These factors impact a lot. If the domain is the same, then that could be the result of better performance than Kinetics pre-trained.\n\n5. The accuracy on the dataset is already between 60 - 70% with only 100 labels per class with baseline architectures. This doesn’t leave a lot of gaps in the improvement of the performance and questions how challenging the dataset actually is.\n\n6. The comparison and a general comment of SI-EN outperforming other architecture for the Repetition counting tasks don’t seem correct to me. In Table 4, if we look into the 1st annotation scheme, MSG3D outperforms SI-EN in 2/4 of activities. For 2nd annotation scheme, STGCN outperforms SI-EN by a huge margin for all four activities (5-10%).\n\n7. Missing citations - UCF101, HMDB51\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject