Ballroom Dance Movement Recognition Using a Smart Watch and Representation Learning
Authors: Varun Badrinath Krishna
Venue:  (2021)
Keywords: ballroom
sequence
deep
learning
machine
markov
prior

Abstract: Smart watches are being increasingly used to detect human gestures and movements. Using a single smart watch, whole body movement recognition remains a hard problem because movements may not be adequately captured by the sensors in the watch. In this paper, we present a whole body movement detection study using a single smart watch in the context of ballroom dancing. Deep learning representations are used to classify well-defined sequences of movements, called \emph{figures}. Those representations are found to outperform ensembles of random forests and hidden Markov models. The classification accuracy of 85.95\% was improved to 92.31\% by modeling a dance as a first-order Markov chain of figures.

Under review as a conference paper at ICLR 2021
BALLROOM DANCE MOVEMENT RECOGNITION USING A SMART WATCH AND REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Smart watches are being increasingly used detect human gestures and move- ments. Using a single smart watch, whole body movement recognition remains a hard problem because movements may not be adequately captured by the sensors in the watch. In this paper, we present a whole body movement detection study using a single smart watch in the context of ballroom dancing. Deep learning representations are used to classify well-defined sequences of movements, called figures. Those representations are found to outperform ensembles of decision trees and hidden Markov models. The classification accuracy of 85.95% was improved to 92.31% % by modeling a dance as a first-order Markov chain of figures.
INTRODUCTION
Recent work has used low-cost smart watches to track the movement of human body parts. ArmTrak tracks arm movement, assuming that the body and torso are stationary (Shen et al., (016)). In this paper, we perform whole body movement recognition using single smart watch, which is a hard problem given that body movements need to be inferred using readings taken from a single location on the body (the wrist). The movements in the study are from ballroom dancing, which engages tens of thousands of competitors in the U.S. and other countries. Competitors dance at different skill levels and each level is associated with an internationally recognized syllabus, set by the World Dance Sport Federation. The syllabus breaks each dance into smaller segments with well-defined body movements. Those segments are called figures. In the waltz, for example, each figure has a length of one measure of the waltz song being danced to; the entire dance is a sequence of 40 to 60 figures (depending on the length of the song). The sequence is random, but the figures themselves well-defined. The sequence is illustrated in Fig.
The International Standard ballroom dances are a subset of ballroom dances danced around the world, and they include the waltz, tango, foxtrot, quickstep and Viennese waltz. A unique charac- teristic of all these dances is that the couple is always in a closed-hold, meaning they never separate. Also, both dancers in the couple maintain rigid frame, meaning the arms and torso move together as one unit. The head and the lower body, however, move independently of that arms-torso unit. Our hypothesis in this paper is that the figures in each of these dances can be recognized with high accuracy using deep learning representations of data obtained from a single smart watch worn by the lead in the couple. That possible because the rigid frame makes unnecessary to separately instrument the arms and torso, and because most figures are characterized by distinct movements (translations and rotations in space) of the arms and torso. We refer the interested reader to the website www bal 1 roomgui de. com for free videos and details on the various syllabus figures in all the International Standard ballroom dance styles.
In this paper, we validate our hypothesis on the quintessential ballroom dance- the waltz. We chose 16 waltz. figures that are most commonly danced by amateurs. The full names of the figures are included in Appendix A Our goal is to accurately classify those figures in real-time using data from a smart watch. That data can be pushed to mobile devices in the hands of spectators at ballroom competitions, providing them with real-time commentary watched being performed. That is an augmented-reality platform serving laymen in the audience who want to become more engaged with the nuances of the dance that they are watching.
the moves that they will have just
The main beneficiary of the analysis of dance movements would be the dancers themselves. The analysis will help them identify whether or not they are dancing the figures correctly. If a figure

---

Under review as a conference paper at ICLR 2021
Dance (Eg: Waltz, Quickstep, Tango, Foxtrot)
Figure
Figure 2
Figure N
Sensor Readings
Sensor Readings
Sensor Readings
Figure I: A dance is random sequence of well-defined figures (movements). If the dancer is instrumented with sensors, the figures emit sensor readings that should be similar for each type of figure.
confused for different figure, it may be because the dancers have not sufficiently emphasized the difference their dancing and need to improve their technique on that figure. That confusion metrics could also be used by competition judges to mark competitors performing figures; that task is currently done by eye-balling multiple competitors on the floor, and challenging when there are over couples to keep track of.
how well dancers
We make three main contributions detecting ballroom dance movements using learning represen- tations.
First, we show that representations using data from a single smart watch are sufficient for discriminating between complex dancing movements.
Second, we identify and evaluate six learning representations that can be used for clas- sifying the figures with varying accuracies. The representations are 1) Gaussian Hidden Markov Model, 2) Extra Trees Classifier, 3) Feed-Forward Neural Network, 4) Recurrent Neural Network (LSTM), 5) Convolution Neural Network, and 6) a Convolution Neural Network that feeds into Recurrent Neural Network.
Finally, we model the sequence figures as Markov chain, using the fact that the tran- sitions between figures are memoryless. We use the rules of the waltz to determine which transitions are possible and which are not. With that transition knowledge, we correct the immediately previous figure's estimate. This leads to an average estimation accuracy im- provement of 5.33 percentage points.
2 DATASET DESCRIPTION
2.1 DATA COLLECTION
The data was collected using Android app Samsung Gear Live smart watch. The app was developed for this work on top of the ArmTrak data collection app. We were able to reliably collect two derived sensor measurements from the Android API:
Linear Acceleration. This contains accelerometer data in the X, Y and Z directions of the smart watch, with the effect of gravity removed.
Rotation Vector. This provides the Fuler angles (roll, pitch and yaw) by fusing accelerom- eter, gyroscope and magnetometer readings in the global coordinate space. We use only the yaw (rotation about the vertical axis) in this study, and that is based on prior knowledge that roll and pitch are insignificant in the waltz figures included the study.
In total, we collected readings from 4 sensor axes (three from the Linear Acceleration and the yaw from the Rotation Vector sensors). The readings were reported by watch operating system asyn- chronously, at irregular intervals, whenever a change was sensed. order to facilitate signal pro- cessing, we downsampled the data such that each figure contained exactly 100 sensor samples, which was possible because the effective sampling rate was greater than that. The downsampling was done by taking the median (instead of the mean, which is sensitive outliers) values of 100
2

---

Under review as a conference paper at ICLR 2021
-2.2
-1.8
1.8
3.0
-20
1.4
2.0
-2.6
-44
-2.8
226
1.0
1.0
-28
-3.0
-32
-3.2
0.6
0.0
(a) Natural Tum (NI): right turning
22
1.0
2.4
2.2
233
0.8
2.0
2.4
1.8
0.6
2.5
1.6
0.4
1.4
266
-27
0.2
1.0
-288
0.0
0.8
(b) Reverse Turn (RI): left turning
Figure 2: Raw measurements of yaw: 4 samples from two figures.
evenly-spaced time windows. From this point in the paper, when refer to "samples", refer to an observation for figure of dimension 4 x 100 as one sample.
The app developed in such way that the button the watch that started recording the afore- mentioned sensor measurements also simultaneously started playing the music via Bluetooth speak- time-synchronizee.
ers. That ensured that the music and the recording of the movements
For all the data that was collected, we used the same rendition of the classic song "Moon River". We performed manual segmentation of the song using its beats offline, and that used segment the time series data for the entire dance sequence into 2. 2.1-second-lome sonooondd-oon--egeent corresponding to figures in the dance. We noted the song intro length (where no dancing was performed) and ignored all data in that period. For each figure, we extended the window of measurements equally at the beginning and the end by 0.35 seconds account for slight errors in dancer timing. That ensured that the window captures the figures even if the dancer was slightly early or late to begin/finish dancing the figure.
The yaw data for 4 figure samples corresponding to two different figures are illustrated in Fig. can be seen that right-turning figures tended to record yaw readings with an upward trend, while left-turning figures recorded yaw readings with downward trend. Slight differences between the samples for each figure can be attributed to differences in the dancers' timing and execution.
2.2 CROSS-VALIDATION GROUPINGS
In total, we collected 818 figure samples across 16 different waltz figures, over dances (figure sequences). Thus, the input data had dimension of 3p818x4x100 for 818 figure samples, sensors, and 100 measurements per sensor per figure sample.
The small size of the dataset and subsequent difficulty in collecting additional data during the Covid- 19 pandemic made the learning problem more challenging. We had only 14 sequences in total, and that is too small of dataset to learn dependencies between figures. Therefore, our focus is independently classifying the figures using the 818 samples, and leveraging the Markov property to enforce dependencies between figures.

---

Under review as a conference paper at ICLR 2021
T
T
T
Figure 1
Figure
Figure 45
E
E
E
Sensor Readings 1
Sensor Readings 2
Sensor Readings 45
Figure 3: Hidden Markov Model representation.
The 818 samples came from 14 separate dances (figure sequences) and we performed 7-fold cross- validation with two dances per voss--valiation group (assigned randomly). That ensured sequences of figures (dances) were not split across different cross-vac coos--alldationn groups. It also allowed us to test our representations' accuracy for each sequence whole.
2.3 LABELING GROUND TRUTH
Each dance was recorded video that labels (ground truth) could be given to the data segments corresponding to the figures. The labels are listed in Appendix
3 MARKOV TRANSITIONS
The sequence of figures in each dance can be modeled as a Markov chain. The probability of observing the next figure is dependent on the current figure, but independent of past figures given the current figure. The reason is as follows.
Certain figures end on the right foot, while others end on the left foot. Similarly, certain figures begin on the left foot, while others begin on the right foot. The probability of going from a figure ending on the right foot another figure beginning on the right foot zero (and the same applies to the left foot). That is because of the physies of the dance and the way weight is distributed between the feet. Similarly, some figures must be followed by figures that move forward while others must be followed by figures that go backward. Therefore, each figure constrains the immediate next figure, but the sequence is memoryless.
Using the above rules, constructed a transition matrix for all figures, and that is given in the Appendix Table 3 We essential gave a zero probability to impossible transitions, and equal probability to all possible transitions. Therefore, not based on real training data. The advantage of the unbiased transition matrix that the same matrix can be used across different couples since it very general. It does encode unique habits of certain couples, where there is a tendency to follow patterns. If a biased approach were taken, unique transition matrix could be learned for each couple, but would not generalize other couples.
transition matrix is completely unbiased, and
4 HIDDEN MARKOV MODEL REPRESENTATION
The dance can be represented as Hidden Markov Model (HMM) where the states represent figures that emit sensor readings, as illustrated in Fig. 3 Although the state space discrete, the emission space is continuous because the sensor readings continuous. As result, the HMM cannot be solved using a discrete emission probability matrix. Instead, we assumed Gaussian emission probabilities, resulting in Gaussian HMM.
We used the HMMLearn Python library (hmm) to estimate the transition and emission probabilities while fitting the input data. We initialized the transition probabilities with the trained transition matrix described in Section and initialized state vector with the actual initial state obtained from the ground truth.
The problem with the HMM approach for this task is that the HMM a generative model, and not discriminative model. At stage does the model take the actual known labels perform
4

---

Under review as a conference paper at ICLR 2021
Input for each figure sample
700 samples x 100 measurements in time x sensors
Conv1D (64, 3)
LSTM (W)
Conv1D (64, 3)
D
Conv1D (64, 3)
Conv1D (64, 3)
LSTM (W)
MaxPooling
MaxPooling
Conv1D (128, 3)
Conv1D (128, 3)
Dense (16)
(Softmax)
Conv1D (128, 3) MaxPooingiFlatte
Conv1D (128, 3) MaxPooinggFlatte
LSTM
Dense (W)
LSTM (W)
50% Dropout
Dense (W)
Dense (16) (Softmax)
50% Dropout
(D: Depth, W: Width)
Dense (16)
(Softmax)
ConvNet LSTM
ConvNet
Figure 4: Deep Neural Networks Representations.
classification. It simple estimates states using the probability information and we assigned labels to the states by fitting the training set, and matching the states estimated by the HMM with the known labels. The approach achieved an accuracy of 35.93% on the validation sets, averaged across the cross validation groups
DECISION TREE REPRESENTATION
We used the Extra Trees Classifier provided in the Scikit-Leam Python library (Pedregosa al., 2011] to classify figures directly from the downsampled data. That classifier is an ensemble method incorporating several (250 in our case) decision trees and aggregating their results. Each input sample was in R2400 (4 sensors, 100 time series points per sensor), so there were 400 features. The approach achieved an average accuracy of 72.2%.
DEEP NEURAL NETWORKS REPRESENTATIONS
We tested three different deep neural network architectures, illustrated in Fig. aand standard feed- forward neural network (not illustrated). In all layers, we used ReLu activations, except for the LSTM layers for which we used Sigmoid activations. The inputs all the networks are the same, and are based on the cross-validation groupings described in Section 2.2 The outputs are also the same, because we want to obtain the probabilities associated with the different figures. Therefore, we use softmax output layer with a categorical cross-entropy loss function.
package for Python, which provides an abstraction for Tensor- flow (Abadi et al. 2015 backend. We trained the networks using the Adam solver (Kingma & Ba 2014
We used the Keras Ker 2017
Feed-forward: All layers are densely connected. There are D dense layers, each of width W. We varied 1) and W, given in Appendix C
Convolutional (ConvNet): Since we are looking at I-dimensional streams, we used 1- dimensional convolution layers. The first two layers convolutional with 64 filters and

---

Under review as a conference paper at ICLR 2021
Table 1: Results for Mean Accuracy with 7-fold Cross- Validation (%)
Classifier + Markov
Classifier Only
Correction
Random Guess
6.25
N.A.
Gaussian HMM
35,93
N.A.
Extra Trees Classifier
72.20
73.48
Feed-forward
80.86
85.63
ConvNet
83.01
88.90
LSTM
83.73
92.31
ConvNet +LSTM
85.95
88.41
kernel size 3. The next two layers are preceded by max pooling operation, and contain 128 filters and a kernel size of 3. Another max pooling operation is added before D dense layers of width W. This particular architecture was inspired from the example in (Ker) 2017)
Recurrent (LSTM): This layer has Long Short-term Memory (LSTM) layers (Hochreiter & Schmidhuber 1997p. The layers have W nodes with time history of 100 and 4 features each (as per the input dimension). This network took the longest time to train because had the most parameters.
Hybrid Convolutional and Recurrent (CoSSTe:+L This network is hybrid of the aforemennioned convolutional and recurrent architectures. One LSTM layer replaces the dense layers in the convolutional architecture. The complexity of this layer is less than that the pure LSTM network because the convolutional layers reduce the input dimension- ality. As result, this network trains faster than the pure LSTM architecture. This hybrid architecture was inspired from related work (Morales Roggen 2016).
As described previously, we did attempt learn the entire sequence because of the number of sequences was too small. Instead, we focused on representations for classifying each figure independently.
MARKOV CORRECTION
In this section, we propose simple approach to combine the results of the learning representations (referred to as classifiers) with the Markov structure of the dance. Let i, E O, 1,.,, be possible state from the 16 different figures, and let X, be the figure at time index t. Then at each time index t,
1. We assume that the classifier is correct for the current figure and suppose X, j
2. We correct the immediately preceding figure X,- as follows.
X,-1 arg max P(X.-1 = =||,, =
arg max P(X: = = P)
We get P(X, = j| = i) from the trained transition matrix described in Section and P(X,1 i) from the classifier.
EVALUATION RESULTS
The evaluation was performed with 7-fold validaaal with groupings described in Section 2.2 The results are summarized below and include the best configurations for the neural networks. The results for all the different configurations given in Appendix C There is directly related work that can be used for comparative evaluation. However, the accuracies presented can be compared with the accuracy of a random guess, which is 6.25%.
From the results, is clear that the neural networks approaches outperform the Extra Trees Classifier (ensemble of decision trees). The hybrid approach with the convolutional and L.S layers performs

---

Under review as a conference paper at ICLR 2021
CTR
CTR
DR
08
DR
08
N2
00.
N2
Acbua
NST
Actur
NST
Contaaio Net
na
nz
P2
0.
01
01
Weave
Wevve
CTR
DR
ACC
DR
CTR
DR
NST
XC
Wenee
Predicted
Predicted
(a) Classifier Only
(b) Classifier with Markov Correction
Figure 5: Confusion matrices. diagonal entry is p/100.
When the actual figure is correctly classified p% of the time, the
140
120
100
Frequency
80
60
40
20
-15
-10
-5
15
20
25
Markov Accuracy Improvement (Percent Points)
Figure 6: Improvement achieved using Markov correction across all neural network configurations and cross-validation test sets.
the best. It is also clear that the number of samples (figures) in the training sets was sufficient to achieve good performance the validation sets. Collecting more data may have improved the performance further, but despite data collection constraints, the Markov correction provided the required boost in performance.
On average, the Markov correction approach proposed in Section is found to benefit all the clas- sifiers. We illustrate this using confusion matrices, which capture the results for individual figures. Ideally, the confusion matrix should be the 16 x 16 identity matrix, because that would mean that the predicted figure was always the actual figure. However it can be seen in Fig. 5(a) that the left-foot closed-change (L.CC) is most often confused for whisk (W). However, whisk is almost always followed by a progressive chasse (PC). The Markov correction approach recognizes this from the transition probabilities and corrects the estimation of whisk to left-foot closed change soon as sees that that figure was not followed by progressive chasse. The improved classification results illustrated in Fig. ()).
Markov correction sometimes hurts the classification results because the assumption that the current figure correct may not be valid. If the current figure has been incorrectly classified, then that emror in classification could be propagated to the previous figure. Fig. 6 shows the distribution of improvements. On average, the improvement was 5.33 percentage points.

---

Under review as a conference paper at ICLR 2021
9 RELATED WORK
To the best of our knowledge, we are the first to use a smart watch for dance recognition and dance analytics. Multiple accelerometers used input for dancing video games in (Crampton al. 2007]. VICON systems were proposed in (Dyaberi et al., 2004) and video recognition was used in (Matthew Faircloth, 2008). Those approaches do not work in scenario where there are multiple ballroom dancers simultaneously on the floor, leading to occlusion. Also, they are expensive and suited for amateurs.
Music segmentation studies for dance detection purposes are presented in (Shiratori et al. 2004). Models for turning motions in Japanese folk dances modeled from observation in (Rennhak al., 2010). Signal processing techniques used in dance detection are reviewed in (Pohl 2010p. Ballroom dance styles are differentiated in (Schuller et al., 2008) from the music that is being played.
For human activity recognition, ensembles of deep LSTM networks were proposed in (Guan & Plötz
2017). but this approach not suitable for teme prediction because is too slow. A single deep LSTM network took nearly whole day to train, from our experiments, and loading the weights for prediction was also very slow. Convolutional neural networks were proposed (Zeng et al. 2014)) Our approach is similar, but we use more convolutional layers, as suggested in the Keras time series classification example (Ker) (017). Our best results were obtained using the hybrid architecture between convolutional and recurrent neural networks, and that was proposed for human activity recognition in (Morales & Roggen 2016).
More generally, for time series classification, convolutional networks were proposed in (Zhoo al. 2017) and (Cui et al. (016).
10 CONCLUSION
In this paper, we presented a study of whole body movement detection using single smart watch in the context of competitive ballroom dancing. Our approach was able to successfully classify move- ment segments from the International Standard Waltz, using deep learning representations. The representations alone achieved a maximum accuracy of 85. 85.95% averaged over 7 cross-validation groups. Using the fact that the segments can be represented improved to 92.31 % by correcting the prediction for each preceding segment. The deep learning representations outperformed ensembles of decision trees, and Gaussian HMM representation per- formed poorly because it was not discriminative. Despite the small size of the training set, the representations were able to generalize well to validation sets.
Markov chain, the accuracy was
ACKNOWLEDGMENT
Redacted for peer review.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar. Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scat scale machine learning on heterogeneous systems, 2015. htthht:s:: /ttp///wwww. wwwsorflow..ooooooo org/ Software avail- able from tensorflow.org.
N. Crampton, K. Fox, H. Johnston, and A. Whitehead. Dance, dance evolution: Accelerometer sensor networks as input to video games. In 2007 IEEE International Workshop on Haptic, Audio and Visual Environments and Games, pp. 107-112, Oct 2007.
doi: 10.1 10.1109/HAAE.2207..05
8

---

Under review as a conference paper at ICLR 2021
Zhicheng Cui, Wenlin Chen, and Yixin Chen. Multi-scale convolutional neural networks for time www.v.org/bb........o.ooooas
series classification. CoRR, abs/1603.066m 0.090, 2016. URL ortp:////////..o...ooorro///////
06995
Vidyarani M. Dyaberi, Hari Sundaram, Jodi James, and Gang Qian. Phrase structure detection in dance. Proceedings of the 12th Annual ACM International Conference on Multimedia, MUL-
TIMEDIA '04, pp. 332 335, New York, NY, USA, 2004. ACM. ISBN --8113-893-8 doi: 10. 1145/1027527.1027668 PRLaatt://dol //t:i.//ii.. acm. wwt.oa//............/... 1ttp11..114/112221111
1027604
Yu Guan and Thomas Plötz. Ensembles of deep Istm leamners for activity recognition using wearables. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1():11:1-1:228, June
2017. ISSN 2474-9567. doi: 10.1145/3090076.
3090078
hmmlearn. hmmlcarn. Available
Eutorial. html available models Accessed December 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput. 98)) 1780, November 1997. ISSN 0899-7667. doi: 10.1 197..9............... URL http://dx.
doi www. 12 neco. 1997 8.8.9.8... 1735
Getting started with the Keras Sequential model. Keras, 2017.
Available http::t
Accessed December 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL http: /arxivv.rrivv.o/aa orw.oogggaoo..o.. babs/1412. 6980
Ryan Matthew Faireloth. Combining audio and video tempo analysis for dance detection. M.s. thesis, University of Central Florida, 2008.
Francisco Javier Ordóñez Morales and Daniel Roggen. Deep convolutional feature transfer across
mobile activity recognition domains, sensor modalities and locations. In Proceedings of the 2016
ACM International Symposium on Wearable Computers, ISWC 16, pp. 92-99, New York, NY, USA, 2016. ACM. ISBN 978-1-4503-4460--9 doi: 10.1145/9971766.297776.. URL http:
7/doi. acm. 10t0//00.115//9117333322200
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Henning Pohl. Motion pattern recognition for interactive dance. M.s. thesis, TECHNISCHE UNI- VERSITÄT DARMSTADT, 2010.
B. Rennhak, T. Shiratori, S. Kudoh, P. Vinayavekhin, and K. Ikeuchi. Detecting dance motion struc- ture using body components and turning motions. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2212-22222222 2269,
Oct 2010. doi: 10. 109//ROS..000.555200 2010.5649303.
Björn Schuller, Florian Eyben, and Gerhard Rigoll. Tango or waltz?: Putting ballroom dance style into tempo detection. EURASIP J. Audio Speech Music Process. 2008:6:1 January 2008. ISSN 1687-4714. doi: 10.1 10.11155/00//4441133, 00C8/846135
htLhttp:://kx /wtp:/xxxxciiiiicoooooo.....doot doi org (t..::://xx.dl..or//11...... 1155/
Sheng Shen, He Wang, and Romit Roy Choudhury. I am smartwatch and can track my user's arm. In Proceedings the 14th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys 16. pp. 85 5699, New York, NY, USA. 2016. ACM. ISBN 978- -2699--2269 8. doi: 10.11452906388.9964444 URL http:: /ttp:///oo... acm. 01g/10.
2906407
Takaaki Shiratori, Atsushi Nakazawa, and Katsushi Ikeuchi. Detecting dance motion structure through music analysis. In Proceedings of the Sixth IEEE International Conference on Auto- matic Face and Gesture Recognition, FGR' 04, pp. 857-863 Washington, DC, USA, 2004. IEEE Computer Society. ISBN 0-7695-2122-3. URL http:://a //d1. acm. http:/o//ggctiiii http:/iiaaiiiin..cmo cfm?
id=1 949767 1949917

---

Under review as a conference paper at ICLR 2021
M. Zeng, L. T. Nguyen, B. Yu, J. Mengshoel, J. Zhu, P. Wu, and J. Zhang. Convolutional neural networks for human activity recognition using mobile sensors. In 6th International Conference Mobile Computing, Applications and Services, pp. 197-205, Nov 2014. doi: 10.4108/icst. mobicase. (rbiicase.2014,25 2014,257786.
B. Zhao, H. Lu, S. Chen, J. Liu, and D. Wu. Convolutional neural networks for time series clas- sification. Journal of Systems Engineering and Electronics, 28(1):162-169, Feb 2017. doi: 10.21.99/
A APPENDIX: WALTZ FIGURE INFORMATION
Ballroom dancing competitions in the U.S. are conducted at multiple skill levels. The syllabus at the skill levels restricts which figures can be used. In this paper, we focus Waltz figures at the Newcomer and Bronze skill levels, because the majority of amatcur dancers compete at these skill levels. The figures also be used by dancers more advanced skill levels, such Silver and Gold. Table [ gives the names of the figures that we consider in this paper, along with the short names used throughout the paper.
Table 2: Waltz Figure Names
Left Foot Figures
Right Foot Figures
Left-foot Closed Change (LCC)
Right-foot Closed Change (RCC)
Natural Turn 4-6 (N2)
Natural Turn 1-3 (N1)
Natural Spin Turn (NST)
Reverse Corte (RC)
Reverse Turn 1-3 (RT)
Reverse Turn 4-6 (R2)
Chasse Right (CTR)
Chasse from Promenade (PC)
Outside Change (OC)
Basic Weave (Weave)
Double Reverse (DR)
Whisk (W)
Back Whisk (BW)
Back Lock (BL)
10

---

Under review as a conference paper at ICLR 2021
B
UNIFORM (UNBIASED) TRANSITION PROBABILITIES
Table 3 descrrbbs the uniform (unbiased) transition probabilities between figures of the international standard waltz.
Table 3: Figure Transition Probabilities
BL
BW
CTR
DR
LCC
N1
N2
NST
OC
PC
R1
IR2
RC
RCC
W
Weave
BL
0.20
0.20
0.00
0.00
0.00
0.00
0.20
0.20
0.20
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BW
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
CTR
0.20
0.20
0.00
0.00
0.00
0.00
0.20
0.20
0.20
0.00
0.00
0.00
0.00
0.00
0.00
DR
0.00
0.00
0.20
0.20
0.20
0.00
0.00
0.00
0.00
0.00
0.20
0.00
0.00
0.00
0.20
0.00
LCC
0.00
0.00
0.00
0.00
0.00
0.33
0.00
0.00
0.00
0.33
0.90
0.00
0.00
0.33
0.00
0.00
N2
0.00
0.00
0.00
0.00
0.00
0.33
0.00
0.00
0.00
0.33
0.00
0.00
0.00
0.33
0.00
0.00
NST
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.33
0.33
0.00
0.00
0.33
OC
0.33
0.00
0.00
0.33
PC
0.00
0.00
0.00
0.00
0.00
0.33
0.00
0.00
0.00
0.33
0.00
0.00
0.33
0.00
0.00
RI
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.33
0.33
0.00
0.00
0.33
R2
0.00
0.20
0.20
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.20
RC
0.20
0.20
0.00
0.00
0.00
0.00
0.20
0.20
0.20
0.00
0.00
0.00
0.00
0.00
0.00
RCC
0.00
0.00
0.20
0.20
0.20
0.00
0.00
0.00
0.00
0.00
0.20
0.00
0.00
0.00
0.20
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
C APPENDIX: DETAILED RESULTS FOR NEURAL NETWORK HYPERPARAMETER CONFIGURATIONS
The following table contains the detailed results and the number of model parameters for each of the architectures described in Fig Note that D refers to the number of hidden layers for the feed- forward neural network, but it refers to the number of LSTM/Dense layers in the other architectures (described in Fig
Architecture
Width (W)
D
Classifier Accuracy (%)
Markov Correction (%)
Model Parameters
500
1
81.07
87.58
671,684
500
2
80.19
85.46
922,184
1000
1
83.01
87.98
1,256,184
ConvNet
1000
2
82,30
87.39
2,257,184
2000
1
82,40
86.46
2,425,184
2000
2
82.04
88.90
6,427,184
1
84.28
89.51
4,619,184
CoMVNet+LSTM
2000
1
85.95
92.31
18,1
3000
1
85.93
91.86
37,683,184
78.87
85.27
208,516
2
80.14
85.63
459,016
3
79.04
83.10
709,516
1000
1
78.17
83.82
417,016
Feed-Forward
1000
78.73
84.23
1,418,016
1000
3
79.76
85.05
2,419,016
2000
1
78.77
83.80
834,016
2000
2
80.86
84.90
4,836,016
2000
3
76.83
80.98
8,838,016
500
1
83.73
88.26
1,018,016
LSTM
1000
1
83.19
88.41
4,036,016
2000
1
78.38
84,85
16,072,016
11


Reviews:
[{"review_id": "QeF9s_iwavE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a system to classify dance figures using a wrist-worn device, exploring a number of different classifiers, and incorporating prior knowledge about dance to improve overall performance.\n\nWeaknesses / comments / suggestions:\n* Motivation / problem setting. The authors envision a scenario where the audience is informed about the individual dance moves through automated recognition. To motivate the use of ML in this setting this could be extended to illustrate the potential of ML in this area, particularly when it comes to amateurs. For example, representation learning would have the potential to allow assessment of the quality of individual moves, or the characterisation of an overall routine w.r.t. variety, where it could be used as a training aid.\n* Automatic segmentation of dance moves is not explored. The authors are significantly simplifying the problem by (manually) segmenting each individual figure in the data-set, and by removing any \"non-dancing\" behaviour from the boundaries. I believe in a practical setting, as envisioned by the authors, this information wouldn't be readily available. Also, it would make for an interesting problem - can you recover the boundaries between the different figures if presented with a continuous signal and potentially features from an audio stream?\n* Learning representations vs classification. While it is true that the NN-based approaches implicitly learn a representation of the data, they are exclusively used as classifiers in this work. It would be interesting to explore what these representations (e.g. activations in penultimate layer) actually look like, and if e.g. the dance moves that start with the same foot are similar in that representation space. \"Trajectories\" in that representation space would further be interesting to potentially 1) segment individual figures an 2) characterise the overall performance / variety of a complete routine.\n* Related work: While this application might be novel it is nevertheless closely related to other applications of human activity recognition. The literature on this is extensive, and the paper would benefit from characterising how the problem investigated differs from settings explored in related work, e.g. for other somewhat related sports (e.g. martial arts, rock climbing).\n\nIn summary, this paper explores some aspects of the recognition of dance figures in a sort of idealised scenario. Overall the paper is clearly written, albeit for a different audience than what is typically found at ICLR. The paper describes and motivates an application, describes data collection and explores a number of (standard) methods. While it is a nice application, the suitability for this community is limited as it will be difficult to transfer findings to other application settings and there is limited technical novelty / insight.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ac3H-dj8LB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose an approach to perform classification of ballroom dance movements (called figures) captured by the sensing mechanism of a smartwatch and discriminated via different ANN architectures. The sequence of figures are modelled as a Marlov chain, which work in  a generative+discriminative fashion to output the final prediction of the model. Authors also present a dataset collected specifically for this work, to perform the inference of the algorithms included in the evaluation. Results show a remarkable accuracy, but are not compared to any existing state of art due to limited related work.\n\n-------------------------------------------------\nContributions\n\nThe paper addresses an application of HAR which I see can be of interest for the community due to its novelty. The dance recognition it’s not a common domain and its analysis in terms of feature representation can fit correctly within the scope of the conference.\n\nThe paper is well written and easy to read. The state of art is adequate and gives enough background on the domain\n\n-------------------------------------------------\nPoints to improve\n\nThe principal flaw of the work, and main reason why I do not recommend its acceptance in the proceddings is its weak experimental setup and overall evaluation. Other, but less important, reasons for my recommendation are the lack of comprehensive reproducibility details and the weak findings included in the conclusions.\n\n-------------------------------------------------\nRecommendations\n\nA single dance (one type of sequence), performed by a single couple (lack of variability in the data collected) and a predefined sensor system (lack of multimodal/multilocation setup) is not sufficient to achieve solid results for the type of algorithm the authors want to evaluate. I understand it’s difficult to collect high quality data, so I’d suggest the authors to add data from related domains where the same representation learning can be employed or augment its own data expanding the sensory system (which in principle should be cheaper).\n\nThe value of the paper could be not just to address a specific application (ballroom dance recognition) using well known feature representations and algorithmic approaches. Honestly a more interesting motivation for me would be to explore how the type of data representation employed may help overcome effects of limited information when exploring domains where only scarce data is available. Specifically in the domain of HAR, where quality datasets are very expensive to collect. For me it’d be very interesting to know which sensors, located in which part of the dances, and modelled under which representation can perform better. As I say an interesting approach could be to augment the sensory infrastructure to characterize the minimum amount of sensor data required based on different representations.\n\nFollowing my last comment, the Markov chain wrapping the figure classifier does not offer a significant contribution in my opinion. Its contribution is limited to just chain predictions from models which are translating sensor signals to human movements, that last part is where (in my opinion) the contribution is. The sequence modelling could be perfectly covered by the ANN using a different network topology.\n\nAlso, when the authors claim to perform sensor classification “in real time” I expect to see some experiments covering that claim. I’m mean, there are no experiments addressing either the timing performance of the models or their power requirements. I’d recommend including in the experimentation some test of the performance of the methods, not just their accuracy.\n\nI know it’s not possible with the current dataset, but when cross validating in HAR it’s more correct to do the splitting by user. Cross-validation based on the subjects offer a stronger vision of the generalization power of the method\n\nTo improve the reproducibility I’d recommend further info. The sensor signal was downsampled to 100 samples, but what was the original resolution? From the rotation sensor only yaw channel is used (because is “based on prior knowledge that roll and pitch are insignificant in the waltz figures included in the study), please improve this explanation. A figure explicitly addressing the input representation would be useful.\n\nThe legend of the figures should be enough to understand the figure as a whole. Please improve theIt’s honestly quite surprising that decision trees can work so well with raw features. Derived features (max, min, avg, kurtosis, skew ) tend to work better for these type of algorithms and are more commonly used.\n\n-------------------------------------------------\nMinor\n\nUsing the same terminology from sampling rate (in “2.1 Data Collection” relates to the frequency domain) and what later becomes the observation/instance (size 4x100) of a figure is a bit confusing in my opinion.\n\n The probability of observing the next figure is really independente of past figures sequence? Does not a dance require to include a number of figures during its execution? I’ve the feeling that the accumulated number of figures also work as a prior for the next figure.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3vOHlZO_UxK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents some classification results for ballroom dancing movements, as measured by inertial sensors on a smartwatch.  The motivation is mixed - as a guide to dancers themselves and as an automatic grading mechanism for competition judges. However the sensors used can only measure a very limited aspect of the dance, and there is no 'gold standard' data describing the full motion of the dancers that can then be compared with the limited data actually measured, to be able to contrast variations in the full body motion with variations in the intertial sensors on the smartwatch.  Describing the project as a study of whole body movement seems a bit brave given that you measure only hand movements. The restriction to the yaw axis will make results very sensitive to how the hand is held.\n\nThe paper is fairly clearly described, but the work uses fairly standard model structures and a very limited training set, so I'm not sure the reader really learns anything novel from the paper. In my opinion, it is below the level of technical challenge and novelty of solution usually associated with an ICLR paper\n\nIt is unclear whether cross-validation splitting was done randomly rather than breaking the time-series into a number of contiguous sections and working with these. If randomly from whole set, then I would worry about  effective independence. The training data seems to be very limited, with little variability in individuals, so it is hard to know whether the results are of any practical use.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper initially received three negative reviews: 4,4,4. The main concerns of the reviewers included limited methodological novelty and an oversimplistic experimental setup. The authors did not submit their responses.
As a result, the final recommendation is reject.
Decision: Reject