Ballroom Dance Movement Recognition Using a Smart Watch and Representation Learning

1 INTRODUCTION . Recent work has used low-cost smart watches to track the movement of human body parts . ArmTrak tracks arm movement , assuming that the body and torso are stationary ( Shen et al. , 2016 ) . In this paper , we perform whole body movement recognition using a single smart watch , which is a hard problem given that body movements need to be inferred using readings taken from a single location on the body ( the wrist ) . The movements in the study are from ballroom dancing , which engages tens of thousands of competitors in the U.S. and other countries . Competitors dance at different skill levels and each level is associated with an internationally recognized syllabus , set by the World Dance Sport Federation . The syllabus breaks each dance into smaller segments with well-defined body movements . Those segments are called figures . In the waltz , for example , each figure has a length of one measure of the waltz song being danced to ; the entire dance is a sequence of 40 to 60 figures ( depending on the length of the song ) . The sequence is random , but the figures themselves are well-defined . The sequence is illustrated in Fig . 1 . The International Standard ballroom dances are a subset of ballroom dances danced around the world , and they include the waltz , tango , foxtrot , quickstep and Viennese waltz . A unique characteristic of all these dances is that the couple is always in a closed-hold , meaning they never separate . Also , both dancers in the couple maintain a rigid frame , meaning the arms and torso move together as one unit . The head and the lower body , however , move independently of that arms-torso unit . Our hypothesis in this paper is that the figures in each of these dances can be recognized with high accuracy using deep learning representations of data obtained from a single smart watch worn by the lead in the couple . That is possible because the rigid frame makes it unnecessary to separately instrument the arms and torso , and because most figures are characterized by distinct movements ( translations and rotations in space ) of the arms and torso . We refer the interested reader to the website www.ballroomguide.com for free videos and details on the various syllabus figures in all the International Standard ballroom dance styles . In this paper , we validate our hypothesis on the quintessential ballroom dance– the waltz . We chose 16 waltz figures that are most commonly danced by amateurs . The full names of the figures are included in Appendix A . Our goal is to accurately classify those figures in real-time using data from a smart watch . That data can be pushed to mobile devices in the hands of spectators at ballroom competitions , providing them with real-time commentary on the moves that they will have just watched being performed . That is an augmented-reality platform serving laymen in the audience who want to become more engaged with the nuances of the dance that they are watching . The main beneficiary of the analysis of dance movements would be the dancers themselves . The analysis will help them identify whether or not they are dancing the figures correctly . If a figure is confused for a different figure , it may be because the dancers have not sufficiently emphasized the difference in their dancing and need to improve their technique on that figure . That confusion metrics could also be used by competition judges to mark competitors on how well dancers are performing figures ; that task is currently done by eye-balling multiple competitors on the floor , and is challenging when there are over ten couples to keep track of . We make three main contributions in detecting ballroom dance movements using learning representations . • First , we show that representations using data from a single smart watch are sufficient for discriminating between complex dancing movements . • Second , we identify and evaluate six learning representations that can be used for classifying the figures with varying accuracies . The representations are 1 ) Gaussian Hidden Markov Model , 2 ) Extra Trees Classifier , 3 ) Feed-Forward Neural Network , 4 ) Recurrent Neural Network ( LSTM ) , 5 ) Convolution Neural Network , and 6 ) a Convolution Neural Network that feeds into a Recurrent Neural Network . • Finally , we model the sequence of figures as a Markov chain , using the fact that the transitions between figures are memoryless . We use the rules of the waltz to determine which transitions are possible and which are not . With that transition knowledge , we correct the immediately previous figure ’ s estimate . This leads to an average estimation accuracy improvement of 5.33 percentage points . 2 DATASET DESCRIPTION . 2.1 DATA COLLECTION . The data was collected using an Android app on a Samsung Gear Live smart watch . The app was developed for this work on top of the ArmTrak data collection app . We were able to reliably collect two derived sensor measurements from the Android API : • Linear Acceleration . This contains accelerometer data in the X , Y and Z directions of the smart watch , with the effect of gravity removed . • Rotation Vector . This provides the Euler angles ( roll , pitch and yaw ) by fusing accelerometer , gyroscope and magnetometer readings in the global coordinate space . We use only the yaw ( rotation about the vertical axis ) in this study , and that is based on prior knowledge that roll and pitch are insignificant in the waltz figures included in the study . In total , we collected readings from 4 sensor axes ( three from the Linear Acceleration and the yaw from the Rotation Vector sensors ) . The readings were reported by watch operating system asynchronously , at irregular intervals , whenever a change was sensed . In order to facilitate signal processing , we downsampled the data such that each figure contained exactly 100 sensor samples , which was possible because the effective sampling rate was greater than that . The downsampling was done by taking the median ( instead of the mean , which is sensitive to outliers ) values of 100 evenly-spaced time windows . From this point on in the paper , when we refer to “ samples ” , we refer to an observation for a figure of dimension 4× 100 as one sample . The app was developed in such a way that the button on the watch that started recording the aforementioned sensor measurements also simultaneously started playing the music via Bluetooth speakers . That ensured that the music and the recording of the movements were time-synchronized . For all the data that was collected , we used the same rendition of the classic song ” Moon River ” . We performed manual segmentation of the song using its beats offline , and that was used to segment the time series data for the entire dance sequence into 2.1-second-long-segments corresponding to figures in the dance . We noted the song intro length ( where no dancing was performed ) and ignored all data in that period . For each figure , we extended the window of measurements equally at the beginning and at the end by 0.35 seconds to account for slight errors in dancer timing . That ensured that the window captures the figures even if the dancer was slightly early or late to begin/finish dancing the figure . The yaw data for 4 figure samples corresponding to two different figures are illustrated in Fig . 2 . It can be seen that right-turning figures tended to record yaw readings with an upward trend , while left-turning figures recorded yaw readings with a downward trend . Slight differences between the samples for each figure can be attributed to differences in the dancers ’ timing and execution . 2.2 CROSS-VALIDATION GROUPINGS . In total , we collected 818 figure samples across 16 different waltz figures , over 14 dances ( figure sequences ) . Thus , the input data had a dimension of < 818×4×100 for 818 figure samples , 4 sensors , and 100 measurements per sensor per figure sample . The small size of the dataset and subsequent difficulty in collecting additional data during the Covid19 pandemic made the learning problem more challenging . We had only 14 sequences in total , and that is too small of dataset to learn dependencies between figures . Therefore , our focus is on independently classifying the figures using the 818 samples , and leveraging the Markov property to enforce dependencies between figures . The 818 samples came from 14 separate dances ( figure sequences ) and we performed 7-fold crossvalidation with two dances per cross-validation group ( assigned randomly ) . That ensured sequences of figures ( dances ) were not split across different cross-validation groups . It also allowed us to test our representations ’ accuracy for each sequence as a whole . 2.3 LABELING GROUND TRUTH . Each dance was recorded on video so that labels ( ground truth ) could be given to the data segments corresponding to the figures . The labels are listed in Appendix A . 3 MARKOV TRANSITIONS . The sequence of figures in each dance can be modeled as a Markov chain . The probability of observing the next figure is dependent on the current figure , but independent of past figures given the current figure . The reason is as follows . Certain figures end on the right foot , while others end on the left foot . Similarly , certain figures begin on the left foot , while others begin on the right foot . The probability of going from a figure ending on the right foot to another figure beginning on the right foot is zero ( and the same applies to the left foot ) . That is because of the physics of the dance and the way weight is distributed between the feet . Similarly , some figures must be followed by figures that move forward while others must be followed by figures that go backward . Therefore , each figure constrains the immediate next figure , but the sequence is memoryless . Using the above rules , we constructed a transition matrix for all figures , and that is given in the Appendix in Table 3 . We essential gave a zero probability to impossible transitions , and equal probability to all possible transitions . Therefore , our transition matrix is completely unbiased , and not based on real training data . The advantage of the unbiased transition matrix is that the same matrix can be used across different couples since it is very general . It does not encode unique habits of certain couples , where there is a tendency to follow patterns . If a biased approach were taken , a unique transition matrix could be learned for each couple , but it would not generalize to other couples . 4 HIDDEN MARKOV MODEL REPRESENTATION . The dance can be represented as a Hidden Markov Model ( HMM ) where the states represent figures that emit sensor readings , as illustrated in Fig . 3 . Although the state space is discrete , the emission space is continuous because the sensor readings continuous . As a result , the HMM can not be solved using a discrete emission probability matrix . Instead , we assumed Gaussian emission probabilities , resulting in a Gaussian HMM . We used the HMMLearn Python library ( hmm ) to estimate the transition and emission probabilities while fitting the input data . We initialized the transition probabilities with the trained transition matrix described in Section 3 , and initialized state vector with the actual initial state obtained from the ground truth . The problem with the HMM approach for this task is that the HMM is a generative model , and not a discriminative model . At no stage does the model take the actual known labels to perform classification . It simple estimates states using the probability information and we assigned labels to the states by fitting the training set , and matching the states estimated by the HMM with the known labels . The approach achieved an accuracy of 35.93 % on the validation sets , averaged across the 7 cross validation groups .

Summary: Authors propose an approach to perform classification of ballroom dance movements (called figures) captured by the sensing mechanism of a smartwatch and discriminated via different ANN architectures. The sequence of figures are modelled as a Marlov chain, which work in  a generative+discriminative fashion to output the final prediction of the model. Authors also present a dataset collected specifically for this work, to perform the inference of the algorithms included in the evaluation. Results show a remarkable accuracy, but are not compared to any existing state of art due to limited related work.