Ballroom Dance Movement Recognition Using a Smart Watch and Representation Learning

1 INTRODUCTION . Recent work has used low-cost smart watches to track the movement of human body parts . ArmTrak tracks arm movement , assuming that the body and torso are stationary ( Shen et al. , 2016 ) . In this paper , we perform whole body movement recognition using a single smart watch , which is a hard problem given that body movements need to be inferred using readings taken from a single location on the body ( the wrist ) . The movements in the study are from ballroom dancing , which engages tens of thousands of competitors in the U.S. and other countries . Competitors dance at different skill levels and each level is associated with an internationally recognized syllabus , set by the World Dance Sport Federation . The syllabus breaks each dance into smaller segments with well-defined body movements . Those segments are called figures . In the waltz , for example , each figure has a length of one measure of the waltz song being danced to ; the entire dance is a sequence of 40 to 60 figures ( depending on the length of the song ) . The sequence is random , but the figures themselves are well-defined . The sequence is illustrated in Fig . 1 . The International Standard ballroom dances are a subset of ballroom dances danced around the world , and they include the waltz , tango , foxtrot , quickstep and Viennese waltz . A unique characteristic of all these dances is that the couple is always in a closed-hold , meaning they never separate . Also , both dancers in the couple maintain a rigid frame , meaning the arms and torso move together as one unit . The head and the lower body , however , move independently of that arms-torso unit . Our hypothesis in this paper is that the figures in each of these dances can be recognized with high accuracy using deep learning representations of data obtained from a single smart watch worn by the lead in the couple . That is possible because the rigid frame makes it unnecessary to separately instrument the arms and torso , and because most figures are characterized by distinct movements ( translations and rotations in space ) of the arms and torso . We refer the interested reader to the website www.ballroomguide.com for free videos and details on the various syllabus figures in all the International Standard ballroom dance styles . In this paper , we validate our hypothesis on the quintessential ballroom dance– the waltz . We chose 16 waltz figures that are most commonly danced by amateurs . The full names of the figures are included in Appendix A . Our goal is to accurately classify those figures in real-time using data from a smart watch . That data can be pushed to mobile devices in the hands of spectators at ballroom competitions , providing them with real-time commentary on the moves that they will have just watched being performed . That is an augmented-reality platform serving laymen in the audience who want to become more engaged with the nuances of the dance that they are watching . The main beneficiary of the analysis of dance movements would be the dancers themselves . The analysis will help them identify whether or not they are dancing the figures correctly . If a figure is confused for a different figure , it may be because the dancers have not sufficiently emphasized the difference in their dancing and need to improve their technique on that figure . That confusion metrics could also be used by competition judges to mark competitors on how well dancers are performing figures ; that task is currently done by eye-balling multiple competitors on the floor , and is challenging when there are over ten couples to keep track of . We make three main contributions in detecting ballroom dance movements using learning representations . • First , we show that representations using data from a single smart watch are sufficient for discriminating between complex dancing movements . • Second , we identify and evaluate six learning representations that can be used for classifying the figures with varying accuracies . The representations are 1 ) Gaussian Hidden Markov Model , 2 ) Extra Trees Classifier , 3 ) Feed-Forward Neural Network , 4 ) Recurrent Neural Network ( LSTM ) , 5 ) Convolution Neural Network , and 6 ) a Convolution Neural Network that feeds into a Recurrent Neural Network . • Finally , we model the sequence of figures as a Markov chain , using the fact that the transitions between figures are memoryless . We use the rules of the waltz to determine which transitions are possible and which are not . With that transition knowledge , we correct the immediately previous figure ’ s estimate . This leads to an average estimation accuracy improvement of 5.33 percentage points . 2 DATASET DESCRIPTION . 2.1 DATA COLLECTION . The data was collected using an Android app on a Samsung Gear Live smart watch . The app was developed for this work on top of the ArmTrak data collection app . We were able to reliably collect two derived sensor measurements from the Android API : • Linear Acceleration . This contains accelerometer data in the X , Y and Z directions of the smart watch , with the effect of gravity removed . • Rotation Vector . This provides the Euler angles ( roll , pitch and yaw ) by fusing accelerometer , gyroscope and magnetometer readings in the global coordinate space . We use only the yaw ( rotation about the vertical axis ) in this study , and that is based on prior knowledge that roll and pitch are insignificant in the waltz figures included in the study . In total , we collected readings from 4 sensor axes ( three from the Linear Acceleration and the yaw from the Rotation Vector sensors ) . The readings were reported by watch operating system asynchronously , at irregular intervals , whenever a change was sensed . In order to facilitate signal processing , we downsampled the data such that each figure contained exactly 100 sensor samples , which was possible because the effective sampling rate was greater than that . The downsampling was done by taking the median ( instead of the mean , which is sensitive to outliers ) values of 100 evenly-spaced time windows . From this point on in the paper , when we refer to “ samples ” , we refer to an observation for a figure of dimension 4× 100 as one sample . The app was developed in such a way that the button on the watch that started recording the aforementioned sensor measurements also simultaneously started playing the music via Bluetooth speakers . That ensured that the music and the recording of the movements were time-synchronized . For all the data that was collected , we used the same rendition of the classic song ” Moon River ” . We performed manual segmentation of the song using its beats offline , and that was used to segment the time series data for the entire dance sequence into 2.1-second-long-segments corresponding to figures in the dance . We noted the song intro length ( where no dancing was performed ) and ignored all data in that period . For each figure , we extended the window of measurements equally at the beginning and at the end by 0.35 seconds to account for slight errors in dancer timing . That ensured that the window captures the figures even if the dancer was slightly early or late to begin/finish dancing the figure . The yaw data for 4 figure samples corresponding to two different figures are illustrated in Fig . 2 . It can be seen that right-turning figures tended to record yaw readings with an upward trend , while left-turning figures recorded yaw readings with a downward trend . Slight differences between the samples for each figure can be attributed to differences in the dancers ’ timing and execution . 2.2 CROSS-VALIDATION GROUPINGS . In total , we collected 818 figure samples across 16 different waltz figures , over 14 dances ( figure sequences ) . Thus , the input data had a dimension of < 818×4×100 for 818 figure samples , 4 sensors , and 100 measurements per sensor per figure sample . The small size of the dataset and subsequent difficulty in collecting additional data during the Covid19 pandemic made the learning problem more challenging . We had only 14 sequences in total , and that is too small of dataset to learn dependencies between figures . Therefore , our focus is on independently classifying the figures using the 818 samples , and leveraging the Markov property to enforce dependencies between figures . The 818 samples came from 14 separate dances ( figure sequences ) and we performed 7-fold crossvalidation with two dances per cross-validation group ( assigned randomly ) . That ensured sequences of figures ( dances ) were not split across different cross-validation groups . It also allowed us to test our representations ’ accuracy for each sequence as a whole . 2.3 LABELING GROUND TRUTH . Each dance was recorded on video so that labels ( ground truth ) could be given to the data segments corresponding to the figures . The labels are listed in Appendix A . 3 MARKOV TRANSITIONS . The sequence of figures in each dance can be modeled as a Markov chain . The probability of observing the next figure is dependent on the current figure , but independent of past figures given the current figure . The reason is as follows . Certain figures end on the right foot , while others end on the left foot . Similarly , certain figures begin on the left foot , while others begin on the right foot . The probability of going from a figure ending on the right foot to another figure beginning on the right foot is zero ( and the same applies to the left foot ) . That is because of the physics of the dance and the way weight is distributed between the feet . Similarly , some figures must be followed by figures that move forward while others must be followed by figures that go backward . Therefore , each figure constrains the immediate next figure , but the sequence is memoryless . Using the above rules , we constructed a transition matrix for all figures , and that is given in the Appendix in Table 3 . We essential gave a zero probability to impossible transitions , and equal probability to all possible transitions . Therefore , our transition matrix is completely unbiased , and not based on real training data . The advantage of the unbiased transition matrix is that the same matrix can be used across different couples since it is very general . It does not encode unique habits of certain couples , where there is a tendency to follow patterns . If a biased approach were taken , a unique transition matrix could be learned for each couple , but it would not generalize to other couples . 4 HIDDEN MARKOV MODEL REPRESENTATION . The dance can be represented as a Hidden Markov Model ( HMM ) where the states represent figures that emit sensor readings , as illustrated in Fig . 3 . Although the state space is discrete , the emission space is continuous because the sensor readings continuous . As a result , the HMM can not be solved using a discrete emission probability matrix . Instead , we assumed Gaussian emission probabilities , resulting in a Gaussian HMM . We used the HMMLearn Python library ( hmm ) to estimate the transition and emission probabilities while fitting the input data . We initialized the transition probabilities with the trained transition matrix described in Section 3 , and initialized state vector with the actual initial state obtained from the ground truth . The problem with the HMM approach for this task is that the HMM is a generative model , and not a discriminative model . At no stage does the model take the actual known labels to perform classification . It simple estimates states using the probability information and we assigned labels to the states by fitting the training set , and matching the states estimated by the HMM with the known labels . The approach achieved an accuracy of 35.93 % on the validation sets , averaged across the 7 cross validation groups .

Summary: The paper presents some classification results for ballroom dancing movements, as measured by inertial sensors on a smartwatch.  The motivation is mixed - as a guide to dancers themselves and as an automatic grading mechanism for competition judges. However the sensors used can only measure a very limited aspect of the dance, and there is no 'gold standard' data describing the full motion of the dancers that can then be compared with the limited data actually measured, to be able to contrast variations in the full body motion with variations in the intertial sensors on the smartwatch.  Describing the project as a study of whole body movement seems a bit brave given that you measure only hand movements. The restriction to the yaw axis will make results very sensitive to how the hand is held.