Using Wearable Sensors for Real-time Recognition Tasks
in Games of Martial Arts – An Initial Experiment
Ernst A. Heinz, Kai Kunze, Matthias Gruber, David Bannach, Paul Lukowicz
Institute for Computer Systems and Networks (CSN)
UMIT – University of Health Systems, Medical Informatics and Technology
Hall in Tyrol, =Austria={ernst.heinz, kai.kunze, matthias.gruber, david.bannach, paul.lukowicz} @ umit.at, URL = http://csn.umit.at/

Abstract— Beside their stunning graphics, modern entertainment systems feature ever-higher levels of immersive userinteraction. Today, this is mostly achieved by virtual (VR) andaugmented reality (AR) setups. On top of these, we envision
to add ambient intelligence and context awareness to gaming
applications in general and games of martial arts in particular.
To this end, we conducted an initial experiment with inexpensivebody-worn gyroscopes and acceleration sensors for the Chum
Kiu motion sequence in Wing Chun (a popular form of Kung
Fu). The resulting data confirm the feasibility of our vision.
Fine-tuned adaptations of various thresholding and patternmatching techniques known from the fields of computational
intelligence and signal processing should suffice to automate theanalysis and recognition of important Wing Chun movements
in real time. Moreover, the data also seem to allow for the
possibility of automatically distinguishing between certain levelsof expertise and quality in executing the movements.

Keywords: Body-worn Sensors, Experiment, Games ofMartial Arts, Kung Fu, Motion Analysis, Movement Recognition, Wearable Computing, Wing Chun
I. I NTRODUCTION
Video analysis and motion capturing are standard tools in
professional sports to monitor and improve athletic performance by recognizing and fine-tuning the quality of movement. Cutting-edge systems with high-quality sensors hardly
suffice to fulfill these professionals’ needs. Quite often,
trainers and other experts still process the recorded data byhand. The whole setup and procedure are not only expensive
and time-consuming but also error-prone in the sense that theeffectiveness of the analysis depends on the humans doing it.
Hence, the large-scale use of similar analyses for the hobbyist
and gaming masses requires a completely different approach.
In particular, as we like to increase the immersiveness of theuser experience and interaction in video games of martial
arts where people’s real-world physical actions directly needto directly affect their playing reality.
We envision to employ inexpensive wearable sensors toachieve this – preferably tiny gyroscopes and accelerometers
worn by people on their bodies integrated into their clothes
and other personal accessories (e.g., watches or jewelery).
Such body-mounted sensors provide an inexpensive alternative for motion analysis while letting users move and roamabout freely, independent of any additional infrastructure. As
decribed in Section III, we conducted an initial experimentcapturing Wing Chun movements with wearable gyroscopes

and acceleration sensors to test the feasibility of our vision.
Section IV discusses and analyzes the experimental results,
proving to be very promising indeed. Based thereon, it
certainly seems worthwhile to continue in this direction andtry to automate at least some parts (if not all) of an expert
analysis for many important Wing Chun movements.
Beside in sports and game play, martial arts from theFar East gain ever more popularity and importance in manyother areas as well. Tai Chi, for instance, is of special
interest because clinical studies show that it helps to reduce
the probability of falling, especially for the elderly [1] andpatients with chronic conditions [2].
II. R ELATED W ORK
By now, many independent researchers have demonstrated
the suitability and excellent further potential of body-worn
sensors for automatic context and activity recognition, e.g.,[3], [4], [5], [6], [7], [8], [9], [11]. The available scientific
literature reports about successful applications of such sensors to various types of activities, ranging from the analysis
of simple modes of locomotion [5] to more complex tasksof everyday life [4] and even workshop assembly [10].
There are much fewer publications, however, about using
wearable sensors in martial arts. In [12], wearable pressure
sensors integrated into body protectors help to control and decide the counting of points for Taekwando. Supposedly helping children with their Kung Fu education, [13] introducessome kind of interactive computerized toy ball. Focusing on
Kung Fu, [14] presents a video capturing system for artificial
and augmented reality games of martial arts. The workemphasizes the specific gaming aspects of the application andsuffers from the usual drawbacks of video-based approaches,i.e., high sensitivity for lighting conditions and demanding
requirements on equipment and infrastructure. Other videobased capture and processing systems for augmented virtual
reality gaming and training are presented in [15] and [16].
The latter introduces a wireless virtual reality system andsome prototype Tai Chi training application on top of it.
Yet, the system features only limited usability and very
restricted degrees of freedom for the user. Moreover, it does
not evaluate full motion but just stances instead.

III. E XPERIMENTAL S ETUP•

Our initial Wing Chun experiment featured the Chum
Kiu motion sequence as test action (see Section III-C) andtwo different persons as test subjects (see Section III-D).
For every subject, we recorded and video-taped five distinct
performances of the motion sequence overall. The sensor data
stem from eight wearable boxes (see Section III-A) affixed
to the test subjects’ rear hip, neck, wrists, knees, and lower
legs directly above their feet (see Fig. 1 and Section III-B).

•

right and left wrist,
right and left lower leg (directly above feet),• right and left knee (directly above knee cap),• neck (on shoulder height), and• rear hip (on backbone origin).
All MT9 boxes were oriented with their cables pointing
skywards when the test subjects stood still and relaxed. Then,the x-axes of the acceleration sensors pointed down towards
the ground while their y- and z-axes pointed horizontally.
Hence, the x-axes of the accelerometers on the wrists always
pointed towards the hands and the x-axes of the accelerometers on the lower legs always pointed towards the feet.
We also affixed the XBus master unit at the hip, but on
the right side of it (see Fig. 1). To strap and keep everythingtight in place, we used flexible bands that work very well insuch settings according to our experience [7], [8], [10].
C. Chum Kiu Motion Sequence
The Chum Kiu motion sequence contains a multitudeof basic Wing Chun movements (forward and backward)
including side steps, full-body turns, arm and leg blows,
and more involved motion combinations. It is a standardized
form of motion training for Wing Chun. Literally, Chum Kiu
means ”casting / seeking a bridge” to the opponent. While
doing so, you should try to adhere to your so-called central
line of action. The full Chum Kiu motion sequence takes
about two minutes to perform. In our experiment, completion
times varied from one-and-a-half to almost three minutes.
D. Test Subjects

Fig. 1. Test subjects wearing the wired sensor boxes while performing(expert on top, amateur below)

Both test subjects participating in the experiment areco-authors of this text. The Wing Chun expert, Matthias
Gruber, is a long-time practitioner and enthusiast of the art.
Kai Kunze, on the other hand, is a Wing Chun amateurwith limited experience of roughly two-and-a-half years of
training overall spanning several years on and off. But you
must not mistake him for an absolute beginner, of course.
IV. E XPERIMENTAL R ESULTS

A. Hardware Details
We used the XBus Master System (XM-B) manufactured
by XSens (http://www.xsens.com/) as the global sensor control. We wired the XBus master unit by physical cableto eight boxed MT9 sensors. Each such MT9 box houses
a 3-axis accelerometer, a 3-axis gyroscope, a 2-axis magnetometer, and a temperature sensor. Up to now, however,
we did not use the final two thereof. To directly collect thecomplete sensor data on some permanent external storage, we
linked an ”oqo” mobile computer (http://oqo.com/) to
the XBus master unit via a wireless Bluetooth connection –thus streaming the captured data in real time.
B. Sensor Placement
We discussed suitable locations for placing the sensors
with the Wing Chun expert and finally decided on thefollowing setup with one MT9 box each at the:As described in Section IV-A, the raw z-axis signals of
the gyroscopes at the necks seem to suffice for counting andrecognizing at least some turns. For the rest of the analyses,
we applied 20 different features to the raw signal data,
including absolute value, frequency entropy, frequence range
power, median, mean, 75%-percentile, standard deviation,
variance, and others over the accelerometer and gyroscope
data for each axis and for the absolute sum using a 100sample sliding window.
A. Raw z-Axis Signals of Gyroscopes at Neck
Fig. 2 shows the expert (top) and amateur (below) perform
three special turns in one direction, followed by anotherthree in the opposite direction. The graphs on the rightvisualize the respective raw z-axis signals for one particularperformance of the turns by each test subject as recordedby the gyroscopes mounted on their necks. The difference
in appearance of these graphs is quite stunning. Whereas theexpert’s signal shows clear and regular peaks and troughsof compellingly steady height, the amateur’s signal almost
completely lacks these characteristics featuring fewer irregular peaks and troughs of rather diminishing height. Taking
the regularity of the expert’s peak signals over time, width,and height into account, the distinction between badly, fairly,
and really well executed turns should not be too hard. With
appropriate thresholding and sliding windows, the real-time
counting and recognition of such turns also proves feasible ingeneral. Fig. 3, for instance, presents two graphs showing the
raw z-axis signals for sequences of turns starting in different
directions (expert on top begins with a left turn, while theamateur below starts with a right turn). Here, the sign of thez-axis signal easily identifies the direction of the turn.
B. Freq. Range Power of Accelerometers at Lower Left Leg

Fig. 2. Sequence of six (2 x 3) special turns, graphs show raw z-axis signalsof gyroscopes at neck (expert on top, amateur below)

Fig. 4. Frequency range power (FRP) of absolute sum of signals from all
three axes over different sampling windows for accelerometers at lower left
leg (expert’s peaks much higher than the amateur’s ones)

Fig. 3. Raw z-axis signals of gyroscopes at neck when first turning left
(on top, expert) vs. first turning right (below, amateur)

An important characteristic for many blows and othermovements in martial arts is the speed, or more excatly,
the explosiveness of execution. A typical feature known tobear the potential of achieving good results in the context
of motion explosiveness runs by the name frequency range
power (FRP). It computes the power of the discrete Fast
Fourier Transform (FFT) components for a given frequency
band. Thus, the frequency range power may serve as somekind of measure for the explosiveness or impulsiveness of
movements and their specific execution.
Fig. 4 illustrates that FRP delivers as hoped for in Wing
Chun, too. The graph plots the FRP values of the absolutesignal sums from all three axes over different sampling
windows for the accelerometers worn by the test subjects attheir lower left legs. The expert’s peak FRP values are clearlymuch larger than the amateur’s according ones for most
sampling windows. Here, again, appropriate thresholding andsliding windows should suffice to recognize different kindsof movements and qualities of motion execution.

C. Frequency Range Power of Remaining Accelerometers
The FRP graphs of the remaining acceleration sensors
look very similar to that of the accelerometer at the lower
left leg (see Fig. 4). In order to avoid almost identicalrepetitions, we refrain from including the other FRP plots
with this text. Please note, however, that the explosivenessof movements and their execution is equally visible for all
the other accelerometers as well. Direct comparisons of the
FRP plots for corresponding left / right accelerometers fromthe same performer may actually hint at some preferred or
better trained body half for the particular person. This inturn provides valuable information for future exercise andpossible improvements.
D. Other Promising but More Complex Features

Fig. 7. Median vs. absolute signal sum from x-axis of accelerometers at
lower left leg (amateur’s values cluster on diagonal bar, experts’s values
show divergence instead)Fig. 5. 75% percentile vs. full FRP of absolute signal from y-axis ofaccelerometers at neck (expert’s values cluster on lower left, amateur’svalues cluster on upper middle)

The frequency entropy (FRE) is defined as Hf req =P− p(Xi ) ∗ log2 (p(Xi )) where Xi are the frequency components of the windowed time-domain signal for a given
frequency band and p(Xi ) the probability of Xi . The frequency entropy is the normalized information entropy of thediscrete FFT component magnitudes for the windowed timedomain signal. Thus, it is a measure of the distribution of
the frequency components in the given frequency bands.
The three combinations of complex features visualized
in Figs. 5 to 7 all exhibit promising cluster structures. We
verified the promise by quick, successful classification trials
with three powerful machine-learning algorithms, namely
C4.5, KNN, and naive Bayes.
V. C ONCLUSION AND F UTURE W ORK

Fig. 6.
Median vs. frequency entropy (FRE) of absolute signal sums
from all three axes of accelerometers at hip (expert’s values cluster arounddiagonal bar, amateur’s values show high divergence instead)

The initial experimental results discussed in Section IV
look extremely promising. Our manual analyses of thecaptured data identify several different features and waysof calculation to apply to the raw sensor signals in order to automate the recognition of important Wing Chunmovements. Of course, we still need to adapt and finetune the respective general thresholding and pattern-matching
techniques to achieve acceptable real-time performance.
This requires us to feed much more input data to machinelearning algorithms among other things. Hence, we continue
to conduct further experimental sessions with additional
Wing Chun motion sequences (e.g., Biu Tse and Siu NimTau) and other test subjects of different skill levels (e.g.,intermediary). At the same time as aiming towards a statistically representative data set and model inferred fromit, we also hope to be able to identify even better novel
features for the recognition tasks at hand. The automatic
distinction between certain levels of expertise and quality
in executing the Wing Chun movements readily deserves
our special interest in this respect. For, once possible, itopens up a whole new realm of applications based on fully
automated and computerized, interactive Wing Chun training
functionality. Immersive games, for instance, could exploit
the functionality for special training modes and for adapting
the skill levels of computer-guided opponents to match thehuman players.

Fig. 8.
Software toolbox to ease the usage of wearable sensors forcomplex context recognition tasks on heterogenous systems (including
mobile platforms)

Last but not least, our envisioned real-time motion analysis
and movement recognition for Wing Chun needs to be implemented and integrated into suitable gaming applications.
To attract interest therein and decrease the required effort toreally do so, we developed a freely available software toolbox easing the usage of wearable sensors on heterogenous
systems [17]. We invite everybody to try and download thecurrent version from our server at http://csn.umit.
at/download/toolbox/. Just to wet your appetite abit, we like to briefly summarize the main advantages andfeatures of our sensor toolbox (see http://csn.umit.
at/research/toolbox/ for more details). The toolbox
is GUI-based (see Fig. 8) enabling users to quickly build
distributed, multi-modal context recognition systems by simply plugging together reusable, parameterizable components.
Thus, the toolbox simplifies the steps from prototypes to
final implementations that might have to fulfill real-time constraints on low-power mobile devices. Moreover, it facilitates
portability between platforms and fosters easy adaptationand extensibility. The toolbox also provides a set of readyto-use parameterizable algorithms including different filters,feature computations and classifiers, a runtime environmentthat supports complex synchronous and asynchronous dataflows, encapsulation of hardware-specific aspects including
sensors and data types (e.g., int vs. float), and the ability
to outsource parts of the computation to remote devices.
R EFERENCES[1] S. Wolf, R. Sattin, and M. Kutner, “Intense T’ai Chi exercise training
and fall occurrences in older, transitionally frail adults: a randomized,
controlled trial,” Journal of the American Geriatric Society, Vol. 1,
2003, pp. 188–189.
[2] C. Wang C., J. Collet, and J. Lau, “The effect of Tai Chi on healthoutcomes in patients with chronic conditions: a systematic review,”
Arch. Intern. Med., Vol. 1, 2004, pp. 188–189.
[3] O. Cakmakci, J. Coutaz, K. V. Laerhoven, and H.-W. Gellersen,“Context awareness in systems with limited resources.” [Online].
Available: citeseer.nj.nec.com/cakmakci02context.html[4] L. Bao and S. Intille, “Activity recognition from user-annotatedacceleration data,” in Pervasive Computing, F. Mattern, Ed., 2004.

[5] L. Seon-Woo and K. Mase, “Recognition of walking behaviors forpedestrian navigation,” in Proceedings of the 2001 IEEE International
Conference on Control Applications (CCA’01) (Cat, 2001, pp. 1152–
1155.
[6] J. Mantyjarvi, J. Himberg, and T. Seppanen, “Recognizing humanmotion with multiple acceleration sensors,” in 2001 IEEE International
Conference on Systems, Man and Cybernetics, vol. 3494, 2001, pp.
747–752.
[7] E. A. Heinz, K. S. Kunze, S. Sulistyo, H. Junker, P. Lukowicz, and
G. Tröster, “Experimental evaluation of variations in primary features
used for accelerometric context recognition,” in Proceedings of the 1st
European Symposium on Ambient Intelligence (EUSAI 2003), E. Aarts,
R. Collier, E. van Loenen, B. de Ruyter (eds.), LNCS 2875, SpringerVerlag, 2003, pp. 252–263, ISBN 3-540-20418-0.
[8] N. Kern, B. Schiele, H. Junker, P. Lukowicz, and G. Tr̈oster, “Wearable sensing to annotate meeting recordings,” in Proceedings Sixth
International Symposium on Wearable Computers ISWC 2002, 2002.
[9] N. Kern, B. Schiele, and A. Schmidt, “Multi-sensor activity context
detection for wearable computing,”in Proceedings of the 1st European
Symposium on Ambient Intelligence (EUSAI 2003), E. Aarts, R. Collier, E. van Loenen, B. de Ruyter (eds.), LNCS 2875, Springer-Verlag,
2003, ISBN 3-540-20418-0.
[10] P. Lukowicz, J. Ward, H. Junker, M. Staeger, G. Tröster, A. Atrash,
and T. Starner, “Recognizing workshop activity using body worn
microphones and accelerometers,” in Pervasive Computing, 2004.
[11] P. H. Veltink, H. B. J. Bussmann, W. de Vries, W. L. J. Martens,
and R. C. van Lummel, “Detection of static and dynamic activities
using uniaxial accelerometers,” IEEE Transactions on Rehabilitation
Engineering. Vol. 4, No. 4, pp. 375–385, 1996.
[12] E. H. Chi, J. Song, and G. Corbin, “”killer app” of wearable
computing: wireless force sensing body protectors for martial
arts,” in UIST ’04: Proceedings of the 17th annual ACM
symposium on User interface software and technology. New York,NY, USA: ACM Press, 2004, pp. 277–285. [Online]. Available:
http://dx.doi.org/10.1145/1029632.1029680[13] H. Markus, H. Takafumi, N. Sarah, and T. Sakol, “Chi-ball, aninteractive device assisting martial arts education for children,” in
CHI ’03: CHI ’03 extended abstracts on Human factors in computing
systems. New York, NY, USA: ACM Press, 2003, pp. 962–963.
[Online]. Available: http://dx.doi.org/10.1145/765891.766095[14] P. Haemaelaeinen, T. Ilmonen, J. HoeysniemI, M. Lindholm,and A. Nykaenen, “Martial arts in artificial reality,” in CHI ’05:
Proceedings of the SIGCHI conference on Human factors in computing
systems. New York, NY, USA: ACM Press, 2005, pp. 781–790.
[Online]. Available: http://dx.doi.org/10.1145/1054972.1055081[15] T. Starner, B. Leibe, B. Singletary, and J. Pair, “Mind-warping:towards creating a compelling collaborative augmented reality game,”in IUI ’00: Proceedings of the 5th international conference onIntelligent user interfaces. ACM Press, 2000, pp. 256–259. [Online].
Available: http://portal.acm.org/citation.cfm?id=325864[16] Chua, Training for physical tasks in virtual environments: Tai Chi,
2003. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs all.jsp?
arnumber=1191125[17] D. Bannach, K. Kunze, P. Lukowicz, O. Amft, “Distributed modular
toolbox for multi-modal context recognition,” in Proceedings of the
19th International Conference on the Architecture of Computing
Systems (ARCS 2006), LNCS, Springer-Verlag, to be published 2006.