WISS 2022

Visualizing Discrepancy of Hand Postures for Performing Piano
Ruofan Liu∗ Erwin Wu∗† Chen-Chieh Liao∗ 西岡勇人 † 古屋晋一 † 小池英樹 ∗概要. Hand postures and ﬁnger movements are the most essential factor for learning many dexterous
skills, such as playing the piano. However, video teaching is limited by the ﬁxed camera angle, a 2D
video is diﬃcult to tell precise 3D hand posture intuitively. This demo presents a real-time visualization
system providing the intuitive discrepancy of hand postures between teacher and user. Through a motion
capture system, the estimated 3D hand postures are visualized in real time and discrepancies based on
distinct metrics are displayed, integrated with modular functions assisting skill acquisition.

1Introduction and Related Work

Most of the skill acquisitions are based on motion imitation hence correct poses are essential.
Especially in some dexterous skills such as playing the piano, understanding a precise ﬁnger posture is proved to be beneﬁcial to the learning eﬃciency [1]. However, current online-piano-learning
are conducted through 2D videos, where an intuitive comparison between ﬁnger posture is notpossible. Liu et al. [2] proposed a training system by visualizing diﬀerences between hands ina virtual world, but their system only consists ofrelatively simple functions. On the other hand,
Wang et al [4] introduced the PPVR system wherea student can follow an instructor intuitively invirtual reality. However, no further feedback areprovided for training.
Based on these previous works, we present areal-time piano self-training system that reconstructs 3D representations and provides visualfeedback on hand pose discrepancies. The 3D
hand postures are ﬁrst estimated from the videos
and visualized in real time. And a modular UIof diﬀerent visualizations is provided to the users
to observe their ﬁnger postures and understandthe diﬀerences. To validate the proposed approaches, a pilot study is conducted. The results
suggest a signiﬁcant improvement in using theproposed system over conventional video training.

Copyright is held by the author(s). This paper is nonrefereed and non-archival. Hence it may later appear
in any journals, conferences, symposia, etc.
∗Tokyo Institute of Technology†
Sony Computer Science Laboratories

2Implementation

2.1 Apparatus
We used a Logitec Brio web camera and an IRbased depth camera Ultraleap 3Di for video capturing. The visualization system is implemented
in Unity3D with a high-end notebook PC. For
the user interface, a touch display is used to show
visual feedback and to be operated by the user.
2.2 Hand Tracking
As mentioned before, a precise 3D ﬁnger position is required for intuitive comparison. We
utilize a computer vision-based hand pose estimation model, InterHand [3], which can precisely estimate 2D and 3D dual hand poses froma single webcam using deep neural networks. To
enhance the tracking precision of wrist position,
the depth camera is also aligned with the RGBcamera.
2.3 Visualization
Figure 1 shows the overview of the system 1.
General Functions: Aimed to evaluate thepreference for various visualization functions, weintegrate them into a modular interface. The
timeline and speed controller are similar to thatin the ordinary video-teaching method, moving
the timeline cursor to decide when to start replay
and the speed ranges from 0 to 2 times.
Discrepancy Metrics: To measure the differences between the recorded (student) and tutorial (teacher) data more comprehensively andholistically, at every frame, we have three discrepancy metrics, Corresponding Joint Interval,
Corresponding Finger Slope, and Adjacent BoneAngle, respectively. Besides, our system displays
the discrepancies between student and teacher’s

WISS 2022

① Before (video-teaching)③ After② Using Proposed System

Visualization

Motion Capture

図 1. System Overview

performance in two aspects, i.e. on particular
hands visualization and the error bars. In termsof hand visualization, it is clear that all the discrepant areas should be explicit, so the colorof diﬀerent parts (including joints and bones)
changes, ranging from green to red. On the other
hand, a series of error bars will be shown on thetimeline, where an error bar represents one error
moment. We use it to emphasize the mistakemade frames in the whole performance and indicate where and what speciﬁed kinds of errors
are located.
Visualization: Also, we design two diﬀerent
virtual keyboard emplacement, straight-forwardly
placing the student’s hand and teacher’s hand ina motion-overlay manner, and side-by-side placing the keyboards to check and compare the twoperformances individually.

3Pilot Study

We recruited ten novices (8 M, 2 F) for a
within-subjects user study. They are told tocomplete a task in two distinct conditions: abaseline conventional video learning and the proposed visualization system. After experiencing
each condition for 10 minutes, their performances
are recorded for a later quantitative calculation.
Finally, a SUS questionnaire is oﬀered and theparticipants provide their overall comments during the interview.
We calculate the disparity of Mean Per Joint
Position Error (MPJPE) between the moment
just completing warm-up task and the moment
after a teaching condition experience. The visualization system had statistically signiﬁcantly(t(18)=2.083, p=0.044) lower MPJPE (−0.69 ±0.48mm) at the end of every 10-minutes piano
performance practice compared to video-teaching

method (−0.34 ± 0.55mm). SUS was used toperceive usability, and the average score is 68.
The score calculated from our questionnaire was72.75, which was above average and acceptable.
From the perspective of the interview conducted,
the majority of users gave a high evaluation forerror bars providing the discrepancies between
their recorded data and tutorial. They note thatit can help them quickly locate the mistakes,
which reduces their load of recognition and observation to ﬁnd the errors so that they can directly focus on how to ﬁx the discrepancies andbeneﬁt a lot from it.

4Discussion and Conclusion

Our main ﬁnding was that this visualization
way results in more correct and eﬀective practicing of ﬁnger-reﬁned movements. As professional
pianists suggest, this is critical to piano performance, especially when considerable movement
control is required to accomplish high levels ofprecision. Almost none of these are currently
possible with only video learning methods. However, we found a limitation of our system that
reduces the user s rhythmic performance. We
assume that this is because users pay more attention to 3D posture instead of temporal information.
Our system implements a visualization system to help users discover hand pose discrepancies between diﬀerent plays. A pilot study
with novices suggests that our system outperforms conventional video-watching approaches.
The positive feedback from users has already suggested that the presented approach provides asuitable complementary option to current methods in piano performance learning.

Visualizing Discrepancy of Hand Postures for Performing Piano

参考文献[1] S. Furuya, M. Flanders, and J. F. Soechting.
Hand kinematics of piano playing. Journal ofneurophysiology, 106(6):2849–2864, 2011.
[2] R. Liu, E. Wu, C.-C. Liao, H. Nishioka, S. Furuya, and H. Koike. Synchronized Hand Diﬀerence Visualization for Piano Learning. In ACM
SIGGRAPH 2022 Posters, SIGGRAPH ’22, NewYork, NY, USA, 2022. Association for Computing

Future Vision
To better discover where the student’s
hands diﬀered from the teacher’s during thewhole performance, it’s indispensable to analyze the positions and angles of their ﬁngers at each unit moment. Since the capability between a novice and an expert isquite diﬀerent for various rhythmical skills
like playing the piano, it is technically notpossible to compare the corresponding handpose at every frame. In our future work,
we’ll align the moments when the two performers are supposed to show the sameMachinery.
[3] G. Moon, S.-I. Yu, H. Wen, T. Shiratori, andK. M. Lee. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation froma single rgb image. In European Conference on
Computer Vision, pp. 548–564. Springer, 2020.
[4] X. Wang. PPVR: Teaching and Learning to Playthe Piano in Virtual Reality*. UNIVERSITY OF
MICHIGAN, 2020.

hand posture. If so, the alignment-based
motion diﬀerence feedback method for providing the discrepancies for piano learning
will deﬁnitely helps a lot.
From the pilot study and result above,
the positive feedback from participants suggested our method as a kind of ”crossvalidation” in a real piano-practice situation. It plays an assistant role in learning
the piano. Also, we believe that in otherskills acquisition, the visualization of pose
discrepancy makes sense. Thus this system
can also be applied to other motor skills
where a correct posture is essential.