Tac-Valuer: Knowledge-based Stroke Evaluation in Table Tennis
Jiachen Wang∗

Dazhen Deng∗

Xiao Xie

Zhejiang Lab and State Key Lab of
CAD&CG, Zhejiang University
wangjiachen@zju.edu.cn

State Key Lab of CAD&CG, Zhejiang
University
dengdazhen@zju.edu.cn

State Key Lab of CAD&CG, Zhejiang
University
xxie@zju.edu.cn

Xinhuan Shu

Yu-Xuan Huang, Le-Wen Cai

Hui Zhang

Department of Computer Science andEngineering, The Hong Kong
University of Science and Technology
xinhuan.shu@connect.ust.hk

National Key Laboratory for Novel
Software Technology, Nanjing
University{huangyx,cailw}@lamda.nju.edu.cn

Department of Sports Science,
Zhejiang University
zhang_hui@zju.edu.cn

Min-Ling Zhang

Zhi-Hua Zhou

Yingcai Wu†

School of Computer Science andEngineering, Southeast University
zhangml@seu.edu.cn

National Key Laboratory for Novel
Software Technology, Nanjing
University
zhouzh@lamda.nju.edu.cn

Zhejiang Lab and State Key Lab of
CAD&CG, Zhejiang University
ycwu@zju.edu.cn

ABSTRACT
Stroke evaluation is critical for coaches to evaluate players’ performance in table tennis matches. However, current methods highly demand proficient knowledge in table tennis and are time-consuming.
We collaborate with the Chinese national table tennis team andpropose Tac-Valuer, an automatic stroke evaluation framework foranalysts in table tennis teams. In particular, to integrate analysts’
knowledge into the machine learning model, we employ the latest
effective framework named abductive learning, showing promising
performance. Based on abductive learning, Tac-Valuer combines thestate-of-the-art computer vision algorithms to extract and embedstroke features for evaluation. We evaluate the design choices of
the approach and present Tac-Valuer’s usability through use casesthat analyze the performance of the top table tennis players inworld-class events.

CCS CONCEPTS• Computing methodologies → Knowledge representation
and reasoning; Activity recognition and understanding; • Humancentered computing → Visual analytics.

KEYWORDS
Table tennis, stroke evaluation, abductive learning∗ Both authors contributed equally to this research.
† Yingcai Wu is the corresponding author.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
KDD ’21, August 14–18, 2021, Virtual Event, Singapore.
© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8332-5/21/08. . . $15.00
https://doi.org/10.1145/XXXXXX.XXXXXXACM Reference Format:
Jiachen Wang, Dazhen Deng, Xiao Xie, Xinhuan Shu, Yu-Xuan Huang, LeWen Cai, Hui Zhang, Min-Ling Zhang, Zhi-Hua Zhou, and Yingcai Wu.
2021. Tac-Valuer: Knowledge-based Stroke Evaluation in Table Tennis. In
Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery andData Mining (KDD ’21), August 14–18, 2021, Virtual Event, Singapore. ACM,
New York, NY, USA, 10 pages. https://doi.org/10.1145/XXXXXX.XXXXXX

1INTRODUCTION

Table tennis is a highly confrontational racket sport, where players
from both sides strike the ball (the action that a player strikes theball once is a stroke) by turn until one wins this rally. Players’
techniques (patterns of single stroke) and tactics (patterns amongconsecutive strokes) are always the focal points of analysis [10].
Thus, many national table tennis teams adopt data analysis methods
to facilitate technical and tactical analysis. In recent four years,
we have worked closely with the Chinese national table tennis
team, one of the top table tennis teams worldwide, and developed adata platform. It provides pre-match and post-match data support
services for the team, including interactive tools for data collection[9], correlation exploration [37], and match simulation [30]. Till
now, the platform has provided data support for the national teamin more than 20 world-class events, e.g., World Cup, World Tour,and World Championships. However, the platform still falls shortof the strong demand from coaches for the stroke evaluation. In
table tennis, stroke evaluation is crucial for evaluating players’
performance since the stroke is the basic unit for analysis [19]. For
example, analysts evaluate a player’s performance of a particulartechnique (e.g., topspin) by investigating whether this player can
win an advantage after performing the stroke with this technique.
Methods for stroke evaluation can be divided into video-driven
ones and data-driven ones. Initially, video-driven methods are
straightforward and widely adopted by analysts. Analysts organize stroke videos in various ways and repeatedly examine thevideos to evaluate the strokes’ performances based on their knowledge [10]. Data-driven methods require analysts to collect the key

stroke attributes from videos in advance. Then analysts use effective statistical indicators [35, 36] and simulation models [19, 30, 34]
to evaluate strokes’ performance. While these methods have helped
discover in-depth insights, they are tedious and time-consuming
because they rely on analysts’ knowledge for video checking anddata collection.
A possible solution is to train a machine learning model to extract
stroke features and evaluate strokes based on the features. Training
such a model requires a large number of labeled data. However,labeled data is insufficient since qualified analysts for data labelingare limited, and the labeling process is time-consuming. Instead, we
have considerable unlabeled data and collection of analysts’ domain
knowledge. Therefore, it is highly desirable to leverage analysts’
knowledge for automatic feature extraction and stroke evaluation
based on limited labeled data and plentiful unlabeled data.
The challenge of proposing such a model lies in combining theanalysts’ knowledge and the machine learning model. It is difficult
to inject symbolic knowledge into the optimization of numericalvalues in machine learning models. To solve this challenge, we referto the latest effective AI framework, ABductive Learning (ABL)[5, 38]. Unlike other frameworks (e.g., probabilistic logic program[6], statistical relational learning [15]) that make one side subsume
to the other, ABL integrates machine learning and logical reasoning(driven by domain knowledge) in a mutually beneficial manner[38]. Furthermore, we propose a knowledge-based framework, TacValuer based on ABL for stroke evaluation.
Tac-Valuer takes stroke videos as input, embed stroke features,and output the evaluation scores of the strokes. It consists of threesteps, namely, video formalizing (VF), stroke embedding (SE), andperformance rating (PR). Among the components, SE uses ABL
to extracts and embeds the stroke features from videos into feature vectors based on analysts’ knowledge. It contains an attributerecognition component (driven by neural networks) and a logicalreasoning component (driven by rules of inference). The interaction
between the two components can help SE achieve decent embedding effects with limited labeled data. The contributions of thiswork are as follows.
• We are the first to address the problem of automatic stroke
evaluation in table tennis, benefiting table tennis analysts.
• We propose a novel framework that leverages ABL to achieve
decent stroke evaluation results based on analysts’ knowledge and limited labeled data.
• We implement our framework and conduct use cases onmatches between top table tennis players. Through the cases,
we obtain valuable insights into players’ strokes.

2

RELATED WORKIn this section, we review the works related to performance evaluation in sports and action recognition with domain knowledge.

2.1

Performance evaluation in sports

In table tennis, researchers have introduced many effective stroke
evaluation approaches [10], which have been discussed in Section
1. Therefore, this section mainly discusses performance evaluationin other sports. In soccer, both Decroos et al. [7, 8] and Bransen
et al. [1] have proposed automatic methods for valuing players’

actions during matches. In basketball, Cervone et al. [2] propose aframework for valuing each moment of possession, and Sicilia etal. [25] construct a deep learning architecture to evaluate players’
actions within a possession. Additionally, many similar studies
have also been conducted in tennis [31–33] and ice hockey [17, 24].
These studies mainly evaluate players’ actions based on position
tracking data and game context data (e.g., events and box scores).
While these studies are effective, they cannot be applied to stroke
evaluation in table tennis for two reasons. First, the gaps between
different competition rules prevent most of the aforementioned
approaches, especially those for team games like ice hockey, soccer,
and basketball. This gap leads to different evaluation metrics. For
example, in soccer, when analysts evaluate a pass, they need tofocus on the whole possession containing this pass to examine thecollaboration among teammates and defense of the opponent team.
However, in table tennis, there is no collaboration but confrontation
in a stroke. Analysts evaluate a stroke by examining whether the
stroke makes the striker at an advantage or not. Second, existingstudies mainly focus on position tracking data and game context
data, not considering players’ motions. Therefore, a new method isrequired for table tennis to utilize motion data for stroke evaluation.

2.2

Action recognition with knowledge

Tac-Valuer extracts and embeds stroke features from videos by using
action recognition techniques. Video-based action recognition has
been a popular topic in computer vision [11, 20]. Considerable techniques have been developed to improve the efficiency and accuracyof the action recognition tasks [13, 26, 29]. To further enhance theperformance of the techniques, various datasets in different fields(e.g., film, sports, music) have been introduced [16, 23, 28]. However, these techniques do not contain enough human knowledge inthe machine learning models, which limits their performance andinterpretability. To solve this issue, many AI frameworks, such asprobabilistic logic program [6] and statistical relational learning[15], have provided solution alternatives. However, these frameworks make knowledge subsume to machine learning or vice versa.
Therefore, we decide to use ABL to improve the state-of-the-art
action recognition techniques with domain knowledge for data collection. According to existing works [5, 38], ABL can combine machine learning and logical reasoning with knowledge in a mutually
beneficial way. Moreover, it also performs well in semi-supervised
fashion, which further improves our proposal since high quality
labeled table tennis data is not sufficient for supervised learning.

3

BACKGROUND AND DATAIn this section, we introduce our collaboration with the Chinese
national table tennis team and the data we use in Tac-Valuer.

3.1

Collaboration with the national team

We have collaborated with the Chinese national table tennis teamfor four years. During the collaboration, we developed a data platform for the national team to empower the analysts in the team tocollect and analyze match data (Figure 1). The platform consists ofthree components, database, data collection, and data analysis. The
database enables the analysts to store data, such as broadcast videos,
player information, and match meta information for table tennis

Figure 1: The structure of the data platform. It consists ofthree components, namely, database, data collection, anddata analysis. The arrows present the data transmission in
the platform.

did not require proficient knowledge. We collected 9522 stroke
videos from 21 matches.
The stroke attributes are used as training labels of stroke videos
for the models in SE. In table tennis, players use rackets to strike
the ball by turn. The basic observation unit in table tennis is astroke, namely, the action that a player hits the ball once [19]. Since
the recognition of stroke attributes demanded knowledge in table
tennis, we asked analysts to collect this data. The data was collected
through the platform [9]. Analysts needed to watch the stroke
videos collected before and annotate the specific stroke attributes.
Figure 2 displays an example of the stroke attribute data. The first
three attributes are used to identify the temporal position of thestroke within a match. The subsequent three attributes (i.e., stroke
technique, stroke position, and stroke placement) characterize thetechnical features of a stroke. The last attribute is derived based on
the game rules. We collected the stroke attributes of all the 9522
stroke videos mentioned above.
The rules of inference are used as the logical reasoning component of ABL to improve the performance of the models in SE.
We summarized analysts’ knowledge during data collection andquantified their knowledge in the manner as follows.
𝑃𝑟𝑒_𝑇 𝑒𝑐ℎ(𝑋, "Block")

Figure 2: An example of a stroke’s attributes. These attributes are manually collected by analysts.

matches with national team players involved. The data collection
leverages an interactive data collection framework, EventAnchor[9], to reduce analysts’ efforts on data retrieval. The data analysis
supports various analysis techniques, such as visualization andsimulation. The visualization part provides a comprehensive visual analytics tool, iTTVis [37], to help analysts investigate thecollected data to gain new insights. The simulation part, Tac-Simur,
uses a hybrid second-order Markov chain model [30] to simulate
the matches against particular players to help coaches improve thetraining plans and playing strategies. Till now, the platform has
served the teams in more than 20 top events in the world, which
provides a good opportunity to design an efficient video-based
framework for automatic stroke evaluation.

3.2

Data description

Tac-Valuer requires four parts of data for model training, namely,
stroke videos, stroke attributes, rules of inference, and performance
scores. The stroke videos are the input of Tac-Valuer. Each stroke
video contains a sequence of eight frames, including a central frame
when the player’s racket touches the ball, four frames before thisframe, and three frames after this frame. We labeled the timestampof the central frame through crowdsourcing since labeling this data→

¬𝐶𝑢𝑟 _𝑇 𝑒𝑐ℎ(𝑋, "Push")∧¬𝐶𝑢𝑟 _𝑇 𝑒𝑐ℎ(𝑋, "Short")∧¬𝐶𝑢𝑟 _𝑇 𝑒𝑐ℎ(𝑋, "Slide")∧¬𝐶𝑢𝑟 _𝑇 𝑒𝑐ℎ(𝑋, "Block"),

where “¬” is negation, “∧” is conjunction, and “→” is implication.
This example rule means if the technique of the previous stroke is"Block", then the technique of the current stroke is impossible tobe “Push”, “Short”, “Slide”, and “Block”. We collected 28 rules.
The performance scores are used as training labels of stroke
videos for the model in PR. The performance scores were manually
rated by professional analysts. We hired seven analysts working forthe professional table tennis teams to evaluate the performance of
players’ strokes in the stroke videos. We let them rate the strokes
into two levels, namely, 1 for advantaged strokes and -1 for disadvantaged ones. Each stroke was rated by three different analysts.
Thus, the score of each stroke was decided according to majority voting. For the strokes rated differently by the three analysts,
we asked another senior analyst to re-evaluate these strokes. The
senior analyst has worked for the Chinese national table tennis
team for more than five years and has rich experience in stroke
evaluation. We collected the scores of all the 9522 stroke videos.

4

FRAMEWORKIn this section, we first define the problem of stroke evaluationin this work and present an overview of Tac-Valuer. Then, we
introduce the implementation details of Tac-Valuer.

4.1

Problem definition

Table tennis is a racket sport that requires players from both sides
to interact with each other’s stroke by turn. We denote a rally as
𝑅 = {𝑆 1, 𝑆 2, . . . , 𝑆𝑖 }, where 𝑆𝑖 represents the video frames of the𝑖 𝑡ℎ stroke in the rally 𝑅. Then, the problem of stroke evaluation inoccurrence of these persons can decline the performance ofthe models in SE and PR.
• C2: How to effectively extract and embed stroke features with limited labeled data? Labeled stroke attributedata is limited due to its high demand for proficient domain
knowledge. Training an effective model for extracting stroke
attributes with limited labeled data is challenging. Moreover, the stroke attributes have been well defined by existing
works in table tennis analysis as shown in Figure 2. However,stroke motions for stroke evaluation have not been extensively studied. Extracting stroke motions from the stroke
videos and embed them with stroke attributes for evaluationis also challenging.
• C3: How to quantify the performance of a stroke? Current studies in table tennis mostly use the scoring rate forquantified analysis. However, this method is not rigorous
since whether the player can score in this rally is related to
all the strokes in this rally. A more accurate way is to ask
analysts to evaluate the performance of a stroke according
to the given stroke video. However, their evaluation criteria
are based on their knowledge and hard to be verbalized.

Figure 3: A player performs the strokes with similar technical attributes in different motions. The player loses balancein A and keep balance in B, which results in different performance of these two strokes.

table tennis can be defined as follows.
𝑆𝑐𝑜𝑟𝑒𝑖 = 𝐹 (𝑆𝑖 )(1)where 𝐹 denotes the evaluation function and 𝑆𝑐𝑜𝑟𝑒𝑖 represents theperformance of the stroke. The last stroke of a rally is evaluated aspoor by default since it belongs to the loser of this rally in our data.
A straightforward alternative to 𝐹 is training an end-to-end
model for stroke evaluation. Specifically, the model takes the stroke
videos as input and directly outputs the evaluation scores. However,an end-to-end model lacks interpretability, which is not accepted byanalysts. To yield an interpretable evaluation result, we decided toimitate analysts’ evaluation process when constructing 𝐹 . Generally,
analysts often need to identify the stroke attributes. Besides, theyalso have to consider the stroke motions. Specifically, for example,
in Figure 3, the effects of the two strokes hit by topspin in forehand
are quite different according to the motion features of the twoplayers. However, from the perspective of attributes, these two
strokes are similar strokes since the technique and position ofboth strokes are identical (Figure 3). With these two kinds of data,
analysts can evaluate a stroke comprehensively. Therefore, thefunction 𝐹 needs first to extract stroke attributes and stroke motions
from videos, and then, evaluates the strokes based on them. The
stroke attributes are primarily depicted by stroke technique, stroke
placement, and stroke position according to existing studies [10].
The stroke motions are depicted by visual features of video frames.
Based on these considerations, we propose Tac-Valuer, a knowledgebased stroke evaluation framework.

4.2

Framework overview

Tac-Valuer takes the stroke videos within a rally as input, extracts
and embeds stroke features from videos, and output the evaluation scores of these strokes. To accurately extract and embed thestroke features with limited labeled data, we use ABL to optimize
pseudo-labels of unlabeled data to enhance model performance.
The challenges during constructing Tac-Valuer are as follows.
• C1: How to obtain effective stroke videos? The stroke
videos contain noise within each frame. The noise is often
brought by irrelevant persons on the court. In a table tennis
match video, apart from the players, the referees and otherstaff, such as caddies, can appear in the scene (Figure 5). The

Given the challenges above, we constructed Tac-Valuer into threesteps (Figure 4) as follows.
• Step 1: video formalizing (VF). We remove the irrelevant
persons in each video frame to address C1. We use an object
detection algorithm to detect the two players in each frame.
Then we add masks to other irrelevant persons to remove
their influence (Figure 4A).
• Step 2: stroke embedding (SE). We leverage ABL to extract and embed the stroke attributes with stroke motions
with limited labeled data (C2). As Figure 4B shows, we firstuse an attribute recognition component to embed strokes
into feature vectors. Then we introduce a logical reasoning
component constructed based on analysts’ knowledge toimprove pseudo-labels of unlabeled data. These labels arefurther used to improve the performance of the attributerecognition component iteratively.
• Step 3: performance rating (PR). We train a classifier thatlearns analysts’ evaluation in video-based methods to obtain
quantified evaluation results (C3). We classify analysts’ evaluation of strokes into two levels as training labels and train
the model to classify the level of each stroke based on theembedding vectors in Step 2 (Figure 4C).

4.3

Video formalizing

VF takes stroke videos as input and attaches a mask to remove
irrelevant persons for each frame (Figure 4A). We use Faster RCNN [22], to detect the bounding boxes of the table and all persons
in each frame. According to the relative position between the boxof the table and boxes of the persons, we identify the two players ineach frame. For the boxes of other persons, we use a mask to coverthem. We reserve both players for two reasons. First, analysts needto investigate both players’ motions. For example, when player
A is performing a stroke, the other player B will anticipate thestroke and move his/her body and racket in advance. If player A’s
motion misleads the anticipation of player B, this stroke is usually

Figure 4: The structure of the framework. The input is the video frames of strokes within a whole rally. The output is the scoreof each stroke. The orange, green, and purple boxes represent the three steps and the blue ones represent the data flowing
among different steps. Video formalizing (A) applies masks to frames to remove irrelevant persons. Stroke embedding (B)extracts and embeds stroke features from videos by using ABL. Performance rating (C) evaluates the performance of eachstroke based on the feature vectors. The contents under each blue box present the example of the data format.

Figure 5: The details of video formalizing. (A) is the original frame with referees and staff. (B) is the processed frame
where the irrelevant persons are removed by masks.
performed with high quality. Second, the motions of two players canprovide more features to improve the training accuracy of stroke
embedding in SE.

4.4

Stroke embedding

Given stroke videos, SE extracts the stroke features and embeds
them into feature vectors. This step is based on Abductive Learning(ABL) and contains two components, an attribute recognition component (Figure 6A) and a logical abduction component (Figure 6B).
Attribute Recognition Component. In the attribute recognition
component, the input data are stroke videos of a whole rally 𝑆𝑖 .
We first use VGG-13 [27] and LSTM [12] to extract feature vectors
𝐹𝑉𝑖 of each 𝑆𝑖 . Then the feature vectors 𝐹𝑉𝑖 are fed to dense layers
and softmax functions to get the result 𝐴𝑖 , which contains thethree technical attributes (i.e., stroke technique, stroke position,and stroke placement) (Figure 2). This component is pre-trained bylabeled stroke videos. Due to limited labeled data, it may produce
incorrect pseudo-labels 𝐴𝑖 for the unlabeled stroke videos.
Logical Abduction Component. The logical abduction component contains domain knowledge rules of table tennis. Given the

Figure 6: The structure of stroke embedding. (A) is an attribute recognition component and (B) is a logical reasoning component. The input of (A) is the unlabeled formalized
stroke videos (𝑆𝑖 ) within a whole rally. The recognized attributes (𝐴𝑖 ) are taken as pseudo-labels of the stroke videos
and input into (B). (B) revises the labels and transfers themback to (A) to improve the performance of (A) iteratively.
The feature vectors (𝐹𝑉𝑖 ) are the final input for performance
rating.

pseudo-labels (technical attributes) of the attribute recognition
component, which may be incorrect, it first checks whether thesepseudo-labels are consistent with the rules. If the pseudo-labels
are consistent with domain knowledge, it returns them without
any revision. Otherwise, it explores logical abduction to revise thepseudo-labels. The revision is conducted under the principle ofminimal inconsistency [5, 38] between the pseudo-labels and domain knowledge, aiming at correcting the most likely inaccurate
pseudo-labels. Specifically, logical abduction assumes that some

Table 1: The mAP of the experiments in SE

Algorithm 1 Abductive Learning in Tac-Valuer
Input: Labeled videos (𝑆𝑙 , 𝐴𝑙 ); Unlabeled videos 𝑆𝑢 ; Knowledge
base 𝐾𝐵
Parameter: Epoch 𝐸
Output: Model 𝑀
1: 𝑀 ← 𝑇 𝑟𝑎𝑖𝑛𝑀𝑜𝑑𝑒𝑙 (𝑆𝑙 , 𝐴𝑙 ) # Pre-train model
2: for 𝑒 B 1𝑡𝑜𝐸 dob𝑢 ← 𝑀 (𝑆𝑢 ) # Generate pseudo-labels 𝐴b𝑢3:𝐴b𝑢 ) # Revise pseudo-labels
4:
𝐴𝑢 ← 𝐴𝑏𝑑𝑢𝑐𝑒 (𝐾𝐵, 𝐴
5:
𝑀 ← 𝑇 𝑟𝑎𝑖𝑛(𝑀, 𝑆𝑙 , 𝐴𝑙 , 𝑆𝑢 , 𝐴𝑢 ) # Update 𝑀
6: end for7: return 𝑀

method

label

tec

pla

pos

strikersupervised
ABL

50%
50%

0.66
0.69

0.56
0.66

0.75
0.79

0.79
0.86

supervised

100%

0.71

0.64

0.8

0.91

Table 2: The result of the experiment in PR
FC

SVM

RF

DF21 XGBoost CatBoost LightGBM

Avg. 82.02 84.90 84.83 85.16
Std. 1.51 1.63 1.71 1.81
pseudo-labels are incorrect and makes them “unknown” (abducible),while other pseudo-labels are fixed. Then the abduction module
will abduce the most compatible labels and use them to replace theoriginal pseudo-labels so that the revised labels are consistent withdomain knowledge.
Abductive Learning (ABL). By ABL [5, 38], the logical abduction
component leverages the knowledge base (summarized rules ofinference) of table tennis and unlabeled data to improve the attributerecognition component’s performance. An outline of our learningalgorithm is shown in Algorithm 1. Given labeled stroke videos,unlabeled stroke videos and domain knowledge, it first uses labeledstroke videos to pre-train the deep learning models in the attributerecognition component. The pre-trained model generates pseudolabels of unlabeled stroke videos, which may contain mistakes. Thenthe logical abduction component tries to revise the pseudo-labels
based on domain knowledge rules. Finally, the revised pseudolabels will be treated as ground-truth labels, and transferred backto the attribute recognition component for training. The model’s
performance is improved by repeating the above circulation.

4.5

Performance ratingWith the feature vectors, PR uses DF211 (an efficient implementation of Deep Forest [39, 40]) as a classifier to rate the performanceof a stroke. After training, the classifier can predict the probabilitiesof the performance, (i.e., 𝑝𝑖,𝑎𝑑 for the probability of advantaged
performance and 𝑝𝑖,𝑑𝑖𝑠 for disadvantaged performance) of 𝑆𝑖 . In
this way, the final score, 𝑆𝑐𝑜𝑟𝑒𝑖 can be defined by the expectation
value of the level as follows.
𝑆𝑐𝑜𝑟𝑒𝑖 = 𝑝𝑖,𝑎𝑑 ∗ 1 + 𝑝𝑖,𝑑𝑖𝑠 ∗ (−1)5(2)

EVALUATIONIn this section, we evaluate our framework from two aspects. First,
we conduct two experiments to justify the design decisions of TacValuer. Then, we present two use cases based on the implementationof the framework. We conduct the cases with analysts in the national table tennis team and discover valuable insights.

5.1

Evaluation of design decisions

We evaluate the design choices on the critical aspect of VF and PR
by comparing the performance of our methods with alternatives.
1 https://www.lamda.nju.edu.cn/deep-forest/

85.11
2.21

84.97
1.75

84.41
1.75

5.1.1 Choice of ABL. In SE, we used ABL to construct a semisupervised model due to the limited labeled data. To validate theeffectiveness of ABL, we compared ABL with the other two supervised conditions in this experiment. First, we used 50% labeled datawith 50% unlabeled data to train the attribute recognition component with ABL. Then, we used 50% labeled data and 100% labeleddata to train the attribute recognition component without ABL
separately. The result of the comparison is displayed in Table 1.
We added one more attribute, “striker,” to evaluate the model performance because each stroke video contains two players and themodel needs to figure out which player is the striker. Among thethree conditions, ABL outperforms the other two conditions in recognizing stroke placement. For other attributes, ABL outperforms
the supervised one with only 50% labeled data and performs nearly
as well as the supervised one with 100% labeled data.
5.1.2 Choice of DF21. In PR, we needed a model that can accurately determine the performance level. Therefore, we tested popular classification models, including a three-layer fully connectedneural network (FC), SVM [18], Random Forest (RF) [18], DF21
[39, 40], XGBoost [3], CatBoost [21], and LightGBM [14]. We used
10-fold cross validation. The result is displayed in Table 2. Although
FC has the lowest standard variance, the average accuracy of DF21(85.16%) is the highest during cross validation. We finally chose
DF21 as the model in for performance rating.

5.2

Use Cases

In this section, we present three cases we conducted with two
professional analysts to demonstrate the effectiveness of Tac-Valuer.
Both of the two analysts have worked for the Chinese table tennis
team for more than five years. They did not participate in thecollection of performance scores in this work. We mainly analyzed
Ito Mima, a strong opponent of the Chinese table tennis players. In
the first case, we compared the evaluation of Tac-Valuer with thoseof traditional indicators and analysts to demonstrate the reliability
and advantage of Tac-Valuer. In the second case, we focused on
Ito’s performance when she played against different opponents.
5.2.1 Case 1: Ito’s push is not good enough. The match in this caseis the final of women singles in 2019 ITTF World Tour, Swedish Openbetween Ito and Chen Meng. Current methods (e.g., the three-phaseWe analyzed Ito’s techniques since she uses pimpled rubber,which other players rarely use. Figure 7A, B, and C display thescoring rates, evaluation of Tac-Valuer, and evaluation of analysts
of Ito’s techniques, respectively. Analysts’ evaluation was collectedby other analysts in advance. According to Figure 7B, C, the evaluation results of Tac-Valuer almost conformed to those of analysts.
However, the results of the direct scoring rate were quite different,
especially the result of “Push”. We examined the result of eachstroke performed by “Push” (Figure 7D). Most of these strokes’
scores are below 0. We examined the first stroke with the lowest
evaluation score (-0.42). This stroke is the third stroke of the rally.
Ito served in this rally, however, she did not take preemptive actions.
Instead, she used “Push” to try to control the ball. After her “Push”,
Chen seized the chance to attack first by using the offensive technique, “Topspin”. In such a condition, Ito totally lost the advantage
in this rally. Through this case, we concluded that Ito’s push is not
as good as the traditional indicators presented.

Figure 7: Case 1: (A), (B), (C) display the performance of Ito’s
technique evaluated by the direct scoring rates, Tac-Valuer,
and analysts, respectively. (D) shows the evaluation scoresof all strokes performed with push.

method [35, 36]) mostly use the direct scoring rates as an important
indicator for stroke evaluation. In a table tennis match, the directscoring rate of Player B’s A-type strokes is computed as follows:
𝑅 = 𝑁 𝑤𝑖𝑛 /(𝑁 𝑤𝑖𝑛 + 𝑁𝑙𝑜𝑠𝑒 )(3)

where 𝑁 𝑤𝑖𝑛 refers to the number of rallies where Player B performs
an A-type stroke, and his opponent fails to receive it, and 𝑁𝑙𝑜𝑠𝑒
refers to the number of rallies where Player B fails to receive his
opponent’s stroke using A-type stroke. In this way, it considers theA-type strokes that directly lead to the results of rallies. However,
the indicator omits many strokes that can potentially affect theresults. Tac-Valuer has solved this limitation.

5.2.2 Case 2: Ito should reduce errors in matches. The matches here
include the final, the semi-final, and the quarter-final of women
singles in 2019 ITTF World Tour, Swedish Open between Ito (purple)
and Chen Meng, Sun Yingsha, and Wang Manyu (orange). We firstexamined the stroke performance from the perspective of stroke
technique (Figure 8A). We found that when playing against Sun andWang, Ito’s technique performed better than her opponents. She
had fewer techniques whose scores were substantially below zero,
especially when she was against Wang. Her control techniques,“Push” and “Short” were much better than Wang’s. However, whenplaying against Chen, Ito did not display many advantages, which
may explain her losing the final. We further examined the stroke
performance of this match in Figure 8B.
We sum the evaluation scores of a player’s strokes within a rally
as the evaluation score of this player in the rally. Then, we visualize
the difference between the summed scores of two players within arally in a customized bar chart (Figure 8A). The height of the bars
presents the absolute values of difference. Thus, if the evaluation
score of Chen’s strokes is large than that of Ito’s strokes, the bar
is encoded by orange and towards up, and vice versa. The dots inthe middle of the chart indicate the winner of each rally. There areseven games in this match, and the numbers in orange and purple
indicate the scores of the two players in each game.
We found some interesting rallies where the player whose strokes
were better lost the rally. The most obvious one was the eighth rallyof the second game. We investigated the strokes within this rally(Figure 8C). According to the bar chart, Ito performed all strokes
well except the last one. This condition was rare and abnormal.
To verify it, we examined the stroke video and found that Ito did
perform the first three strokes well. After she served, she seized thechance to use topspin to attack first. However, her offense did notlast for many strokes since she made a mistake when performing
the last stroke, which was a pity for her.

6

DISCUSSION

Significance: We have solved a critical domain problem that heavily
relies on analysts’ labour and domain knowledge in an automaticmanner for efficient match analysis. The significance of Tac-Valuer
lies in four aspects. First, Tac-Valuer can benefit all analysts andFigure 8: (A) presents the performance of different players in three matches. (B) visualizes summary difference between theevaluation scores of two players within each rally. (C) is a rally where Ito’s performance is better than Chen’s but Ito lost.
table tennis teams. It can automatically evaluate strokes without
extra input or interactions. Second, Tac-Valuer can be implemented
with a small amount of labeled data by using abductive learning.
It combines machine learning models and analysts’ knowledge
to improve model performance with unlabeled data, alleviating
analysts’ burden on data annotation. Third, Tac-Valuer facilitatesknowledge mining. The insights from the cases not only verify some
empirical knowledge but also renew analysts’ findings discoveredby traditional methods. Fourth, Tac-Valuer is efficient enough forreal-time analysis. It finishes the stroke evaluation of a match inhalf an hour, making real-time stroke evaluation becomes practical.

is time-consuming and error-prone. A semi-supervised learning
framework can greatly alleviate the burden of data labeling.
Limitation: The lack of real-world high-quality videos is a major
limitation of this work. The effectiveness of our work could largely
depend on the quality of the training match videos. For instance,
the low resolution and slow frame rate of existing broadcast match
videos could affect the performance of stroke embedding. To address this issue, we have adopted data augmentation to extend ourtraining data in video resolution, brightness, contrast, and rotation.
In the future, we plan to extend our training dataset by incorporating more real-world high-quality videos from different sources.

Insights: We obtain two insights. First, the interpretability of amachine learning model is important for domain experts and often
the key for successful deployment. During our collaboration with
the Chinese national table tennis team, the coaches and players
were always concerned with the interpretability of our methods.
They would not accept the model’s results unless the results areinterpretable. Second, a semi-supervised learning framework canbenefit the domain. The application of machine learning models
to a domain problem often requires a large amount of labeled data.
The labeling process demands proficient domain knowledge andGeneralizability: Tac-Valuer is not limited to table tennis stroke
evaluation. It can easily be extended to other similar racket sports,such as tennis and badminton by following our data format toprepare the training data and rules of inference. Moreover, given
the use of ABL, analysts can achieve a decent result with limitedlabeled training data, which lowers the cost of extension.

7

CONCLUSION

In this work, we establish the problem of stroke evaluation in table
tennis. To solve this problem, we introduce a knowledge-based

framework, Tac-Valuer for automatic stroke evaluation. Tac-Valuer
is the first attempt to extract and embed stroke features from videos
based on analysts’ knowledge for comprehensive stroke evaluation.
Besides, Tac-Valuer enables decent model performance with limitedlabeled training data. We evaluate the usefulness and priority of
the framework by two use cases and discover valuable insights
approved by analysts in the Chinese national table tennis team.
In the future, we plan to improve the framework from two aspects. First, we will try to improve the robustness of the framework.
We plan to expand the types of training data so that we can cover
sufficient types of videos. Second, we plan to refer to the conceptof causality and use causal models to construct the relationships
among multiple strokes in the framework. Third, we plan to extend
our framework to other similar sports like tennis.

ACKNOWLEDGMENTS
The work was supported by the National Key R&D Program ofChina (2018YFB1004300), National Natural Science Foundation ofChina (62072400), and Zhejiang Provincial Natural Science Foundation (LR18F020001). We thank all participants and reviewers fortheir thoughtful feedback and comments.

REFERENCES[1] Lotte Bransen, Pieter Robberechts, Jan Van Haaren, and Jesse Davi. 2019. Chokeor Shine? Quantifying Soccer Players’ Abilities to Perform Under Mental Pressure.
In MIT Sloan Sports Analytics Conference.
[2] Dan Cervone, Alexander D’Amour, Luke Bornn, and Kirk Goldsberry. 2014.
POINTWISE: Predicting Points and Valuing Decisions in Real Time with NBA
Optical Tracking Data. In Proceedings of MIT Sloan Sports Analytics Conference.
[3] Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting
System. In Proceedings of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining. 785–794.
[4] François Chollet et al. 2015. Keras. https://keras.io.
[5] Wang-Zhou Dai, Qiuling Xu, Yang Yu, and Zhi-Hua Zhou. 2019. Bridging machinelearning and logical reasoning by abductive learning. In Advances in Neural
Information Processing Systems 32. 2811–2822.
[6] Luc De Raedt and Kristian Kersting. 2008. Probabilistic inductive logic programming. In Probabilistic Inductive Logic Programming. Springer, 1–27.
[7] Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. 2019. Actions
Speak Louder than Goals: Valuing Player Actions in Soccer. In Proceedings of the25th ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining. 1851–1861.
[8] Tom Decroos, Jan Van Haaren, Vladimir Dzyuba, and Jesse Davis. 2017. STARSS:
A Spatio-Temporal Action Rating System for Soccer. In In ECML/PKDD 2017
Workshop on Machine Learning and Data Mining for Sports Analytics.
[9] Dazhen Deng, Jiang Wu, Jiachen Wang, Yihong Wu, Xiao Xie, Zheng Zhou, Hui
Zhang, Xiaolong Zhang, and Yingcai Wu. 2021. EventAnchor: Reducing Human
Interactions in Event Annotation of Racket Sports Videos. In CHI Conference on
Human Factors in Computing Systems.
[10] Michael Fuchs, Ruizhi Liu, Ivan Malagoli Lanzoni, Goran Munivrana, Gunter
Straub, Sho Tamaki, Kazuto Yoshida, Hui Zhang, and Martin Lames. 2018. Table
Tennis Match Analysis: A Review. Journal of Sports Sciences 36, 23 (2018), 2653–
2662.
[11] Samitha Herath, Mehrtash Harandi, and Fatih Porikli. 2017. Going deeper into
action recognition: A survey. Image and vision computing 60 (2017), 4–21.
[12] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neuralcomputation 9, 8 (1997), 1735–1780.
[13] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2012. 3D convolutional neuralnetworks for human action recognition. IEEE transactions on pattern analysis
and machine intelligence 35, 1 (2012), 221–231.
[14] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma,
Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting
Decision Tree. In Advances in Neural Information Processing Systems 30. 3146–
3154.
[15] Daphne Koller, Nir Friedman, Sašo Džeroski, Charles Sutton, Andrew McCallum,
Avi Pfeffer, Pieter Abbeel, Ming-Fai Wong, Chris Meek, Jennifer Neville, et al.
2007. Introduction to statistical relational learning. The MIT Press.
[16] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, andThomas Serre. 2011. HMDB: a large video database for human motion recognition.

In International conference on computer vision. 2556–2563.
[17] Guiliang Liu and Oliver Schulte. 2018. Deep Reinforcement Learning in Ice
Hockey for Context-Aware Player Evaluation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 3442–3448.
[18] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel,
Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss,
Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journalof machine learning research 12, Oct (2011), 2825–2830.
[19] Mark Pfeiffer, Hui Zhang, and Andreas Hohmann. 2010. A Markov Chain Modelof Elite Table Tennis Competition. International Journal of Sports Science andCoaching 5, 2 (2010), 205–222.
[20] Ronald Poppe. 2010. A survey on vision-based human action recognition. Imageand vision computing 28, 6 (2010), 976–990.
[21] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2018. CatBoost: unbiased boosting with categoricalfeatures. In Advances in Neural Information Processing Systems 31. 6638–6648.
[22] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN:
Towards Real-Time Object Detection with Region Proposal Networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems,
Vol. 1. 91–99.
[23] Mikel D Rodriguez, Javed Ahmed, and Mubarak Shah. 2008. Action MACH aspatio-temporal Maximum Average Correlation Height filter for action recognition. In IEEE conference on computer vision and pattern recognition. 1–8.
[24] Kurt Routley and Oliver Schulte. 2015. A Markov Game model for valuing player
actions in ice hockey. In Proceedings of the Thirty-First Conference on Uncertainty
in Artificial Intelligence. 782–791.
[25] Anthony Sicilia, Konstantinos Pelechrinis, and Kirk Goldsberry. 2019. DeepHoops:
Evaluating Micro-Actions in Basketball Using Deep Feature Representations of
Spatio-Temporal Data. In Proceedings of the 25th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining. 2096–2104.
[26] Karen Simonyan and Andrew Zisserman. 2014. Two-Stream Convolutional
Networks for Action Recognition in Videos. In Proceedings of the 27th International
Conference on Neural Information Processing Systems, Vol. 1. 568–576.
[27] Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks
for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[28] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A
dataset of 101 human actions classes from videos in the wild. arXiv preprint
arXiv:1212.0402 (2012).
[29] Heng Wang and Cordelia Schmid. 2013. Action recognition with improved
trajectories. In International conference on computer vision. 3551–3558.
[30] Jiachen Wang, Kejian Zhao, Dazhen Deng, Anqi Cao, Xiao Xie, Zheng Zhou,
Hui Zhang, and Yingcai Wu. 2020. Tac-Simur: Tactic-based Simulative Visual
Analytics of Table Tennis. IEEE Transactions on Visualization and Computer
Graphics. 26, 1 (2020), 407–417.
[31] Xinyu Wei, Patrick Lucey, Stuart Morgan, Peter Carr, Machar Reid, and Sridha
Sridharan. 2015. Predicting Serves in Tennis Using Style Priors. In Proceedings ofthe 21th ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining. 2207–2215.
[32] Xinyu Wei, Patrick Lucey, Stuart Morgan, and Sridha Sridharan. 2013. Predicting
Shot Locations in Tennis Using Spatiotemporal Data. In Proceedings of 2013
International Conference on Digital Image Computing: Techniques and Applications.
1–8.
[33] Xinyu Wei, Patrick Lucey, Stuart Morgan, and Sridha Sridharan. 2013. “SweetSpot”: Using Spatiotemporal Data to Discover and Predict Shots in Tennis. In
Proceedings of MIT Sloan Sports Analytics Conference.
[34] S. Wenninger and Martin Lames. 2016. Performance Analysis in Table Tennis Stochastic Simulation by Numerical Derivation. International Journal of Computer
Science in Sport 15, 1 (2016), 22–36.
[35] Huanqun Wu. 1963. The statistical method of techniques and tactics in table
tennis. The Material of Sport Science and Technique, 21 (1963), 18–20.
[36] Huanqun Wu, Zhenbiao Li, Z. Tao, W. Ding, and J. Zhou. 1989. Methods of ActualStrength Evaluation and Technical Diagnosis in Table Tennis Match. Journal of
National Research Institute of Sports Science 1 (1989), 32–41.
[37] Yingcai Wu, Ji Lan, Xinhuan Shu, Chenyang Ji, Kejian Zhao, Jiachen Wang, and
Hui Zhang. 2018. iTTVis: Interactive Visualization of Table Tennis Data. IEEE
Transactions on Visualization and Computer Graphics 24, 1 (2018), 709–718.
[38] Zhi-Hua Zhou. 2019. Abductive learning: towards bridging machine learningand logical reasoning. Science China Information Sciences 62, 7 (2019), 76101:1–
76101:3.
[39] Zhi-Hua Zhou and Ji Feng. 2017. Deep Forest: Towards an alternative to deepneural networks. In Proceedings of the 26th International Joint Conference on
Artificial Intelligence. 3553–3559.
[40] Zhi-Hua Zhou and Ji Feng. 2019. Deep forest. National Science Review 6, 1 (2019),
74–86.

A

A.2

APPENDIX
Table 3: The architecture of SE network.
Layer
Sequential
Stacked Frames
VGG Block 1
MaxPooling 1
VGG Block 2
MaxPooling 2
VGG Block 3
MaxPooling 3
VGG Block 4
MaxPooling 4
VGG Block 5
MaxPooling 5
Flatten
LSTM
Pla. BranchTec. BranchPos. Branch
Stk. Branch

Parameters

Output Shape-(10,180,320,30)[(3, 3, 64)] × 2(2 × 2)[(3, 3, 128)] × 2(2 × 2)[(3, 3, 256)] × 3(2 × 2)[(3, 3, 512)] × 3(2 × 2)[(3, 3, 512)] × 3(2 × 2)(2 × 2)1024(10, 180, 320, 64)(10, 90, 160, 64)(10, 90, 160, 128)
(10, 90, 160, 64)(10, 45, 80, 256)
(10, 90, 160, 64)(10, 22, 40, 512)
(10, 90, 160, 64)(10, 11, 20, 512)
(10, 5, 10, 512)
(10, 25600)(10, 1024)(1024, 8)(1024, 7)(1024, 5)(1024, 2)(10, 8)(10, 7)(10, 5)(10, 2)*The abbreviations Tec., Pla., Pos., and Stk. here represent four
branches for classifying stroke attributes, namely, strike
techniques, ball positions, strike positions, and strikers.

A.1

Experimental setup and implementation

The experiments were conducted in Python.
• Random Forest. The random forest is the implementationfrom scikit-learn Python package 2 . We set the number of
estimators to be 100 and the maximum depth to be 50, with
criterion of Gini impurity.
• SVM. The SVM is the implementation from scikit-learn
Python package with default parameters 3 .
• DF21 The DF21 is the offical Python version 4 . We use thedefault parameters.
• Catboost. The CatBoost is the offical Python version 5 . We
set the iteration times to 1000.
• XGBoost. The XGBoost is the offical Python version 6 . We
use the default parameters.
• LightGBM. The LightGBM is the offical Python version 7 .
We use the default parameters.
• ConvNets & LSTM. The ConvNets and LSTM for stroke
embedding in SE (Table 3) were implemented with Keras[4].
The models were optimized with Adam, and the initial learning rate was 1e-4. If the loss stopped reducing for more than
10 epoch, the learning rate would be cut off 0.7. Each model
was trained for 40 epoches.
2 https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
3 https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
4 https://www.lamda.nju.edu.cn/deep-forest/
5 https://catboost.ai/
6 https://xgboost.readthedocs.io/en/latest/
7 https://lightgbm.readthedocs.io/en/latest/

Experimental environment

The models were trained on a server running Ubuntu 18.04 with
192GB of RAM and CPU Intel(R) Xeon(R) Gold 6126 CPU 2.60GHz,
providing a maximum of 12 cores and 24 threads. The computing
server is equipped with 4 Tesla V100, each with 32GB memory.