Study of Action Spotting via Transformer with Pre-content and Post-content Encoders
Supervisor: Takayoshi Yamashita
TP19051 Yuzhi Shi
1.Introduction
・・・・・・・・・・・・・・・・・・・

FeatureExtractor

FeatureExtractor

Norm

FeatureExtractor

Positional
Encoding⨁
・・・・・・・・𝑡−𝑇
2𝑡 −1

𝑡𝑡+・

We propose a method, which split pre-content Tpre ={1, . . . , T2 } and post-content Tpost = { T2 + 1, . . . , T }frames to learn speciﬁc actions in the semantics of eachsubset inspired by NetVLAD++ [3]. Two encoders aredesigned based on Transformer [4], have diﬀerent weights.
We ﬁrst input temporal information by adding positional encoding to each feature vector extracted via feature extractor. The value is represented as pre-content
pt and post-content p′ t , where t = 1, . . . , T2 . The posi-⨁
Multi-HeadAttention

・・・・・

FeatureExtractor

Norm

Post-content Encoder

・・・

3.2 Network・

Pre-content Encoder𝑇2

Figure 1: Proposed Network Architecturetional encoding is represented the components are sinusoids of diﬀerent wavelength according to [4].
The two encoders learn latent features for the action
labels for each time step via a self-attention mechanism with L layers and N heads, respectively. The selfattention of pre-content encoder learn the query matrix
TT22
Qp = ϕpq ({pt }t=1), the key matrix K p = ϕpk ({pt }t=1)
T2
and the value matrix V p = ϕpv ({pt }t=1), and the selfattention of post-content encoder learn the query matrix
TT22
Qf = ϕfq ({p′ t }t=1), the key matrix K f = ϕfk ({p′ t }t=1)
T2
and the value matrix V f = ϕfv ({p′ t }t=1), where all ϕ·are MLP layers. It computes the attention w.r.t. precontent byheadpn=Attentionpn (Qp , K p , V p ),

(1)

P

=

ϕpo ([headpn ]Nn=1 ),(2)=Attentionfn (Qf , K f , V f ),(3)=

ϕfo ([headfn ]Nn=1 ),(4)and post-content byheadfnF

where ϕpo and ϕfo is an MLP layer, and the Attentionfunction the scaled dot-product attention in [4].
We merge the output of the two encoders, and average
the output temporally. We then use a sigmoid layer anda dropout layer σ to suppress overﬁtting. Finally using
an MLP ϕr layer to accurately classify action as:c

3.1 Video encoding
Our proposed architecture is shown in Figure 1. We
assume that there are speciﬁc actions (including background) in soccer video in time t = 1, 2, . . . , T . We
use the same feature extractor as SoccerNet-v2 [1]. Thefeatures ht ∈ R1×d , t = 1, 2, ..., T are extracted from thevideo at 2fps with a resolution of xt ∈ RH×W ×C , t =1, 2, ..., T by a feature extractor, where (H × W × C) isthe resolution of the original image and d is the featurevector dimension.

・・・・In this section, we describe the process of video encoding at ﬁrst. Next, we present the structure of ournetwork. At last,the details of inference are introduced.

⨁MLP・

3.Method

×𝐿・

Action spotting is the localization of an action anchored with a single timestamp. It’s a challenging task
because of rapid change of scenes in videos and the imbalance number of annotations for each action. Priorworks proposed some eﬃcient methods for action spotting. A regression and masking approach with RMS-Net
was introduced [5], they drop pre-content during training, expecting the model to focus on the post-contentframes. [6] introduces a context-aware loss function,deﬁnes a high-level semantic context from diﬀerent temporal regions far distant, just before and just after anaction occurs. NetVALD++ [3] uses a novel poolingmodule for action spotting that learns a temporallyaware vocabulary for past and future temporal context.
However, chunk size is important for localizing action
on soccer videos because diﬀerent actions has diﬀerent
temporal lengths of related video frames. Diﬀerent fromprevious works [3, 6] use ﬁxed chunk size for all actions,
we resample chunks of an appropriate size to take advantage of diﬀerent ranges of frames in videos per class.

Pre-/Post- content Encoder

・

2.Related work

ActionSigmoid + Dropout + MLP

・The production of sports broadcasts takes huge manpower and material resources. Automatic production
will save huge time for producers. Action spotting is asigniﬁcant task to understand high-level semantic information, supports automatic production. In this paper,
we focus on the action spotting task on videos, which
temporally localizes the speciﬁc actions.
The existing works generally use same temporal lengthof video frame features. The chunk size is deﬁned as
the temporal length of video clips. We propose a novel
model based on transformer encoder for action spotting.
In addition, we analyze the inﬂuence of chunk size foraction spotting per action and use an appropriate chunksize for each action to train our proposed model. The experiment results demonstrate that our proposed method
improves the Average-mAP for the action spotting task
and achieves state-of-the-art performance.

my=
=
=

[P ; F ],
T1 ∑

(5)

ct ,(6)ϕr (σ(m)),

(7)

Tt

where [;] is concatenation operator.

3.3 Inference
Considering that diﬀerent actions need diﬀerent sizesof chunks to localize the speciﬁc frame in a video morecorrectly, we resample features from diﬀerent sizes ofchunks to improve the performance. We set an appropriate chunk size for each class. As an example, we set
the chunk size of the goal class as 45 seconds(90 frames).
We deﬁne the t as the index of chunk’s starting frame insoccer videos. First, the features (90×512) of a chunk(from t to t + 90 frames) size of 90 frames is used toaverage resample to a chunk (20×512) size of 40 frames.
Secondly, we put the resampled features into the trained
network followed by a sigmoid layer. We use the prediction on the goal class as the result of classiﬁcation

Confidence

𝑡

28:54

29:07

29:19

29:32

29:44

𝑡

74:24

74:37

(a) Goal

74:49

74:55

30:48

31:00

𝑡

75:14

72:30

72:43

72:55

(b) Offside

31:13

31:25

73:08

73:20

(c) Substitution

𝑡

30:35

Frame of Ground Truth

𝑡

10:24

10:37

(d) Goal

10:49

11:02

𝑡

11:14

89:28

89:41

89:53

(e) Offside

90:06

90:18

(f) Substitution

Figure 2: Visual examples of conﬁdence.
on the goal class at t + 45 frame. Then we use anotherchunk of diﬀerent size to predict classiﬁcation results onother actions. Next, concatenate the prediction resultsof all actions as the classiﬁcation result at t + 45 frame.
We use the same way to predict the classiﬁcation resultat t + 46 frames. Finally, we obtain the prediction result of the entire game via sliding chunk frame by frameand predict classiﬁcation results for every chunk. We
use Non-Maximum Suppression (NMS) to improve prediction results. The NMS threshold is 0 and the NMS
window is 40 seconds.

Table 2: Relationship between action and chunksize.
Chunk Size

101520253045Penalty

49.89

69.17

71.6773.18

70.85

67.41

Kick-oﬀ

54.19

57.09

57.97

57.56

56.70

48.79

Goal

61.03

70.23

69.14

71.02

71.46

66.55

Substitution

65.03

67.66

67.06

64.93

62.39

50.19

Oﬀside

34.37

40.16

41.42

41.48

38.94

33.05

Shots on target

38.89

39.24

39.51

39.21

38.59

35.23

Shots oﬀ target

39.51

38.21

39.61

37.06

34.93

26.08

Clearance

53.59

54.84

54.34

50.96

48.08

38.98

Ball out of play

69.43

69.83

68.53

66.52

64.10

56.94

4.ExperimentThrow-in

65.59

66.21

64.52

62.05

59.37

49.69

In this section, we introduce the datasets and evaluation metrics used, compare our model with several priormethods and analyze experiment results.

Foul

62.15

62.71

61.86

60.32

58.99

49.64

Indirect free-kick

41.09

42.34

45.14

44.82

43.74

37.72

Direct free-kick

56.54

56.00

54.09

50.69

47.64

42.47

Corner

80.33

78.61

75.88

72.86

69.56

58.50

4.1 Dataset

Yellow card
Red card

51.74
6.66

52.35
14.76

51.44
13.45

51.38
13.86

50.69
17.01

39.82
24.58

Yellow ->red card

15.73

19.93

24.39

21.06

12.08

5.72

We train and test models on SoccerNet-v2. Both ofthem have imbalance action labels. SoccerNet-v2 annotated 300k timestamps has 17 classes of actions.
SocceNet-v2 is divided into train/val/test (300/100/100games), and we follow them.

of actions’ appropriate chunk size in vanilla transformer
and our proposed network only 5 seconds.

4.2 Results

5.Conclusion

Avg-mAP

VisibleUnshown

NetVLAD [7]

31.4

34.4

23.3

CALF [6]

40.7

42.1

29.0

In this paper, we propose a novel model based ontransformer with the pre-content and post-content encoders and obtain 52.94% average-mAP. By setting anappropriate chunk size for each class, we reach 54.6%
average-mAP on the test dataset, exceeding the current
state-of-the-art by 1.2%. Our future works will focus
on improving the performance of the few-shot class forincreasing understanding of videos.

Table 1: State-of-the-art comparison. The visible
action is shown in broadcast video and the unshown
action must be inferred by the viewer.
MethodNetVLAD++ [3]

53.4

59.4

34.8

References

Ours

54.6

60.0

33.8

[1] A. Deliège, et al., “SoccerNet-v2 : A Dataset and Benchmarks for Holistic Understanding of Broadcast SoccerVideos”, CVPRW, 2021.

Table 1 shows that our method achieves an AveragemAP of 54.56% on the test dataset and gets the state-ofthe-art performances on the SoccerNet-v2 benchmark.
We assume that irrelevant frames provide useless information and relevant frames provide signiﬁcant information to localize a speciﬁc action. The chunk size cancontrol which frames are used to predict. We change
chunk size (from 5 seconds to 90 seconds, 5 seconds asstep size) on testing to ﬁnd an appropriate chunk size
per action for action spotting. As shown in Table 2,
the appropriate chunk size per action is diﬀerent. For
instance, corner action need 10 seconds as the chunksize to get a high Average-AP score. Kick oﬀ action’sbest chunk size is 20 seconds. The best chunk size foreach action in vanilla transformer is very similar with
our proposed network. There are nine classes of actions
have the same best chunk size in vanilla transformer andour proposed network. And the diﬀerence of four classes[2] S. Giancola, et al., “SoccerNet: A Scalable Dataset for
Action Spotting in Soccer Videos”, CVPRW, 2018.
[3] S. Giancola, et al., “Temporally-Aware Feature Pooling for Action Spotting in Soccer Broadcasts”,
arXiv:2104.06779.
[4] A. Vaswani, et al., “Attention is All you Need”, NeurIPS,
2017.
[5] M. Tomei, et al., “RMS-Net: Regression and Masking
for Soccer Event Spotting”, ICPR, 2021.
[6] A. Cioppa, et al., “A Context-Aware Loss Function for
Action Spotting in Soccer Videos”, CVPR, 2020.
[7] A. Relja, et al., “NetVLAD: CNN architecture for weakly
supervised place recognition”, CVPR, 2016.

Research Achievements[1] Y. Shi, et al., “Action Spotting in Soccer Videos via
Transformer with Past and Future Encoders”, MIRU,2021.
(Other: 1 conference presentation)