# Multimodal Shot Prediction Based on Spatial-Temporal Interaction between Players in Soccer Videos
## 2 Proposed Multimodal Shot Prediction in Soccer Videos
### Graph Construction  
To effectively incorporate the match situation obtained from the soccer video, we employ the relationship between players in each frame as a graph. In the proposed method,  
Figure 1: An example of broadcast soccer videos and unedited soccer videos. To capture lengthy spatial–temporal relationships between players, the proposed method uses unedited soccer videos. Figure 2: Overview of the proposed method for multimodal shot prediction in soccer videos. In the proposed method, we adopt a graph with audio-visual features. We then obtain latent features that account for spatial-temporal relationships by using the GCRNN. Finally, we predict the probability of the shot occurrence as well as output the prediction uncertainty based on the BNN. A part of this figure is sourced from our previous paper [39]. we obtain audio segments and two types of images with full-frame size and each player region, respectively, at each time step from the soccer video. Although the audio segment does not include commentary, it contains the cheers of spectators and other auditory responses, which are considered to indicate the importance of the play and the tension involved. Especially, significant or sudden increases in cheers suggest a high likelihood of important plays, such as scoring opportunities or plays leading to them, making it effective for shot prediction. As for the object detector, we employ Mask R-CNN [42], which also performs instance segmentation simultaneously. The detected humans are used for classifying the team, referee, or other class that each detected person belongs to [43]. After converting the audio sequence into a spectrogram image, we extract features from the respective type of image. As feature extractors in the proposed method, we use ResNet-101 [44] pre-trained with ImageNet [45]. Then, to enhance the representation capability of the node features, the proposed method utilizes the features that concatenated all types of extracted audio-visual features as graph nodes \(\mathbf{X}_{t}\in\mathbb{R}^{N\times 3D}\) calculated through the following equation:  
\[\mathbf{X}_{t}=\Big{[}\Big{[}\mathbf{x}_{1,t}^{\text{player}};\mathbf{x}_{t}^{\text{frame }};\mathbf{x}_{t}^{\text{audio}}\Big{]},\Big{[}\mathbf{x}_{2,t}^{\text{player}};\mathbf{x} _{t}^{\text{frame}};\mathbf{x}_{t}^{\text{audio}}\Big{]},\dots,\Big{[}\mathbf{x}_{N, t}^{\text{player}};\mathbf{x}_{t}^{\text{frame}};\mathbf{x}_{t}^{\text{audio}}\Big{]}\Big{]}, \tag{1}\]  
where \(\mathbf{x}_{n,t}^{\text{player}}\in\mathbb{R}^{D}\) (\(n=1,\dots,N;N\) representing the total number of detected players) is the visual feature of the \(n\)-th player's region at time step \(t\). \(\mathbf{x}_{t}^{\text{frame}}\in\mathbb{R}^{D}\) and \(\mathbf{x}_{t}^{\text{audio}}\in\mathbb{R}^{D}\) are audio and visual features of the full-frame image and the audio segment, respectively. Notably, \(D\) denotes the dimension of features. In this way, with the use of audio-visual information as graph nodes, the proposed method can construct a graph that has high feature representation capability. Next, we describe the detailed settings of the graph edge in the proposed method. In soccer as a dynamic team sport, players' positions on the pitch and their relative distances from other players dramatically change over time. All of these factors significantly contribute to the instantaneous understanding of the players' environment and their decision-making [46; 47]. Thus, we use the graph edge based on players' field positions and the team information, as well as their distances from other players. The weights of the graph edges at time step \(t\) are represented by the adjacency matrix \(\mathbf{A}_{t}\in\mathbb{R}^{N\times N}\) as follows:  
\[\mathbf{A}_{t}^{(ij)} =\left\{ \begin{array}{cc}0&(\text{team}(i)=\text{team}(j))\\ \dfrac{a_{t}^{(ij)}}{\sum_{i}a_{t}^{(ij)}}&(\text{team}(i)\
eq\text{team}(j)) \end{array}