# Multimodal Shot Prediction Based on Spatial-Temporal Interaction between Players in Soccer Videos
## 3 Experiments
### Settings  
In this experiment, we used a dataset consisting of video clips derived from scouting videos of 31 matches in the J1 League for the 2019 and 2020 seasons. Video clips that included a shot event were defined as positive samples, and video clips that included attack scenes without a shot event were defined as negative samples. As a result, therewere 400 video clips with 200 positive and 200 negative samples. Each clip consisted of 6 s of a video acquired at 10 fps, with the frame size being \(1280\times 720\) pixels. We used 80% of the video clips as a training dataset and the remaining 20% as a test dataset, and 5-fold cross-validation was conducted. The dimension \(D\) of each image feature \(\mathbf{x}_{t}^{\text{player}}\), \(\mathbf{x}_{t}^{\text{frame}}\), and \(\mathbf{x}_{t}^{\text{audio}}\) applied to fully connected layers after being extracted from ResNet-101 was 256 dimensions, that is, the graph node feature \(\mathbf{X}_{t}\) that combined them was 768 dimensions. The maximum number of detected players was set at 20, corresponding to the number of field players excluding the goalkeeper. Furthermore, the GCN had two layers, and the first layer transformed graph node \(\mathbf{X}_{t}\) into 256-dimensional features. In the second layer, by inputting the hidden state \(\mathbf{H}_{t-1}\) of the previous time step in addition to the output of the first GCN layer, the latent feature \(\mathbf{F}_{t}\) was obtained. Here, the dimensions of \(\mathbf{H}_{t-1}\) and \(\mathbf{F}_{t}\) were both 256. As for the GCRN, the hidden state \(\mathbf{H}_{t}\) was updated using a graph whose nodes were 1024-dimensional features concatenated from graph node feature \(\mathbf{X}_{t}\) and latent feature \(\mathbf{F}_{t}\) in the same manner as the hidden state \(\mathbf{H}_{t-1}\) at the previous time step. Lastly, by flattening and then inputting the latent feature \(\mathbf{Z}_{t}\), the prediction results were calculated with two-layer BNNs that output 64 and 2 dimensions, respectively, that is, the prediction probabilities of positive and negative event occurrences. In the training phase, two forward passes were conducted (\(M=2\)), and weights were sampled twice (\(K=2\)) in each pass by Monte Carlo sampling to learn the probability distribution of weights. Thereby, the proposed model captured the uncertainty and improved the reliability of the prediction results. In the inference phase, the proposed model made eleven forward passes (\(M=11\)) (10 stochastic forward passes and a deterministic forward pass) and ten samplings of weights (\(K=10\)). In this way, by utilizing the probability distribution of the weights and the prediction uncertainty acquired in the training phase, our model provided more robust predictions for unknown input data. This setting emphasized computational efficiency in training and more accurate uncertainty evaluation and prediction accuracy in inference. The event occurrence time \(T_{\text{s}}\) of positive samples varied with each video clip and was manually determined. As shown in Figure 4, the distribution of \(T_{\text{s}}\) is concentrated in the latter half of the video. The video clips included in the training dataset were given target labels \(\mathbf{y}=(y^{(n)},y^{(p)})^{\top}\) for the entire video clip, which were defined as \(\mathbf{y}=(0,1)^{\top}\) for positive and \(\mathbf{y}=(1,0)^{\top}\) for negative. The \(w_{a}\) in Equation (3) was set experimentally to 1050, which is the length of the touchline of the field. Moreover, \(w_{1}\) and \(w_{2}\) in Equation (19) were set to 0.001 and 10, respectively, in order to balance the edge weights and loss values. In the training phase, the learning rate was initialized to \(5\times 10^{-4}\), and we employed [PERSON] [57] and ReduceLROnPlateau as the optimizer and scheduler, respectively. The batch size was set to 16, and it was trained in 60 epochs. To validate the effectiveness of the proposed model (PM) for shot prediction from soccer videos, we first conducted ablation studies that excluded the novelty from the PM.