# Multimodal Shot Prediction Based on Spatial-Temporal Interaction between Players in Soccer Videos
## 2 Proposed Multimodal Shot Prediction in Soccer Videos
### Graph Convolutional Recurrent Neural Network  
To learn the spatial-temporal interaction of graphs and calculate latent features, we adopt GCRNN. In previous studies [48; 49; 50], several architectures of GCRNN have been proposed. The basic GCRNN architecture first updates the graph node features by applying a GCN to the graph data. The updated graph features are used to acquire hidden states by RNNs or its derivative methods such as gated recurrent unit (GRU) [51] and long short-term memory (LSTM) [52]. By inputting such hidden states to the GCN at the next time step, the node features of the graph can be updated to take into account not only the spatial relationships of the graph but also the temporal-dependent relationships. In the GCRNN model used in the proposed method, as input to the constructed graph in Section 2.1, we calculate the latent feature \(\mathbf{F}_{t}\in\mathbb{R}^{N\times D}\) through the following equation:  
\[\mathbf{F}_{t} =\text{ReLU}(\mathbf{\hat{A}}_{t}[\text{ReLU}(\mathbf{\hat{A}}_{t}\mathbf{X} _{t}\mathbf{W}_{f}^{(0)});\mathbf{H}_{t-1}]\mathbf{W}_{f}^{(1)}), \tag{4}\] \[\mathbf{\hat{A}}_{t} =\mathbf{I}_{N}+\mathbf{D}^{-\frac{1}{2}}\mathbf{A}_{t}\mathbf{D}^{-\frac{1}{2}}. \tag{5}\]  
Here, \(\mathbf{\hat{A}}_{t}\) represents the normalized adjacency matrix with added self-connections. \(\mathbf{I}_{N}\) and \(\mathbf{D}\) are the identity and degree matrices, respectively. \(\mathbf{W}_{f}^{(0)}\in\mathbb{R}^{3D\times D}\) and \(\mathbf{W}_{f}^{(1)}\in\mathbb{R}^{2D\times D}\) are trainable weight matrices. \(\text{ReLU}(\cdot)=\text{max}(0,\cdot)\) is a widely used activation function in neural networks. The proposed method uses two stacked GCN layers to calculate latent feature \(\mathbf{F}_{t}\). In the constructed complete bipartite graph, the node features are propagated two hops away in the graph, which means that the graph is updated by considering all players through the players of the opposite team. Furthermore, the final layer of GCNs is enhanced with the hidden state \(\mathbf{H}_{t-1}\in\mathbb{R}^{N\times D}\) of the GRU-based GCRN at the previous time step. \(\mathbf{H}_{t-1}\) is calculated from the following equation:  
\[\mathbf{Z}_{t} =\beta(\mathbf{W}_{xz}*_{\mathcal{G}}\mathbf{G}_{t}+\mathbf{W}_{tz}*_{ \mathcal{G}}\mathbf{H}_{t-1}), \tag{6}\] \[\mathbf{R}_{t} =\beta(\mathbf{W}_{xz}*_{\mathcal{G}}\mathbf{G}_{t}+\mathbf{W}_{hr}*_{ \mathcal{G}}\mathbf{H}_{t-1}),\] (7) \[\mathbf{\hat{H}}_{t} =\tanh(\mathbf{W}_{sh}*_{\mathcal{G}}\mathbf{G}_{t}+\mathbf{W}_{hh}*_{ \mathcal{G}}(\mathbf{R}_{t}\odot\mathbf{H}_{t-1})),\] (8) \[\mathbf{H}_{t} =(1-\mathbf{Z}_{t})\odot\mathbf{\hat{H}}_{t}+\mathbf{Z}_{t}\odot\mathbf{H}_{t-1}. \tag{9}\]  
To determine the hidden state \(\mathbf{H}_{t}\) at each time step, GCRN uses the update gate \(\mathbf{Z}_{t}\in\mathbb{R}^{N\times D}\), the reset gate \(\mathbf{R}_{t}\in\mathbb{R}^{N\times D}\), and the new memory content \(\mathbf{\tilde{H}}_{t}\in\mathbb{R}^{N\times D}\). The update gate \(\mathbf{Z}_{t}\) helps determine how much past information to retain and pass to the next time step. The reset gate \(\mathbf{R}_{t}\) determines how much past information to forget. The new memory content \(\mathbf{\tilde{H}}_{t}\) is composed of the input \(\mathbf{G}_{t}\) and the past information retained by the reset gate \(\mathbf{R}_{t}\). Each  
Figure 3: The expected goals (xG) used in the proposed model. Each zone is assigned a value based on the mean xG of shots taken within that zone in Major League Soccer. gate of the GCRN receives the graph \(\mathbf{G}_{t}=([\mathbf{X}_{t};\mathbf{F}_{t}],\mathbf{A}_{t})\) with features that combine the initial graph node and the nodes obtained through GCNs. The operator \(*_{\mathcal{G}}\) is the graph convolution operator, and \(\odot\) denotes the Hadamard product. \(\beta(\cdot)\) and \(\tanh(\cdot)\) are the sigmoid function and the hyperbolic tangent function, respectively. \(\mathbf{W}_{x}\). \(\in\mathbb{R}^{D\times 4D}\) and \(\mathbf{W}_{h}\).