# A Graph-Based Approach to Recognizing Complex Human Object Interactions in Sequential Data
## 3 Materials and Methods
### Human-Object Interaction Recognition (HOIR)  
The final step of the proposed system is the classification of interactions that is performed by a graph convolution network (GCN) [52]. GCNs are a very powerful neural network architectures for machine learning on graphs. In fact, they are so powerful that even a randomly initiated 2-layer GCN can produce useful feature representations of nodes in networks. The relative nearness of nodes in the network is preserved in the 2-dimensional representation, even without any training. The classic method to perform image classification is using convolutional neural networks (CNN). Images are represented in the form of pixels, and the CNN runs sliding kernels (or filters) across the images; the model subsequently learns important features by looking at the adjacent pixels. GCNs, on the other hand, view images as complete graphs. Each node represents each pixel. Node feature represents the pixel value. Edge feature represents the Euclidean distance between each pixel. The closer two pixels are to each other, the larger the edge values. Figure 12 shows a general architecture of a GCN.  
The feature representation at any layer in a neural network can be represented by Equation (13):  
\[H_{i}=\sigma(W_{i-1}H_{i-1}+b_{i-1}) \tag{13}\]  
where \(\mathbf{H}_{i}\) is the feature representation at layer \(\mathbf{i}\), \(\mathbf{\sigma}\) is the activation function, \(\mathbf{W}_{i-1}\) represents the weight matrix at layer \(\mathbf{i}-1\), \(\mathbf{H}_{i-1}\) is the feature representation at layer \(\mathbf{i}-1\), and \(\mathbf{b}_{i-1}\) is the bias at layer \(\mathbf{i}-1\). For the forward propagation equation of a GCN, the adjacency matrix \(\mathbf{A}\) is also taken into account, as shown in Equation (14):  
\[\mathbf{H}_{i}=\sigma(\mathbf{W}_{i-1}\mathbf{H}_{i-1}\mathbf{A}^{*}) \tag{14}\]  
where \(\mathbf{A}^{*}\) is the normalized adjacency matrix. The GCN architecture used in this research consists of two GCN layers with output dimensionalities of 1024 and 2048, respectively. For both layers, the Leaky ReLU activation function with the negative slope of 0.2 is used. Equation (15) represents the Leaky ReLU activation function, where \(\mathbf{x}\) represents the input vector. It is a non-linear activation function which leads to faster convergence in experiments. Moreover, each convolutional layer is followed by a max pooling layer. The output from the final pooling layer is flattened and entered into a fully convolutional (FC) layer which is followed by a softmax layer, as represented by Equation (16), where \(\mathbf{x}_{j}\) represents the input vector. Equation (17) represents the output of this model. During training, the input images are cropped and resized into 448 \(\times\) 448. For network optimization, stochastic gradient descent (SGD) is used as the optimizer. The momentum is set to be 0.9 and weight decay is \(10^{-4}\). The initial learning rate is 0.01, which is decayed by a factor of 10 for every 40 epochs, and the network is trained for 100 epochs in total.  
\[\mathbf{LeakyReLU}(\mathbf{x})=\left\{\begin{array}{rl}\mathbf{0.01x},&\mathbf{x}<\mathbf{0}\\ &\mathbf{x},&\mathbf{x}\geq\mathbf{0}\end{array}\right. \tag{15}\]  
Figure 12: A general architecture of a GCN.  
\[\mathbf{softmax}(\mathbf{x}_{i})=\frac{\mathbf{e}^{\mathbf{x}_{i}}}{\sum_{j=1}^{n}\mathbf{e}^{\mathbf{x}_{ j}}} \tag{16}\]  
\[\mathbf{Z}=\mathbf{softmax}(A^{*}\mathbf{LeakyReLU}(\mathbf{W}_{i-1}\mathbf{H}_{i-1}A^{*})\mathbf{W}_{i}) \tag{17}\]