## 3 The Proposed Methodology
### Key Frame Extraction  
Instead of frame-by-frame analysis of the entire video sequence for the extraction of human skeleton features, which could then be used for action recognition, we propose the extraction of key frames from the input video, which are subsequently analyzed for the extraction of human skeleton features. These human skeleton features, also known as human keypoints, are then used to distinguish between different actions using a classifier. Key frames dramatically reduce the amount of data that must be processed to extract the features that we propose to use for action recognition and capture only those parts of the video which undergo a significant transition. Key frames are identified by utilizing the gradient of the optical flow displacement field between every pair of consecutive video frames. Specifically, a two-step procedure is followed to determine the key frames.  
First, dense optical flow is used to compute the changes in intensity \(I(x,y,t)\) or velocity that each pixel undergoes in two successive frames [40] along both the horizontal and vertical directions. The velocity's horizontal and vertical components are summed over all the pixels for each frame to compute an estimate of the total velocity, which is used as a motion metric \(M(t)\) for every frame \(t\). The relation for computing the motion metric \(M(t)\) is given in Equation (1):  
\[M(t)=\sum_{1}^{m}\sum_{1}^{n}V_{x}(x,y,t)+V_{y}(x,y,t), \tag{1}\]  
where \(m\) represents the width of the image in pixels, \(n\) represents the height of the image in pixels, \(V_{x}(x,y,t)\) is the horizontal or \(x\) component of the velocity or optical flow that estimates the gradient of intensity along the \(x\)-axis for the pixel at coordinates \((x,y)\) between two successive frames \(t\) and \(t+1\), which is given by  
\[V_{x}(x,y,t)=\frac{\partial I(x,y,t)}{\partial x}, \tag{2}\]  
and \(V_{y}(x,y,t)\) is the vertical or \(y\) component of the velocity or optical flow that estimates the gradient of intensity along the \(y\)-axis for the pixel at coordinates \((x,y)\) between two successive frames \(t\) and \(t+1\), which is given by  
\[V_{y}(x,y,t)=\frac{\partial I(x,y,t)}{\partial y}. \tag{3}\]  
Figure 1: The proposed framework for the real-time vision system of a social robot that incorporates action recognition with face and emotion recognition.  
As dense optical flow tracks the intensity variations across all the pixels of a frame, therefore, the motion metric in Equation (1) provides an estimate of the amount of motion along consecutive frames.  
Secondly, the proposed motion metric \(M(t)\) is analyzed as a function of time by looking at its variation across different frames. The gradient of the proposed motion metric \(\frac{dM}{dt}\) can be used to identify the frames between which there is significant activity. This can be easily performed by looking at the local extrema of \(\frac{dM}{dt}\) to identify the key frames (i.e., frames where there is a significant transition or change taking place). Thus, by quantifying motion or the dense optical flow across consecutive video frames into a single metric, and then by analyzing the gradient of this motion metric, we identify the key frames, which are then used for the extraction of skeleton features using pose estimation models as described in Section 3.2.