Athlete Pose Detection Using OpenCV in Deep Learning
By: Manali Seth
Tags: Machine Learning, Pose Estimation, Deep Learning, Opencv

Nowadays, the craze of sports is increasing lavishly. In the same way, the risk of attempting to play sports in an imprecise manner is also increasing. A small injury may lead to a bad impact. Keeping in mind such consequences, here is a proposed solution in order to analyze joint movements of a sportsperson which will help the athlete to improve the pose.

Human pose estimation is an important problem in the field of Computer Vision. Its algorithms help to locate the parts such as wrists, ankles, knees etc. This is done in order to provide personalized athletic training experiences using concepts of Deep Learning and Convolutional Neural Network. It is especially for sports activities in which the training quality largely depends on the correctness of human poses in an image or a video sequence.

Detecting an athlete’s pose from an image or video sequence

The technology has huge implications. Applications may include video surveillance, advanced driver assistance systems, sport analysis and many more. In athlete pose detection, it is required to detect and localize the major parts/joints of the body. Below is the proposed system which can lead to a remarkably better user training experience.

Datasets

The proper selection of datasets to have an appropriate impact on the results is also very necessary. In this pose detection, the model is pre-trained on two different datasets namely, COCO Keypoint Dataset and MPII Human Pose Dataset.

1. COCO: The COCO keypoints dataset is a multi-person 2D Pose Estimation dataset with images collected from Flickr. COCO is the largest 2D Pose Estimation dataset, till date, and is considered as a benchmark for testing 2D Pose Estimation algorithms. COCO model produces 18 points.

COCO Output Format: Nose — 0, Neck — 1, Right Shoulder — 2, Right Elbow — 3, Right Wrist — 4, Left Shoulder — 5, Left Elbow — 6, Left Wrist — 7, Right Hip — 8, Right Knee — 9, Right Ankle — 10, Left Hip — 11, Left Knee — 12, Left Ankle — 13, Right Eye — 14, Left Eye — 15, Right Ear — 16, Left Ear — 17, Background — 18

2. MPII: The MPII human pose dataset is a multi-person 2D Pose Estimation dataset comprising of nearly 500 different human activities, collected from Youtube videos. MPII was the first dataset to contain such a diverse range of poses and the first dataset to launch a 2D Pose estimation challenge in 2014. MPII model outputs 15 points.

MPII Output Format: Head — 0, Neck — 1, Right Shoulder — 2, Right Elbow — 3, Right Wrist — 4, Left Shoulder — 5, Left Elbow — 6, Left Wrist — 7, Right Hip — 8, Right Knee — 9, Right Ankle — 10, Left Hip — 11, Left Knee — 12, Left Ankle — 13, Chest — 14, Background — 15

These points are generated when the dataset is processed upon and trained thoroughly through Convolutional Neural Network (CNN).

Implementation

Step 1: Requirement Gathering (Model Weights) and Load Network

The trained models need to be loaded in OpenCV. The models are trained on Caffe Deep Learning Framework. Caffe model consists of two files namely .prototxt file and .caffemodel file.

.prototxt file specifies the architecture of neural network. .caffemodel file stores the weights of the trained model.

Then we load these two files in the network .

# Specify the paths for the 2 files

protoFile = "pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt"

weightsFile = "pose/mpi/pose_iter_160000.caffemodel" # Read the network into Memory

net = cv2.dnn.readNetFromCaffe(protoFile, weightsFile)

Step 2: Read image and prepare input to the network

Firstly we need to convert the image from OpenCV format to Caffe blob format using the blobFromImage function so that it can be fed as input to the network. The parameters are to be provided in the blobFromImage function. There is no need to swap the R and B channels since both OpenCV and Caffe use BGR format.

# Read image

frame = cv2.imread("image.jpg") # Specify the input image dimensions

inWidth = 368

inHeight = 368 # Prepare the frame to be fed to the network

inpBlob = cv2.dnn.blobFromImage(frame, 1.0 / 255, (inWidth, inHeight), (0, 0, 0), swapRB=False, crop=False) # Set the prepared object as the input blob of the network

net.setInput(inpBlob)

Step 3: Make predictions and parse keypoints

Once the image is passed to the model, the predictions can be made using the forward method for the DNN class in OpenCV which makes a forward pass through the network which is just another way of saying it is making a prediction.

output = net.forward()

The output is a 4D matrix :

The first dimension being the image ID ( in case you pass more than one image to the network ). The second dimension indicates the index of a keypoint. The model produces Confidence Maps (probability distribution over the image, representing the confidence of the joint location at every pixel) and Part Affinity maps which are all concatenated. For COCO model it consists of 57 parts — 18 keypoint confidence Maps + 1 background + 19*2 Part Affinity Maps. Similarly, for MPI, it produces 44 points. We will be using only the first few points which correspond to Keypoints. The third dimension is the height of the output map. The fourth dimension is the width of the output map.

Then we check whether each keypoint is present in the image or not. We get the location of the keypoint by finding the maxima of the confidence map of that keypoint. We also use a threshold to reduce false detections.

Confidence Map

Once the keypoints are detected, we plot them on the image.

H = out.shape[2]

W = out.shape[3]

# Empty list to store the detected keypoints

points = []

for i in range(len()):

# confidence map of corresponding body's part.

probMap = output[0, i, :, :] # Find global maxima of the probMap.

minVal, prob, minLoc, point = cv2.minMaxLoc(probMap) # Scale the point to fit on the original image

x = (frameWidth * point[0]) / W

y = (frameHeight * point[1]) / H if prob &amp;gt; threshold :

cv2.circle(frame, (int(x), int(y)), 15, (0, 255, 255), thickness=-1, lineType=cv.FILLED)

cv2.putText(frame, "{}".format(i), (int(x), int(y)), cv2.FONT_HERSHEY_SIMPLEX, 1.4, (0, 0, 255), 3, lineType=cv2.LINE_AA)

# Add the point to the list if the probability is greater than the threshold

points.append((int(x), int(y)))

else :

points.append(None) cv2.imshow("Output-Keypoints",frame)

cv2.waitKey(0)

cv2.destroyAllWindows()

Step 4: Draw the skeleton

Since we already plotted the keypoints, we can now draw the skeleton just by joining the pairs.

for pair in POSE_PAIRS:

partA = pair[0]

partB = pair[1]

if points[partA] and points[partB]:

cv2.line(frameCopy, points[partA], points[partB], (0, 255, 0), 3)

Result

Image Pose Detection

The output shown above shows us the exact posture of an athlete at a particular moment.

Check out the Athlete Pose Detection from video sequence in the below links.

https://youtu.be/UdtLLrS1ft0 (input video)

https://youtu.be/SgAlHivxJ8E (output video)

Entire source code for reference is provided in my Github Repository Link https://github.com/ManaliSeth/Athlete-Pose-Detection

Hope this entire document be useful for all working upon concepts of OpenCV, Deep Learning and Pose Estimation.