Mean Shift and Cam Shift Object Tracking | by Andrew Udell | Towards Data Science

Among the most demanded features of computer vision is object tracking. Unfortunately, this is also one of the most difficult aspects of computer vision to implement.
Not only does an object need to be identified, it needs to be identified quickly enough to render in real time while it moves. The object may change orientation or change its distance from the camera, which further complicates matters.
While still an adamant challenge in computer vision, the mean shift and cam shift algorithms present a simplified solution. Working primarily with color data rather than structural data, these methods can efficiently track objects under the right conditions.
The mean shift algorithm is actually a broader statistical concept related to clustering. Like most other clustering algorithms, the mean shift algorithm attempts to look for places in the data set with a high concentration of data points, or clusters.
The algorithm places a kernel at each data point and sums them together to make a Kernel Density Estimation (KDE). The KDE will have peaks and valleys, which correspond to places with a high and low data point density, respectfully.
The algorithm takes a copy of the data points and shifts the points by a small amount to their nearest KDE peak in a single iteration. The algorithm will continue to go through iterations of shifting the points until the points no longer move much. This can help simplify the data and make the clusters more pronounced.
So how does this related to computer vision? Imagine a rainbow, like in the image below:
The image may be broken down into the HSV color space, essentially turning it into a big data set. When making a KDE, each of the colors of the rainbow and the blue in the sky will create a peak. When shifting the points iteratively, each color of the rainbow will become more distinct by eliminating the subtle fading between colors. Consequently, each stripe become more easily identifiable by a computer.
This process is a form of image segmentation.
To take the concept further, if a particular color in the rainbow needed to be identified, the mean shift algorithm makes it more easily seen. If this color needed to be tracked, each frame of a video of a rainbow would undergo the algorithm in real time.
For a more realistic example, image a top-down view of a tennis match. The court, the players, and the bright yellow ball will each have a KDE peak. If the ball needs to be tracked, the mean shift algorithm will make the mass of bright yellow pixels more defined and track it at each frame as it moves between the players.
Before implementing the mean shift algorithm, an initial tracking box needs to be established. As the mean shift algorithm iteratively shifts points, the tracking box will move until surrounds the object of interest.
Unfortunately, if the object shifts in size or changes orientation, the tracking box doesn’t accommodate the change. The cam shift (Continuously Adaptive Mean Shift)algorithm addresses this issue.
Working very similarly as the mean shift, the cam shift algorithm simply adjusts it so that the tracking box may change in size or even rotate to better correlate to the movements of the tracked object.
Unlike deep neural networks or other methods of identification, mean shift and cam shift don’t require any training. If a tennis ball needs to be tracked, there’s no need to feed the algorithm hundreds or thousands of labeled images of a tennis balls. Instead, the algorithm analyzes the initial color input of the tennis ball and tracks it immediately throughout the rest of its lifetime.
Unfortunately, mean shift and cam shift only work with color analysis. If the object being tracked varies significantly in color or texture, it will be harder to track. Likewise, if an image or video has a “busy” or “noisy” background with a lot of color variation, it may conflict with the object being tracked.
Ultimately, mean shift and cam shift work best under controlled environments. For example, tracking product of uniform colored packaging through a warehouse or assembly line might perfectly suit these algorithms.
As cam shift is the more robust of the two algorithms, its implementation in python and OpenCV will be explained in deeper detail. A brief over of implementing mean shift, however, may be found here.
import numpy as npimport cv2
The only two libraries needed are NumPy and OpenCV. The former helps with efficient calculation and the latter provides many of tools used in computer vision.
# initialize video from the webcamvideo = cv2.VideoCapture(0)# Read the videocap, frame = video.read()
The first line simply activates the webcam (if there’s a camera present). The read method actually captures two important pieces of information:
the cap variable, which indicates whether or not the camera successfully captured a framethe frame variable, which is an actual still image, or frame, from the camera feed
the cap variable, which indicates whether or not the camera successfully captured a frame
the frame variable, which is an actual still image, or frame, from the camera feed
# set up initial coordinates for the tracking windowx, y = 0, 0# Set up initial size of the tracking windowheight, width = 25, 25track_window = (x,y,width,height)# set up region of interest (roi)roi = frame[y:y + height, x:x + width]
This next section establishes the initial Region of Interest (ROI), which is simply the area of the screen that features the object being tracked.
The x and y variables are the coordinates within the frame. In this example, coordinates (0, 0) are used because its easy to find once the camera starts streaming.
The height and width indicate the size of the ROI. It’s very important that the size of the ROI at least roughly correspond with the size of the object being tracked.
The track_window variable stores the position and size of the ROI and the roi variable actually uses splicers to takes a small subsection of the frame corresponding to the ROI.
Whatever the camera “sees” in this ROI will dictate the color range it tracks. For example, if a red ball is in this initial ROI, the algorithm will know to track a red object for the rest of the feed.
If, however, part of the ball is out of the ROI, and more of a white background is captured, the algorithm will be more likely to track a white object. Capturing the correct object in this initial ROI will be critical to ensuring it works properly.
TIP: When I was learning how to do this, I “cheated” by holding the object of interest up close to the camera so that the whole screen was filled. This forced the algorithm to read the correct color information, no matter the initial parameters I used. This, however, is not practical for most applications.
# apply maskhsv_frame =  cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)mask = cv2.inRange(hsv_frame, np.array((0, 20, 20)), np.array((180, 250, 250)))hist_frame = cv2.calcHist([hsv_frame], [0], mask, [180], [0,180])cv2.normalize(hist_frame, hist_frame, 0, 255, cv2.NORM_MINMAX)
The next couple of lines convert the frame into the HSV color space and apply a “mask”, which simply looks for pixel values that fit within a certain range.
The histogram is calculated for the frame after the the mask is applied and the values are normalized. This serves to calculate the colors that the algorithm will track moving forward.
# Setup the termination criteria: 10 iteration 1 pxl movementterm_crit = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1 )
After the initial values are captured, a termination criteria needs to be established. Recalling that both mean shift and cam shift are iterative functions that continuously shift points, some sort of criteria must be established to stop its iteration and return a final set of values.
The above line stop the iterations if either one of two things happen:
the centroid of the ROI hasn’t moved, orthere are more than 10 iterations
the centroid of the ROI hasn’t moved, or
there are more than 10 iterations
The first criteria makes intuitive sense. If an object is being tracked and it doesn’t move, there’s no need to run the algorithm. The ROI doesn’t change, so it doesn’t need to be recalculated. Doing so would only waste computational power.
The second criteria is more concerned with performance. In theory, more iterations will render a more accurate ROI; however, more iterations take more time to run. Setting a higher limit than 10 may build a better bounding box, but the results will be lagging.
while True:        cap, frame = video.read()    if cap == True:        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)        dst = cv2.calcBackProject([hsv],[0],hist_frame,[0,180],1)
The first line starts an infinite loop as the camera will stream indefinitely until an exit command is triggered.
The next line, like before, reads data from the camera. The following if-statement checks to determine if the camera successfully captured a frame.
The last two lines convert the frame into the HSV color space and uses histogram backprojection to find which pixels that best correspond to the color of the object captured in the initial ROI.
        # apply camshift to get the new location        ret, track_window = cv2.CamShift(dst, track_window, term_crit)
Instead of implementing the cam shift algorithm from scratch, OpenCV has a built-in module to apply it. This will return the size and coordinates of the new ROI.
       # Draw a box around the ROI        pts = cv2.boxPoints(ret)        pts = np.int0(pts)        img2 = cv2.polylines(frame,[pts],True, 255,2)
Finally, draw a box around the ROI to visually indicate it.
        cv2.imshow('img2',img2)        # Use the q button to quit the operation        if cv2.waitKey(60) & 0xff == ord('q'):            break    else:        breakcv2.destroyAllWindows()video.release()
The image is shown in real time in the first line. The next line simply creates an exit condition. If the letter “q” on the keyboard is pressed, it breaks the loop and exits the video.
Likewise if the camera is suddenly unable to successfully capture video, it will also break the loop.
Finally, in the last two lines, any related windows are closed and the camera stops streaming.
If all goes well, the final result should look something like this:
Note how the bounding box increases and decreases in size as the envelope goes closer and further away from the camera. It also rotates along with envelope and even compresses when is turned on its side.
The mean shift and cam shift algorithms are both incredibly useful and powerful tools to use in object tracking. The latter, in particular, can shift its ROI as the object rotates or changes distance from the camera. The result is a robust method of tracking.
Using color as a primary method of identification, neither algorithm can identify something with a particular shape, which makes it less powerful than other methods. Likewise, complex or noisy backgrounds will make identification more problematic. As a tradeoff, however, using color allows for immediate implementation without the need for training the model.
Either work well in controlled, well-lit environments and offer an efficient method of object tracking.