Video Analysis with Tensor Decomposition in Python | by Célestin Hermez | Towards Data Science

Artificial Intelligence. Deep Learning. Convolutional Neural Networks. Reinforcement Learning. These are revolutionary advances in the field of machine learning, and have made the impossible possible. Despite all these qualities, there are also drawbacks and limitations. For instance, neural networks are prone to overfitting due to the huge training sets they require. They are often task-specific, and their prowess does not translate well to other contexts.
While ongoing research addresses these issues (I am personally fascinated by OpenAI’s efforts towards artificial general intelligence), alternative methods can also come to the rescue. For instance, one of the main challenges in computer vision is the amount of data involved: one image is often represented as a matrix with millions of elements, and a video contains thousands of such images. Moreover, noise is often present in such data. As a result, unsupervised learning techniques which reduce dimensionality are potential sources of improvement.
With this in mind, the concept of tensor decomposition is particularly powerful in a high dimensional data context. Implementing these ideas in Python to analyze videos yields significant insights, and can act as a useful pre-processing stage before other methods.
The concept of high dimensional data analysis refers to a group of problems where the number of features is great compared to the number of observations. In many applications (regression being the most familiar), this leads to speed and model learning issues, such as overfitting or even impossibility to produce a model. This is commonplace in computer vision, material science or even business, as tremendous amounts of data are captured on the Internet.
One of the solutions is to find a low dimensional representation of the data, and use it as training observations in our models since dimensionality reduction alleviates the issues listed above. This lower dimensional space often contains most of the information, and is thus an adequate replacement. Splines, regularization and tensor decomposition are examples of this approach. We will dive into the latter method and see one of its applications.
The core concept of this article is that of a tensor, which is simply a multidimensional array:
a number is a 0-dimensional tensor
a vector is a 1-dimensional tensor
a matrix is a 2-dimensional tensor
beyond this, we simply refer to the dimensions of the tensor
This data structure is particularly useful to store images or videos. In the traditional RGB model, a single image is represented by a 3D tensor:
each pixel has (x,y) coordinates in a matrix; as a result the size of the matrix depends on the resolution of the image
each color channel (red, green, blue) has its own matrix, where the value of a given pixel in a matrix encodes the intensity in this specific color
Going a step further, a video is simply a succession of frames, where each frame is an image. It becomes a bit harder to visualize, but this can be stored in a 4D tensor: 3 dimensions are used to store a single frame, and the 4th dimension encodes the passage of time.
To make it more concrete, let’s take the example of a 60s video at 60fps (frames per seconds) and resolution 800 x 600. This video can be stored in a 800 x 600 x 3 x 3600 tensor. It will thus have 5 billion elements! This is too many to hope to build a reliable model. This is where tensor decomposition comes in.
There is extensive literature on tensor decomposition, and I point the interested reader to Kolda and Balder’s great survey on the topic1. In particular, Tucker decomposition has many applications, such as tensor regression, with tensors as target2 or predictor3 variables. The key takeaway is that it allows the extraction of a core tensor, a compressed version of the original data. If this reminds you of PCA, you are correct: one of the steps for Tucker decomposition is actually Higher Order Singular Value Decomposition, an extension of SVD.
Existing algorithms allow extracting the core tensor along with factorizing matrices (not used in our application). The hyperparameter is the n-rank. Without going into too much detail, the main idea is that the higher the n-rank value, the more precise the decomposition. The n-rank also controls the size of the core tensor. With a small n-rank, the reconstructed tensor may not exactly match the original, but we gain a lot in terms of dimensionality reduction: the tradeoff depends on the application at hand.
Extracting this core tensor yields important benefits, as we can see in an application to actual data.
As a toy example, I used my phone to capture three 10s videos:
the patio of a cafe I like
a parking lot
cars driving on the highway during my afternoon commute
I uploaded them (and the development notebook) on GitHub. My main objective was to determine whether we could rigorously rank potential pairs in terms of similarity, with the assumption that the parking lot and commute videos would be most similar.
I leveraged the popular OpenCV Python library to load and manipulate this data prior to analysis. The steps are as follows:
create VideoCapture objects and extract the number of frames for each one — I use the shorter video to truncate the other two in order to have a common basis for comparison
# Import librariesimport cv2import numpy as npimport randomimport tensorly as tlfrom tensorly.decomposition import tucker# Create VideoCapture objectsparking_lot = cv2.VideoCapture('parking_lot.MOV')patio = cv2.VideoCapture('patio.MOV')commute = cv2.VideoCapture('commute.MOV')# Get number of frames in each videoparking_lot_frames = int(parking_lot.get(cv2.CAP_PROP_FRAME_COUNT))patio_frames = int(patio.get(cv2.CAP_PROP_FRAME_COUNT))commute_frames = int(commute.get(cv2.CAP_PROP_FRAME_COUNT))
read them in a tensor of appropriate dimensions. I chose TensorLy to work with tensors
# Create function to read all frames of a video in an arraydef read_frames(video_capture, max_frames):    """    INPUTS:    video_capture: an OpenCV VideoCapture object whose frames we   want to read    max_frames: the maximum number of frames we want to read        OUTPUT:    array of all the frames until max_frames    """    # Initialize empty array    frames_array = []        # Keep track of the frame number    frame_nb = 0        # iterate through the frames and append them to the array    while video_capture.isOpened() and frame_nb < max_frames:        ret, frame = video_capture.read()        if not ret:            break        frames_array.append(frame)        if cv2.waitKey(1) & 0xFF == ord('q'):            break        frame_nb += 1        # release the video capture    video_capture.release()    cv2.destroyAllWindows()        # return the array    return(frames_array)# Read in all the videosparking_lot_array = read_frames(video_capture=parking_lot, max_frames=commute_frames)patio_array = read_frames(video_capture=patio, max_frames=commute_frames)commute_array = read_frames(video_capture=commute, max_frames=commute_frames)# Create tensors from matricesparking_lot_tensor = tl.tensor(parking_lot_array)patio_tensor = tl.tensor(patio_array)commute_tensor = tl.tensor(commute_array)
randomly sample 50 frames from these tensors to speed up future operation
# Set the seed for reproducibilityrandom.seed(42)random_frames = random.sample(range(0, commute_frames), 50)# Use these random frames to subset the tensorssubset_parking_lot = parking_lot_tensor[random_frames,:,:,:]subset_patio = patio_tensor[random_frames,:,:,:]subset_commute = commute_tensor[random_frames, :, :, :]# Convert three tensors to doublesubset_parking_lot = subset_parking_lot.astype('d')subset_patio = subset_patio.astype('d')subset_commute = subset_commute.astype('d')
After these steps, three 50 x 1080 x 1920 x 3 tensors are available. We can move forward with analysis.
In order to determine how similar these videos are to one another, we need a measure to rank them. The L2 norm of the difference between two tensors is a common measure of similarity. A smaller value is associated with greater similarity. Mathematically, the norm of a tensor is
The norm of the difference is thus analogous to Euclidean distance.
Implementing this with the full tensors yields unsatisfactory results.
# Parking and patioparking_patio_naive_diff = tl.norm(subset_parking_lot - subset_patio)# Parking and commuteparking_commute_naive_diff = tl.norm(subset_parking_lot - subset_commute)# Patio and commutepatio_commute_naive_diff = tl.norm(subset_patio - subset_commute)
Displaying the similarity as a table:
Not only is there no clear ranking between pairs, but the parking lot and patio videos also appear to be most similar, which contrasts with the initial assumption. This is where Tucker decomposition comes into play.
The TensorLy library makes it relatively easy (although slow) to decompose tensors: all we need is the tensor and its n-rank. Although AIC criterion is a common method to find the optimal value of this parameter, reaching optimality is actually unnecessary in this specific case, as the objective is to make comparisons. We need a common n-rank across all three variables. We thus select n-rank = [2, 2, 2, 2], a good compromise between precision and speed. As a side note, n-rank = [5, 5, 5, 5] exceeded the capabilities of LAPACK (linear algebra routine under the hood), which shows how computationally expensive these methods are.
Once we have extracted the core tensor, we can make the same comparisons.
# Get core tensor for the parking lot videocore_parking_lot, factors_parking_lot = tucker(subset_parking_lot, ranks = [2,2,2,2])# Get core tensor for the patio videocore_patio, factors_patio = tucker(subset_patio, ranks = [2,2,2,2])# Get core tensor for the commute videocore_commute, factors_commute = tucker(subset_commute, ranks = [2,2,2,2])# Compare core parking lot and patioparking_patio_diff = tl.norm(core_parking_lot - core_patio)int(parking_patio_diff)# Compare core parking lot and commuteparking_commute_diff= tl.norm(core_parking_lot - core_commute)int(parking_commute_diff)# Compare core patio and commutepatio_commute_diff = tl.norm(core_patio - core_commute)int(patio_commute_diff)
Once again, displaying the results in a table:
These results make sense: while the patio video is as dissimilar to the parking lot and commute videos, the latter two are the closest by an order of magnitude.
In this article, I showed how unsupervised learning methods can provide insights into our data. Only after reducing the dimensionality through Tucker decomposition to extract core tensors from videos does their comparison make sense. We confirm that the parking lot and commute videos are the most similar.
As video is becoming an increasingly common data source, this technique has many potential applications. The first that comes to mind (due to my passion for TV and how video streaming services are using data) is to improve existing recommender systems by examining how trailers or some key scenes of movies/TV shows are similar to one another. A second is in material science, where infrared videos of heated metal could be categorized based on their similarity with a benchmark. In order for these methods to be fully scalable, computational cost should be addressed: Tucker decomposition was slow on my computer for three 10s videos. Parallelization is a potential approach to speed things up.
Beyond these immediate applications, this technique is also promising in combination with some of the methods presented in the introduction. Using the core tensor rather than the full image or video as a training point in a neural network could help address overfitting and speed up training, boosting the approach by addressing two of its main issues.
[1]: Kolda, Tamara G., and Brett W. Bader. “Tensor decompositions and applications.” SIAM review 51.3 (2009): 455–500.
[2]: Yan, Hao, Kamran Paynabar, and Massimo Pacella. “Structured Point Cloud Data Analysis via Regularized Tensor Regression for Process Modeling and Optimization.” Technometrics (2019): 1–19.
[3]: Fang, Xiaolei, Kamran Paynabar, and Nagi Gebraeel. “Image-based prognostics using penalized tensor regression.” Technometrics (2019): 1–29.