Quick ML Concepts: Tensors. Tensorflow, Tensorlab, Deep Tensorized... | by Enoch Kan | Towards Data Science

Tensorflow, Tensorlab, Deep Tensorized Networks, Tensorized LSTMs... it’s no surprise that the word “tensor” is embedded in the names of many machine learning technologies. But what are tensors? And how do they relate to machine learning? In part one of Quick ML Concepts, I aim to provide a short yet concise summary of what tensors are.
Don’t let the word “tensor” scare you. It is nothing more than a simple mathematical concept. Tensors are mathematical objects that generalize scalars, vectors and matrices to higher dimensions. If you are familiar with basic linear algebra, you should have no trouble understanding what tensors are. In short, a single-dimensional tensor can be represented as a vector. A two-dimensional tensor, as you may have guessed, can be represented as a matrix.
Even though it’s easy to generalize tensors as multi-dimensional matrices ranging from zero to N dimensions, it is important to remember that tensors are dynamic. That is, tensors will transform when interacting with other mathematical entities. Matrices, on the other hand, don’t always have this property. As quoted from Steven Steinke’s wonderful article about the fundamental differences between tensors and matrices: any rank-2 tensor can be represented as a matrix, but not every matrix is really a rank-2 tensor.
Tensor operations are simple. Consider the following tensors:
a = np.array([[[4,1,2],               [3,5,2],               [1,6,7]]               [[2,1,0],               [5,4,3],               [6,8,9]]               [[1,2,3],               [2,3,4],               [5,5,5]]])b = np.array([[[1,1,1],               [2,2,2],               [3,3,3]]               [[4,4,4],               [5,5,5],               [6,6,6]]               [[7,7,7],               [8,8,8],               [9,9,9]]])
It is very easy to see that both tensors are of rank 3. That is, they have 3 axes. You can check a tensor’s dimension by the following code in Python
a.ndim>>> 3
Addition of the above two tensors results in a tensor in which each scalar value is the element-wise addition of scalars in the parent tensors.
print(a+b)>>> [[[5,2,3],      [5,7,4],      [4,9,10]]      [[6,5,4],      [10,9,8],      [12,14,15]]      [[8,10,11],      [10,11,12],      [14,14,14]]]
If you are interested in learning more about tensor operations, Jason Brownlee’s excellent tensor tutorial has summarized a few commonly seen tensor operations including tensor dot and Hadamart products.
So how are tensors related to machine learning?
Remember, most machines cannot learn without having any data. And modern data is often multi-dimensional. Tensors can play an important role in ML by encoding multi-dimensional data. For example, a picture is generally represented by three fields: width, height and depth (color). It makes total sense to encode it as a 3D tensor. However, more than often we are dealing with tens of thousands of pictures. Hence this is where the forth field, sample size comes into play. A series of images, such as the famous MNIST dataset, can be easily stored in a 4D tensor in Tensorflow. This representation allows problems involving big data to be solved easily.
A real-life example would be in most recommender systems, model-based Collaborative Filtering approaches such as Matrix Factorization do not the flexibility of integrating context information into the models. By using a N-dimensional tensor instead of the traditional 2D User-Item matrix, researchers have developed a more robust model that provides context-aware recommendations. Increase in computing power in the recent years also allows these heavy tensor operations to be realized.
At this point you may wonder: why do most machine learning algorithms rely on vectors and matrices, while deep learning algorithms and neural networks mostly rely on tensors? A simple answer is that deep learning usually involves hundreds, if not thousands, of dimensions and fields. As we discussed previously, this is best represented by tensors since they can represent anything ranging from zero to N dimensions. In computer vision problems such as the Merck Molecular Activity Challenge, images can easily be broken down into a few hundred features. Therefore, tensors can be best viewed as containers that wrap and store data features in the context of machine learning and deep learning.
Thanks for reading my article.