Deep Learning: Introduction to Tensors & TensorFlow | by Victor Roman | Towards Data Science

The goal of this article is to cover the following topics:
Introduction to Tensors
Graphs, variables & operations
Problem resolution with TensorFlow
TensorFlow is a framework developed and maintained by Google that enables mathematical operations to be performed in an optimized way on a CPU or GPU. We are going to focus on the GPU since it is the fastest way we have to train a deep neural network.
Why Tensorflow?
Because of its flexibility and scalability
Because of its popularity
The key features that make TensorFlow the most popular Deep Learning library are:
TensorFlow uses tensors to perform the operations.
In TensorFlow, you first define the activities to be performed (build the graph), and then execute them (execute the graph). This allows the process to be optimized to the task at hand, reducing greatly the computation time.
TensorFlow enables code to be run in parallel or on one or more GPUs.
Okay, but what’s a tensor?
Although tensors were invented by physicists to be able to describe interactions, in the field of Artificial Intelligence they can be understood simply as containers of numbers.
Let’s put all this now into practice. We will use code some tensors in python in order to understand better what they are and how they work.
Let’s imagine that we want to store the average grade of a student. We will use a 0D tensor, which is just a simple number or scalar.
import numpy as nptensor_0D = np.array(5)print("Average grade: \n{}".format(tensor_0D))print("Tensor dimensions: \n{}".format(tensor_0D.ndim))
Let’s try now to store the grade of every subject that this student courses. We can use a 1D tensor to do so:
tensor_1D = np.array([4,6,8])print("Subject grades: \n{}".format(tensor_1D))print("Tensor dimensions: \n{}".format(tensor_0D.ndim))
But wait... What if we want to store each grade of every exam the student took on each subject? How could we do this if each subject had 3 exams?
By using a 2D tensor! Which, as we have seen before, is a matrix.
# 2D Tensor (matrix)tensor_2D = np.array([[0, 1, 1],  # Subject 1                      [2, 3, 3],  # Subject 2                      [1, 3, 2]])  # Subject 3print("Exam grades are:\n{}".format(tensor_2D))print("Subject 1:\n{}".format(tensor_2D[0]))print("Subject 2:\n{}".format(tensor_2D[1]))print("Subject 3:\n{}".format(tensor_2D[2]))print("Tensor dimensions: \n{}".format(tensor_2D.ndim))
We want now to store the grades of the subjects (which are annual) for four quarters, so it will be easier to access them if necessary in a future, don’t you think? How do you think we could organize them?
What if we add a dimension to our 2D tensor that indicates the quarter?
We would get a 3D tensor (3D matrix or cube).
tensor_3D = np.array([[[0, 1, 1],  # First quarter                      [2, 3, 3],                      [1, 3, 2]],                     [[1, 3, 2],  # Second quarter                      [2, 4, 2],                      [0, 1, 1]]])print("Exam grades per quarter are:\n{}".format(tensor_3D))print("First quarter:\n{}".format(tensor_3D[0]))print("Second quarter:\n{}".format(tensor_3D[1]))print("Tensor dimensions: \n{}".format(tensor_3D.ndim))
What if we add a dimension to our tensor, so that we can have the grades per term of each subject for each student?
It will be a 4D tensor (3D matrix vector or cube vector).
tensor_4D = np.array([[[[0, 1, 1], # Jacob                      [2, 3, 3],                      [1, 3, 2]],                     [[1, 3, 2],                      [2, 4, 2],                      [0, 1, 1]]],                      [[[0, 3, 1], # Christian                      [2, 4, 1],                      [1, 3, 2]],                     [[1, 1, 1],                      [2, 3, 4],                      [1, 3, 2]]],                     [[[2, 2, 4], # Sofia                      [2, 1, 3],                      [0, 4, 2]],                     [[2, 4, 1],                      [2, 3, 0],                      [1, 3, 3]]]])print("The grades of each student are:\n{}".format(tensor_4D))print("Jacob's grades:\n{}".format(tensor_4D[0]))print("Christian's grades:\n{}".format(tensor_4D[1]))print("Sofia's grades:\n{}".format(tensor_4D[2]))print("Tensor dimensions: \n{}".format(tensor_4D.ndim))
And so we could go on to infinity adding dimensions to our tensors, to be able to store more data.
To give you an idea of how often tensors are used in the world of Deep Learning, the most common types of tensors are:
3D tensors: used in time series.
4D-Tensors: used with images.
5D tensioners: used with videos.
Normally, one of the dimensions will be used to store the samples of each type of data. For example, with images:
If we want to store 64 RGB images of 224x224 pixels, we will need a 3D matrix vector, or what is the same, a 4D tensor. How many dimensions do we need?
We have 64 images of 224 pixels x 224 pixels x 3 channels (R, G and B).
Therefore: (64, 224, 224, 3)
If you want to go deeper into tensors or more examples, here is a very good resource for it: Tensors illustrated with cats
We said before that in TensorFlow, first you define the operations to be carried out and then you execute them. To do this, you use a graph.
And what is a graph?
A simple example of a sum of a + b.
And here is a more complex example, for now, you do not have to understand completely, but this is just a snippet of how intricate they can get.
First of all, we need to define and understand some basic concepts of TensorFlow (from now on TF):
tf.Graph: represent a set of tf.Operations
tf.Operation: Are the operations determined by the equations defined by us
tf.Tensor: Where we store the results of tf.Operations
In the beginning, tf.Graph is transparent to us, because there is one default graph where are added all our defined operations. It is called tf.get_default_graph().
# We import the Tensorflow package and matplotlib for the chartsimport tensorflow as tfimport matplotlib.pyplot as plt %matplotlib inline
Let’s start with something very simple, just a simple multiplication in Tensorflow.
# Variables definitionx = tf.constant(6)  y = tf.constant(8)# Operations definitionresult = tf.multiply(x, y)print(result)
As you can see, it didn’t give us back the result. What it has done so far is create the network.
To give you an example, it’s like riding in a car. Now we have it assembled, but it still doesn’t do what it was designed to do, move. To do that, we should turn it on. To do so, we will use tf.Session().
sess = tf.Session() output = sess.run(result)  print(output)
To be able to visualize the graph, we should define a couple of functions.
from IPython.display import clear_output, Image, display, HTMLdef strip_consts(graph_def, max_const_size=32):    """Strip large constant values from graph_def."""    strip_def = tf.GraphDef()    for n0 in graph_def.node:        n = strip_def.node.add()         n.MergeFrom(n0)        if n.op == 'Const':            tensor = n.attr['value'].tensor            size = len(tensor.tensor_content)            if size > max_const_size:                tensor.tensor_content = "<stripped %d bytes>"%size    return strip_defdef show_graph(graph_def, max_const_size=32):    """Visualize TensorFlow graph."""    if hasattr(graph_def, 'as_graph_def'):        graph_def = graph_def.as_graph_def()    strip_def = strip_consts(graph_def, max_const_size=max_const_size)    code = """        <script>          function load() {{            document.getElementById("{id}").pbtxt = {data};          }}        </script>        <link rel="import" href="https://tensorboard.appspot.com/tf-graph-basic.build.html" onload=load()>        <div style="height:600px">          <tf-graph-basic id="{id}"></tf-graph-basic>        </div>    """.format(data=repr(str(strip_def)), id='graph'+str(np.random.rand()))iframe = """        <iframe seamless style="width:1200px;height:620px;border:0" srcdoc="{}"></iframe>    """.format(code.replace('"', '"'))    display(HTML(iframe)
And now we will show the previously defined graph:
As we can see, the graph consist of 2 nodes of constant type and one of operator type (multiplication). However, their names are not quite indicatives. Let’s change this:
x2 = tf.constant(5.0, name='x2')y2 = tf.constant(6.0, name='y2')result = tf.multiply(x2, y2)# Let's see what it looks like now:show_graph(tf.get_default_graph().as_graph_def())
Once we have performed the operation, we should close the session to free the resources with sess.close().
Lastly, we can also indicate to TF the GPU that we want it to execute the operations. To do so, we can print a list of the available devices:
from tensorflow.python.client import device_libdef get_available_devices():    local_device_protos = device_lib.list_local_devices()    return [x.name for x in local_device_protos]print(get_available_devices())
We will select the GPU:0 and perform the multiplication [3 3] by [2 2], which may result in 3x3 +2x2 = 12. Let’s check it out:
By using the with ___ as ___: we make python to free the resources of the TF session by itself:
with tf.Session() as sess:  with tf.device("/GPU:0"):    matrix1 = tf.constant([[3., 3.]])    matrix2 = tf.constant([[2.],[2.]])    product = tf.matmul(matrix1, matrix2)      output = sess.run(product)  print(output)
Let’s create now a 1D Tensor of 32 values equally sparsed between -5 and 5:
n_values = 64x = tf.linspace(-5.0, 5.0, n_values)sess = tf.Session()result = sess.run(x)print(result)
In addition to sess.run(_), there are other ways to evaluate Tensors
x.eval(session=sess)
We need to always remember to close the session:
sess.close()
We can also use an interactive session, which can help us by not having to be constantly calling the .run() to have the results executed:
sess = tf.InteractiveSession()x.eval()
Let’s see now a tf.Operation, to do so, we will use “x” to create and visualize a gaussian distribution. Its formula is:
sigma = 1.0mean = 0# To implement the gaussian distributio formula:g1d = (tf.exp(tf.negative(tf.pow(x - mean, 2.0) / (2.0 * tf.pow(sigma, 2.0)))) *     (1.0 / (sigma * tf.sqrt(2.0 * 3.1415))))# to check that this operation has been succesuflly included in our tf.Graphif g1d.graph is tf.get_default_graph():  print('All good')
plt.plot(g1d.eval())
# to see the dimensionsprint(g1d.get_shape())print(type(g1d.get_shape()))print(g1d.get_shape().as_list())print(type(g1d.get_shape().as_list()))
There will be times when we don’t know the dimensions of a variable until the operation that returns its value is executed. For these cases, we can use tf.shape(variable), which returns a Tensor that will calculate at run time the dimensions of our result.
This is known as ‘static shape’ and ‘dynamic shape’, where static is calculated taking into account the dimensions of the Tensors and Operations involved, and the dynamics at execution time.
What happens if we define x as a ‘placeholder’? A placeholder is like a reserve, it indicates that there will be a tensor there, but it is not necessary to define it at that moment. For example, by defining:
x = tf.placeholder(tf.int32, shape=[5])
We know that x will hold a 1D 5 dimensional tensor, as confirmed by x.get_shape():
print(x.get_shape())
But we don’t know what values will form it until we tell it.
Differences between placeholder and variable:
Variables
They are used to accommodate parameters that are learned during training.
Therefore, the values are derived from training — They require an initial value to be assigned (can be random)
Placeholders
Reserve space for data (e.g. for the pixels in an image)
They do not require a value to be assigned to start (although they can)
The obtained value, 5, it is the static value of the x dimensions. But, what happens if we apply a tf.unique() to x?
y, _ = tf.unique(x)print(y.get_shape())
What happens is that tf.unique() returns the unique values of x, which are not known at first, since x is defined as a placeholder, and a placeholder does not have to be defined until the moment of execution, as we said before. In fact, let’s see what happens if we feed “x” with two different values:
with tf.Session() as sess: print(sess.run(y, feed_dict={x: [0, 1, 2, 3, 4]}).shape) print(sess.run(y, feed_dict={x: [0, 0, 0, 0, 1]}).shape)
Look at that! The size of y changes depending on what tf.unique() returns. This is called “dynamic shape”, and it is always defined, it will never return a question by answer. Because of this, TensorFlow supports operations like tf.unique() which can have variable size results.
So, now you know, every time that you use operations with output variables, you will need to use tf.shape(variable) to calculate the dynamic shape of a tensor.
sy = tf.shape(y)# Returns a list with the dimensionsprint(sy.eval(feed_dict={x: [0, 1, 2, 3, 4]}))print(sy.eval(feed_dict={x: [0, 0, 0, 0, 1]}))# We access the dimension of interestprint(sy.eval(feed_dict={x: [0, 1, 2, 3, 4]})[0])print(sy.eval(feed_dict={x: [0, 0, 0, 0, 1]})[0])
Now we can perform the operations after taking into account the size of the output of the operation, which we do not know in the first instance.
print(tf.shape(y).eval(feed_dict={x: [0, 1, 4, 1, 0]}))print(type(tf.shape(y).eval(feed_dict={x: [0, 1, 4, 1, 0]})))print(tf.stack([y, y[::-1], tf.range(tf.shape(y)[0])]).eval(feed_dict={x: [0, 1, 4, 1, 0]}))
Let’s see now a Gaussian distribution in 2D
g1d_r = tf.reshape(g1d, [n_values, 1])print(g1d.get_shape().as_list())print(g1d_r.get_shape().as_list())# We multiply the row vector of the 1D Gaussian by the column to obtain the 2D versiong2d = tf.matmul(tf.reshape(g1d, [n_values, 1]), tf.reshape(g1d, [1, n_values]))# To visualize itplt.imshow(g2d.eval())
To see the list of the operations included in our tf.Graph
ops = tf.get_default_graph().get_operations()print([op.name for op in ops])
As always, I hope you enjoyed the post, and that you have learned the basics of Tensors and TensorFlow and how they are used.
If you liked this post then you can take a look at my other posts on Data Science and Machine Learning here .
If you want to learn more about Machine Learning and Artificial Intelligence follow me on Medium, and stay tuned for my next posts!