First 5 things to start learning PyTorch Tensors in Sagemaker Notebooks
By: Carlos E. Cortez Bazan
Tags: Deep Learning, Pytorch, Torch, AWS, Awscloud

Let’s begin our TorchAdventure in AWS! with this 11 basic functions distributed in the following sections:

Creating tensors: Empty() and Zeros()

Empty() and Zeros() Measuring tensors: Size() and Shape(), Sum() and Dimensions

Size() and Shape(), Sum() and Dimensions Changing and copying tensors: Reshape(), view() and randn()

Reshape(), view() and randn() Modifying tensor s: Unsqueeze()

s: Unsqueeze() Comparing tensors: Element-Wise Equality: Eq

First lets spin up a new sagemaker instance:

Now go to Jupyter Lab and import or create a new Notebook with this kernel for Pytorch:

Select the appropiate kernel, for this case it could be: conda_pytorch_p36

Here is where I will start from scratch.

Before we begin, let’s install and import PyTorch

# Uncomment and run the appropriate command for your operating system, if required

# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f # Linux / Binder# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f # Windows# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html # MacOS

# !pip install numpy torch torchvision torchaudio

Import Pytorch from the Notebook instance

import torch

Function 1 — Empty and Zeros — how to initialize tensors

We need to start working with the basics pytorch functions, and the first thing is create our matrix

# Creates a 3 x 2 matrix which is empty

a = torch.empty(3, 2)

print(a) tensor([[1.5842e-35, 0.0000e+00],

[4.4842e-44, 0.0000e+00],

[ nan, 0.0000e+00]])

Here is how PyTorch is allocating memory for this tensor. Whatever, it will not erase anything previous content in the memory.

by default, when you initializes a tensor is used the float32 dtype. you can review it here: https://pytorch.org/docs/stable/generated/torch.set_default_tensor_type.html#torch.set_default_tensor_type

But you can also start working with torch.zeros

# Applying the zeros function and

# storing the resulting tensor



a = torch.zeros([3, 5])

print("a = ", a)



b = torch.zeros([2, 4])

print("b = ", b)



c = torch.zeros([4, 1])

print("c = ", c)



d = torch.zeros([4, 4, 2])

print("d = ", d)

the result will be:

a = tensor([[0., 0., 0., 0., 0.],

[0., 0., 0., 0., 0.],

[0., 0., 0., 0., 0.]])

b = tensor([[0., 0., 0., 0.],

[0., 0., 0., 0.]])

c = tensor([[0.],

[0.],

[0.],

[0.]])

d = tensor([[[0., 0.],

[0., 0.],

[0., 0.],

[0., 0.]], [[0., 0.],

[0., 0.],

[0., 0.],

[0., 0.]], [[0., 0.],

[0., 0.],

[0., 0.],

[0., 0.]], [[0., 0.],

[0., 0.],

[0., 0.],

[0., 0.]]])

this tensor is filled with zeros, so PyTorch allocates memory and zero-initializes the tensor elements inside

You cannot change the way a tensor is created, if you create a zeros tensor, make sure is not referenced to any other.

# correctly initialized

a = torch.empty(3,3)

print(a) #also correct

b = torch.empty(1,2,3) print(b)

the results will be:

tensor([[2.4258e-35, 0.0000e+00, 1.5975e-43],

[1.3873e-43, 1.4574e-43, 6.4460e-44],

[1.4153e-43, 1.5274e-43, 1.5695e-43]])

tensor([[[2.3564e-35, 0.0000e+00, 1.4013e-45],

[1.4574e-43, 6.4460e-44, 1.4153e-43]]])

Lets see now, how you cannot use the zeros function:

# incorrect reference, you must create a new one

c = torch.zeros(b,1)

print("c = ", c)

output:

---------------------------------------------------------------------------

TypeError Traceback (most recent call last)

<ipython-input-35-c044e2995879> in <module>()

1 # incorrect reference, you must create a new one

----> 2 c = torch.zeros(b,1)

3 print("c = ", c) TypeError: zeros() received an invalid combination of arguments - got (Tensor, int), but expected one of:

* (tuple of ints size, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)

* (tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)

lets review the zero method closely:

Our documentation says:

Syntax: torch.zeros(size, out=None) Parameters:

size: a sequence of integers defining the shape of the output tensor

out (Tensor, optional): the output tensor Return type: A tensor filled with scalar value 0, of same shape as size.

torch.zeros and torch.empty are the first functions to start working with pytorch tensors and learning a little bit of matrix and vectors

Function 2 — Tensor Size, Shape and Dimension Operations

Lets understand dimensions in Pytorch.

Now lets create some tensors and determine the size of every one

import torch

# Create a tensor from data

c = torch.tensor([[3.2 , 1.6, 2], [1.3, 2.5 , 6.9]])

print(c) print(c.size())

output

tensor([[3.2000, 1.6000, 2.0000],

[1.3000, 2.5000, 6.9000]])

torch.Size([2, 3])

Lets see torch.shape and take a closer look at how size is given here

Now in the next example lets use shape functions to get the size of a tensor

In [ ]:

x = torch.tensor([

[1, 2, 3],

[4, 5, 6]

])

x.shape

torch.Size([2, 3])

Out[ ]:

torch.Size([2, 3])

Shape and Size give us the same correct dimensions of the tensor.

in this case we have a 3D-tensor (with 3 dimensions)

Dimension 0 Dimension 1 and Dimension 2

lets create a new tensor:

In [ ]:

y = torch.tensor([

[

[1, 2, 3],

[4, 5, 6]

],

[

[1, 2, 3],

[4, 5, 6]

],

[

[1, 2, 3],

[4, 5, 6]

]

]) y.shape

Out[ ]:

torch.Size([3, 2, 3])

Lets se how we can make operations using a 3d tensor now for every dimension layer and see how it behaves

In [ ]:

sum1 = torch.sum(y, dim=0)

print(sum1) sum2 = torch.sum(y, dim=1)

print(sum2) sum3 = torch.sum(y, dim=2)

print(sum3) tensor([[ 3, 6, 9],

[12, 15, 18]])

tensor([[5, 7, 9],

[5, 7, 9],

[5, 7, 9]])

tensor([[ 6, 15],

[ 6, 15],

[ 6, 15]])

we can see now a 3d tensor is more complicated as we advance, and we can perform custom operations within every dimension

its limited right now to 3 dims so we cannot perform this:

In []

sum2 = torch.sum(y, dim=3)

print(sum1)

Out [ ]:

---------------------------------------------------------------------------

IndexError Traceback (most recent call last)

<ipython-input-56-7caf723c6102> in <module>()

----> 1 sum2 = torch.sum(y, dim=3)

2 print(sum1) IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)

to accomplish adding more layers of dimensions we can review the unsqueeze functions → .unsqueeze() but for now let’s go to the next basic function

Function 3 — Reshape, View and Random

There is a function in NunPy called ndarray.reshape() for reshaping an array.

Now in pytorch, there is torch.view(tensor) for the same purpose, but at the same time, there is also a torch.reshape(tensor).

let’s figure out the differences between them and when you should use either of them.

First of all we are going to use a new functions for randomize our tensor.

In [ ]:

import torch

x = torch.randn(5, 3) print(x)

Out []

tensor([[-0.5793, 0.6999, 1.7417],

[-0.9810, 0.0626, 0.4100],

[-0.6519, -0.0595, -1.2156],

[-0.3973, -0.3103, 1.6253],

[ 0.2775, -0.0045, -0.2985]])

this is basic usage of torch.randm, so now lets use View from another variable “y” and describe all elements

In [ ]:

# Return a view of the x, but only having

# one dimension and max number of elements

y = x.view(5 * 3)



#lets see the size of every tensor

print("lets see the size of every tensor")

print('Size of x:', x.size())

print('Size of y:', y.size())



#and the elements of very tensor to compare

print("and the elements of very tensor to compare")

print("X:", x)

print("Y:", y)

Out []

lets see the size of every tensor

Size of x: torch.Size([5, 3])

Size of y: torch.Size([15])

and the elements of very tensor to compare

X: tensor([[-0.5793, 0.6999, 1.7417],

[-0.9810, 0.0626, 0.4100],

[-0.6519, -0.0595, -1.2156],

[-0.3973, -0.3103, 1.6253],

[ 0.2775, -0.0045, -0.2985]])

Y: tensor([-0.5793, 0.6999, 1.7417, -0.9810, 0.0626, 0.4100, -0.6519, -0.0595,

-1.2156, -0.3973, -0.3103, 1.6253, 0.2775, -0.0045, -0.2985])

take a look at Y tensor, it only has 1 dimension. so Viewing another tensor may be difficult for some operations

Now lets use Reshape to replicate the exact dimensions of X

In [ ]:

# Get back the original tensor with reshape()

z = y.reshape(5, 3)

print(z) tensor([[-0.2927, 0.0329, 0.8485],

[ 1.9581, 0.8313, -0.1529],

[-0.2330, -0.1887, 1.8206],

[ 1.5252, 1.0909, 0.0547],

[-0.1231, -0.4238, -0.6724]])

we cannot only reshape the original, we can also change the dimensions with some limited actions related to the maximum elements:

first lets reshape to 1 more dimension

In [ ]:

# Get back the original tensor with reshape()

z = y.reshape(15)

print(z) #reshaping 15 elements, 1 dim

z = y.reshape(3*5)

print(z) #reshaping in different order, 3 dimensions

z = y.reshape(3,5)

print(z) #Reshaping with more dimensions but its 15 elements always

z = y.reshape(3,5,1)

print(z) tensor([-0.5793, 0.6999, 1.7417, -0.9810, 0.0626, 0.4100, -0.6519, -0.0595,

-1.2156, -0.3973, -0.3103, 1.6253, 0.2775, -0.0045, -0.2985])

tensor([-0.5793, 0.6999, 1.7417, -0.9810, 0.0626, 0.4100, -0.6519, -0.0595,

-1.2156, -0.3973, -0.3103, 1.6253, 0.2775, -0.0045, -0.2985])

tensor([[-0.5793, 0.6999, 1.7417, -0.9810, 0.0626],

[ 0.4100, -0.6519, -0.0595, -1.2156, -0.3973],

[-0.3103, 1.6253, 0.2775, -0.0045, -0.2985]])

tensor([[[-0.5793],

[ 0.6999],

[ 1.7417],

[-0.9810],

[ 0.0626]], [[ 0.4100],

[-0.6519],

[-0.0595],

[-1.2156],

[-0.3973]], [[-0.3103],

[ 1.6253],

[ 0.2775],

[-0.0045],

[-0.2985]]])

Now lets reshape exceeding the number of elements in the tensor:

In [ ]:

z = y.reshape(16)

print(z)

Out []

---------------------------------------------------------------------------

RuntimeError Traceback (most recent call last)

<ipython-input-78-c7ae174fce73> in <module>()

----> 1 z = y.reshape(16)

2 print(z) RuntimeError: shape '[16]' is invalid for input of size 15

it will fail also for z = y.reshape(3*6) or putting more elements that does not exist in the tensor.

Now let’s keep going to the next section.

Function 4 — Unsqueeze()

Mainly, it allows us to add more dimensions at specific index you define.

lets take a look:

In [ ]:

import torch #dim=1, that is (3)

x = torch.tensor([1, 2, 3])

print('x: ', x)

print('x.size: ', x.size()) #x1 becomes a matrix of (3,1)

x1 = torch.unsqueeze(x, 1)

print('x1: ', x1)

print('x1.size: ', x1.size())

Out []

x: tensor([1, 2, 3])

x.size: torch.Size([3])

x1: tensor([[1],

[2],

[3]])

x1.size: torch.Size([3, 1])

x2: tensor([[1, 2, 3]])

x2.size: torch.Size([1, 3])

Our initial tensor is Tensor([1,2,3]), and the output size is [3].

And then we proceed with adding 1 dimenson with unsqueeze operation, namely torch.unsqueeze(x, 1), the size of x1 is [3,1].

In [ ]:

#x2 becomes a matrix of (1,3)

x2 = torch.unsqueeze(x, 0)

print('x2: ', x2)

print('x2.size: ', x2.size())

Out []

x2: tensor([[1, 2, 3]])

x2.size: torch.Size([1, 3])

When we perform torch.unsqueeze(x, 0), the size of x2 is [1,3].

In [ ]:

# Example 3 - breaking (to illustrate when it breaks)

print(x.unsqueeze())

Out []

---------------------------------------------------------------------------

TypeError Traceback (most recent call last)

<ipython-input-89-5a320a828907> in <module>()

2

3

----> 4 print(x.unsqueeze()) TypeError: unsqueeze() missing 1 required positional arguments: "dim"

We must specified the dimension correctly, although we are just adding 1 dim, it is necessary to put like this: x.unqueeze(0)

Function 5 — Torch Eq (Element Wise equality)

This function is under comparison category and it computes equality in element-wise and returns a boolean tensor. True if equal, False otherwise.

Lets review how we can operate with different sizes of tensors:

In [ ]:

# Example 1 - working

x1 = torch.tensor([[1, 2], [3, 4.]])

x2 = torch.tensor([[2, 2], [2, 5]])

x3 = torch.randn(3,5) #size x1 and z2

print(x1.size())

print(x2.size()) # tensors 1 and 2

print(x1)

print(x2)

#size x3

print(x3.size()) #tensors 3

print(x3) torch.eq(x1,x2) torch.Size([2, 2])

torch.Size([2, 2])

tensor([[1., 2.],

[3., 4.]])

tensor([[2, 2],

[2, 5]])

torch.Size([3, 5])

tensor([[-1.3040, -0.4658, -0.5269, 0.7409, 0.9135],

[ 1.0780, 2.0584, -0.9629, -1.1412, -0.3105],

[ 0.3613, -1.4196, 2.1145, 0.3649, 0.2037]])

Out[ ]:

tensor([[False, True],

[False, False]])

x1 and x3 have the same size, but x3 is [3,5], has bigger size.

comparing x1 and x2 is Ok.

In [ ]:

# Example 2 - working (with broadcasting)

x4 = torch.tensor([[1, 2], [3, 4]])

print(x4.size())

x5 = torch.tensor([2, 5])

print(x5.size())

torch.eq(x4, x5) torch.Size([2, 2])

torch.Size([2])

Out[ ]:

tensor([[False, False],

[False, False]])

we can also compare, different sizes only if the seconf value, in this case x5, that has size of [2] is broadcastable with the frist one thst is [2,2]

In [ ]:

# Example 3 - breaking (to illustrate when it breaks)

x6 = torch.tensor([[0, 2, 4], [3, 4, 5]])

print(x6.size())

x7 = torch.tensor([[2, 3], [2, 4]])

print(x7.size()) torch.eq(x6, x3) torch.Size([2, 3])

torch.Size([2, 2]) ---------------------------------------------------------------------------

RuntimeError Traceback (most recent call last)

<ipython-input-98-ac2ad1ecd5b0> in <module>()

5 print(x7.size())

6

----> 7 torch.eq(x6, x3) RuntimeError: The size of tensor a (3) must match the size of tensor b (5) at non-singleton dimension 1

finally, we cant compare different sizes if the second arguments shape is not broadcastable with the first argument.

Conclusion

we review 5 basic topics covering more than 10 PyTorch functions.

in the next post i’ll talk about Linear Regression.

Reference Links

Provide links to your references and other interesting articles about tensors

Official documentation for tensor operations: https://pytorch.org/docs/stable/torch.html

Squeeze and unsqueeze functions here: https://www.programmersought.com/article/65705814717/

Understanding dimensions in Pytorch: https://towardsdatascience.com/understanding-dimensions-in-pytorch-6edf9972d3be

#aws #reinvent2020 #awsperu #awsugperu #awscloud

Carlos Cortez — AWS UG Perú Leader / AWS ML Community Builder

ccortez@aws.pe

@ccortezb

Podcast: imperiocloud.com @imperiocloud

twitch.tv/awsugperu

cennticloud.thinkific.com