I am working on the file "Enums.ipynb".
The first task is:
Nomenclature
Python has a specific nomenclature for enums.
The class Color is an enumeration (or enum)
The attributes Color.red, Color.green, etc., are enumeration members (or enum members).
The enum members have names and values (the name of Color.red is red, the value of Color.blue is 3, etc.)
Printing and Representing Enums
Enum types have human readable string representations for print and repr:
Can you write Python code for it?


I am working on the file "Enums.ipynb".
The first task is:
The type of an enumeration member is the enumeration it belongs to:
Can you write Python code for it?


I am working on the file "Envolvente.ipynb".
The first task is:
Luego de hacer la importación de las librerías que se van a utilizar, en la función main_eos() definida por un usuario se realiza la especificación de la sustancia pura junto con el modelo de ecuación de estado y parámetros que se requieren en la función "pt.function_elv(components, Vc, Tc, Pc, omega, k, d1)" que realiza los cálculos del algoritmo que se describió previamente.
Can you write Python code for it?

def main_eos():
    print("-" * 79)
    components = ["METHANE"]
    MODEL = "PR"
    specification = "constants"
    component_eos = pt.parameters_eos_constans(components, MODEL, specification)
    #print(component_eos)
    #print('-' * 79)

    methane = component_eos[component_eos.index==components]
    #print(methane)
    methane_elv = methane[["Tc", "Pc", "k", "d1"]]
    #print(methane_elv)

    Tc = np.array(methane["Tc"])
    Pc = np.array(methane["Pc"])
    Vc = np.array(methane["Vc"])
    omega = np.array(methane["Omega"])
    k = np.array(methane["k"])
    d1 = np.array(methane["d1"])

    punto_critico = np.array([Pc, Vc])

    print("Tc main = ", Tc)
    print("Pc main = ", Pc)
    print("punto critico = ", punto_critico)

    data_elv = pt.function_elv(components, Vc, Tc, Pc, omega, k, d1)
    #print(data_elv)

    return data_elv, Vc, Pc


I am working on the file "Equation_Notes2.ipynb".
The first task is:

Can you write Python code for it?

#!pip3 install torch torchvision
import torch

# Requires_grad=True turns on differential mode
a=torch.tensor(([1.0]),requires_grad=True)
b=torch.tensor(([2.0]),requires_grad=True)
c=torch.tensor(([3.0]),requires_grad=True)

d=a+b
e=d+c
e.backward()
print(a.grad,b.grad,c.grad)

print(d.grad) # Intermediate gradient value is not saved, and is empty

print(a.grad_fn) # The first node's .grad_fn is empty
print(e.grad_fn)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
.shape is an alias for .size()
Converting Tensors of Pandas, Numpy
Can you write Python code for it?

import torch
import numpy as np
import pandas as pd

nparray = np.array([0.0,1.0,2.0,3.0,4.0])
print("\n Original Array in numpy: ",nparray)

torcharray = torch.from_numpy(nparray)
print("\n Torch Array from numpy: ",torcharray)

back_to_nparray = torcharray.numpy()
print("\n Numpy array converted back from torch: ",back_to_nparray)

pdarray = pd.Series([19.0,0.1,5.8,2.3])
print("\n Original Array in Pandas: \n",pdarray)

ptorcharray = torch.from_numpy(pdarray.values) #pdarray.values returns a numpy array
print("\n Torch Array from numpy: ",ptorcharray)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Accessing Parameters and Modules
Can you write Python code for it?

seq_net = nn.Sequential(
	nn.Linear(2,4),
    nn.Tanh(),
    nn.Linear(4,1)
)

print(seq_net[0])
print(seq_net[1])
print(seq_net[0].weight)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Additional operators
Can you write Python code for it?

a = torch.randn(4)
print(a)
torch.clamp(a, min=0, max=1) # If less than zero make it zero, if greater than make it 1 and leave rest as is

a = torch.tensor([[2,4,3],[5,6,1]])
b = torch.tensor([[10,11,10],[10,2,8]])
torch.mul(a, b) # Each element of the tensor input is multiplied by each element of the Tensor other, must be broadcastable

torch.dot(torch.tensor([2, 3]), torch.tensor([2, 1])) # This function does not broadcast.

mat1 = torch.randn(2, 3)
mat2 = torch.randn(3, 3)
print(torch.mm(mat1, mat2)) # matrix multiplication This function does not broadcast. For broadcasting matrix products, use torch.matmul().

print(mat1 @ mat2) # A @ B is the matrix product

mat1 = torch.randn(3, 3)
mat2 = torch.randn(3, 3)
print(mat1 * mat2) # A * B the element-wise product


I am working on the file "Equation_Notes2.ipynb".
The first task is:
After the backward call is called, the derivative is automatically calculated.
Because the automatic derivation is done twice, and old gradients are accumulated, so the first gradient 6 and the second gradient 6 are added to get 12.
In the example below we do x.grad.zero_() before the graph is reconstructed.
Can you write Python code for it?

#x = torch.tensor(2.0, requires_grad=True)
x.grad.zero_()                            # Zero the gradients before reconstructing the graph
z = x**2 + 2*x+1                          # Reconstruct the graph
z.backward()                              # Use chain rule to calculate the gradient
print(x.grad)
print(x)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
All of the options (o1, o2, o3, o4) appear to have produced the same tensors except for the first one.
The first option (o1) has dots after the number indicating that the numbers are floats, while the next three options have a type of int64.
The difference here arises in the fact that the torch.Tensor() constructor uses the default dtype when building the tensor.
The other calls choose a dtype based on the incoming data. This is called type inference. The dtype is inferred based on the incoming data. Note that the dtype can also be explicitly set for these calls by specifying the dtype as an argument:
Can you write Python code for it?

print(torch.tensor(datanp, dtype=torch.float32))
print(torch.as_tensor(datanp, dtype=torch.float32))


I am working on the file "Equation_Notes2.ipynb".
The first task is:
As long as you use retain_graph=True in your backward method, you can do backward any time you want.
Can you write Python code for it?

d.backward(retain_graph=True) # fine....graph is retained so no need to reconstruct again
e.backward(retain_graph=True) # fine....graph is retained so no need to reconstruct again
d.backward() # also fine
e.backward() # error will occur!


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Axes of a tensor
If we have a tensor, and we want to refer to a specific dimension, we use the word axis in deep learning.
An axis of a tensor is a specific dimension of a tensor.
If we say that a tensor is a rank 2 tensor, we mean that the tensor has 2 dimensions, or equivalently, the tensor has two axes.
Elements are said to exist or run along an axis. This running is constrained by the length of each axis.
The length of each axis tells us how many indexes are available along each axis.
Can you write Python code for it?

dd = torch.tensor([
[1,2,3],
[4,5,6],
[7,8,9]])

#Each element along the first axis, is an array:
print(dd[0])
print(dd[1])
print(dd[2])

#Each element along the second axis, is a number:
print(dd[0][0])
print(dd[1][0])
print(dd[2][0])
print(dd[0][1])
print(dd[1][1])
print(dd[2][1])
print(dd[0][2])
print(dd[1][2])
print(dd[2][2])


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Broadcasting of Tensors
Tensors of different dimensionality can be combined in the same expression. Tensors with smaller dimension are broadcasted to match the larger tensors, without copying data. Broadcasting has two rules.

Rule 1:  For broadcasting to be successful, the dimensions of the tensors need to be compatible. Two dimensions are compatible when they are equal

Rule 2: Two dimensions of different tensors are also compatible when one of the dimensions is 1. The resulting tensor size is actually the maximum size along each dimension of the input tensors. In other words, given x(3,4) and y (4,) the maximum size along each dimension of x and y is taken to make up the shape of the new, resulting tensor is of shape (3,4)
Can you write Python code for it?

x = torch.ones((3,4))
y = torch.arange(4,dtype=torch.float32)
print(x)
print(y)
print("\n")
print(x-y)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Checkpoint saving
When saving using a general checkpoint (to be used for either inference or resuming training) we must save more than just the model’s state_dict. It is important to also save the optimizer’s state_dict, as this contains buffers and parameters that are updated as the model trains.
Other items that we may want to save are the epoch we left off on, the latest recorded training loss, external torch.nn.Embedding layers, etc.
To save multiple components, organize them in a dictionary and use torch.save() to serialize the dictionary. A common PyTorch convention is to save these checkpoints using the .tar file extension.
To load the items, first initialize the model and optimizer, then load the dictionary locally using torch.load(). From here, you can easily access the saved items by simply querying the dictionary as you would expect.
Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results. If you wish to resuming training, call model.train() to ensure these layers are in training mode.
Can you write Python code for it?

# Checkpoint save
torch.save({
            'epoch': epoch,
            'model_state_dict': mymodel.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
            ...
            }, PATH)

# Checkpoint load
model = TheModelClass(*args, **kwargs)
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

model.eval()
# - or -
model.train()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Data Access

trainset[0] where first element is image and second element is label (tuple)
trainset[0][0] is image
trainset[0][1] is class of image (torch.LongTensor)
Can you write Python code for it?

# Print out the tuple (image,label)
#print(trainset[0])

# Print out the first image
#print("\n",trainset[0][0])

# Print out first label or class
print(trainset[0][1])

print(trainset[0][0].type())
print(trainset[0][1].type())


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Data Visualization
Can you write Python code for it?

import matplotlib.pylab as plt
import numpy as np

print(trainset[3][0].shape)

plt.imshow(trainset[3][0].numpy().reshape(28,28),cmap='gray')
print(trainset[3][1].item())
plt.title('y= '+ str(trainset[3][1].item()))

#print(trainset[30])


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Debugging
With its clean and minimal design, PyTorch makes debugging a breeze. We can place breakpoints using pdb.set_trace() at any line in our code. We can then execute further computations, examine the PyTorch Tensors/Variables and pinpoint the root cause of the error.
Inplace Operations
torch.nn.functional.relu(input, inplace=False) takes a tensor of any size as input, applies ReLU on each value to produce a result tensor of same size.
inplace indicates if the operation should modify the argument itself.  This may be desirable to reduce the memory footprint of the processing.
Wrap-up

Inherit a class from torch.nn.module - initialize: define modules, forward: build network
Create an object (instance) - Construct graphs when class in initialized and forward is called
Define optimizer and loss function
Run loop - reset gradient, backward, update step

Multi-processing:
torch.nn.dataparallel
torch.multiprocessing
hogwild(async)
Homework
Solve the previous celsius to fahrenheit problem using pytorch classes and functions. Hook to the backpropogation for each iteration and plot the change of values.
Finally, Save the model, load the model and run in evaluation mode and evalute the performance of the model for a given sample.
Check if a gpu is present and use gpu for accelerating the entire model and training.
ANNs for Images
Image Theory
Pixels are the raw building blocks of an image. Every image consists of a set of pixels. There is no finer granularity than the pixel. Normally, a pixel is considered the “color” or the “intensity” of light that appears in a given place in our image.
Most pixels are represented in two ways:

Grayscale/single channel
Color - 3 Channel (RGB)

In a grayscale image, each pixel is a scalar value between 0 and 255, where zero corresponds to “black” and 255 being “white”. Values between 0 and 255 are varying shades of gray, where values closer to 0 are darker and values closer to 255 are lighter.

Grayscale = Image Batch x Channel x Height x Weight  = Image Batch x 1 x Height...
Can you write Python code for it?

from torch.utils.data import Dataset

class c2fdata(Dataset):
  def __init__(self):
    self.celsius = torch.tensor([(float)(c) for c in range(-273,1000)])
    self.fahrenheit = torch.tensor([c*1.8+32.0 for c in self.celsius])

  def __getitem__(self,index):
    return self.celsius[index],self.fahrenheit[index]

  def __len__(self):
    return self.celsius.shape[0]

mydataset = c2fdata()

print(mydataset[0])
print(len(mydataset))


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Every torch.Tensor has these attributes:

torch.dtype
torch.device
torch.layout

Looking at our Tensor a, we can see the following default attribute values:
Can you write Python code for it?


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Flatten a tensor
A flatten operation on a tensor reshapes the tensor to have a shape that is equal to the number of elements contained in the tensor. This is the same thing as a 1d-array of elements.
Flattening a tensor means to remove all of the dimensions except for one.
Can you write Python code for it?

f = torch.ones(4, 3)
print(f)
print(torch.flatten(f)) #https://pytorch.org/docs/master/torch.html

print("\n")
f1 = torch.tensor([[[1, 2],
                   [3, 4]],
                   [[5, 6],
                   [7, 8]]])
print("Original Array: \n",f1)

print("\n")
print(torch.flatten(f1))

print(torch.flatten(f1, start_dim=1))


I am working on the file "Equation_Notes2.ipynb".
The first task is:
For varying dimensional matrices, we can clearly see the gradients of z are computed w.r.t to each dimension of x, because the operations are all element-wise.
Can you write Python code for it?

x = torch.randn(2, 2, requires_grad=True) #x is a leaf created by user, thus grad_fn is none
# Each tensor has a .grad_fn attribute that references a "Function" that has created the tensor (except for tensors created by the user – their grad_fn is None).

print('x', x)

y = 2 * x  # define an operation on x and construct the graph
z = y ** 3 # define one more operation to check the chain rule and continue constructing the graph

print('z shape:', z.size())
z.backward(torch.FloatTensor([[1, 1], [1, 1]]))
print('x gradient for its all elements:\n', x.grad)

x.grad.zero_()
x.grad.data.zero_() #the gradient for x will be accumulated, it needs to be cleared.
y = 2 * x
z = y ** 3
z.backward(torch.FloatTensor([[0, 1], [0, 1]]))
print('x gradient for the second column:\n', x.grad)

x.grad.zero_()
x.grad.data.zero_()
y = 2 * x
z = y ** 3
z.backward(torch.FloatTensor([[1, 1], [0, 0]]))
print('x gradient for the first row:\n', x.grad)

x = torch.randn(2, 2, requires_grad=True) #x is a leaf created by user, thus grad_fn is none
print('x', x)

y = 2 * x #define an operation on x
print('y', y)
z = y ** 3 #define one more operation to check the chain rule

out = z.mean()
print('out', out)
out.backward()
print('x gradient:\n', x.grad)

x.grad.data.zero_()
y = 2 * x
z = y ** 3
out = z.mean()
#out.backward(torch.FloatTensor([[1, 1], [1, 1]]))
out.backward(torch.ones(z.size())) #Note the use of size() function of the output vector
print('x gradient second time', x.grad)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Forward and Backward Hooks
For inspecting / modifying the output and grad_output of a layer during forward or backward propogation we introduce hooks.
A hook is a function that can be registered on a module or a tensor.
The hook can be a forward hook or a backward hook.
The register_forward_hook & register_backward_hook functions are similar to the register_hook variable function, which can be register_hook when the module propagates forward or backpropagates.
A forward hook function is executed each time the forward propagation execution ends.
The forward-propagating hook function has the following form:
hook(module, input, output) -> None,
and backpropagation has the following form:
hook(module, grad_input, grad_output) -> Tensor or None .
The hook function should not modify the input and output, and should be deleted in time after use to avoid running the hook every time which can increase the running load.
Hook functions are mainly used in scenarios where some intermediate results are obtained, such as the output of a layer in the middle or the gradient of a layer. These results should have been written in the forward function, but if you add these processing to the forward function, it may make the processing logic more complicated. At this time, it is more appropriate to use the hook technique.
Let's consider a scenario. There is a pre-trained model that needs to extract the output of a layer (not the last layer) of the model as a feature, but does not want to modify its original model definition file. You can use the hook function. The code of the implementation is given below.
PyTorch recursively applies any hook to all submodules on a register_forward_hook() call.
Can you write Python code for it?

#!pip3 install torch torchvision

import torch

import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt

x = torch.randn(1, 2).cuda()
y = torch.randn(1, 1).cuda()

class net(torch.nn.Module):
  def __init__(self):
    super(net,self).__init__()
    self.linear1=torch.nn.Linear(2,1)
    self.linear2=torch.nn.Linear(1,2)

  def forward(self,x):
    h_relu = torch.nn.functional.relu(self.linear1(x))
    y_pred = self.linear2(h_relu)
    return y_pred

def forwardcall(module,input,output):
  print("\nInside forward of ", module," : ",input," : ", output)

def backwardcall(module, grad_input, grad_output):
  print("Inside backward of ", module," : ",grad_input," : ", grad_output)

mymodel = net()

print(mymodel)
hook1 = mymodel.linear1.register_forward_hook(forwardcall)
hook2 = mymodel.linear2.register_backward_hook(backwardcall)

if torch.cuda.is_available():
  mymodel.cuda()

optimizer = torch.optim.SGD(mymodel.parameters(), lr=0.0009, momentum=0.9)

loss_col = []
epochs = 5
for epoch in range(epochs):
  optimizer.zero_grad()
  model_output = mymodel(x)
  loss = torch.nn.functional.mse_loss(model_output, y)
  loss_col.append(loss)
  #print(epoch, "{:.20f}".format(loss.item()))
  loss.backward()
  optimizer.step()

hook1.remove()  # removes the hook
hook2.remove()  # removes the hook


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Gradient Descent and Optimizer
We create a stochastic gradient descent optimizer, and we specify the learning rate (which we’ve passed to the function as 0.01) and a momentum of 0.9. The other ingredient we need to supply to our optimizer is all the parameters of our network – thankfully PyTorch make supplying these parameters easy by the .parameters() method of the base nn.Module class that we inherit from in the mymnist class.
Next, we set our loss criterion to be the negative log likelihood loss – this combined with our log softmax output from the neural network gives us an equivalent cross entropy loss for our 10 classification classes.
Can you write Python code for it?

# create a stochastic gradient descent optimizer
optimizer = torch.optim.SGD(net1.parameters(), lr=0.01, momentum=0.9)

# create a loss function
criterion = nn.NLLLoss().cuda()

#optimizer = torch.optim.Adam(net1.parameters(), lr=0.01)
#criterion = nn.CrossEntropyLoss()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Gradients are accumulated with every cycle which allow us to get the correct gradient for all the computations with a given variable.
Thus we may need to manually reset the values to 0 so that the gradients computed previously do not interfere with the ones we are currently computing.
Can you write Python code for it?

#x = torch.tensor(2.0, requires_grad=True)
z = x**2 + 2*x+1                          # Reconstruct the graph
z.backward()                              # Use chain rule to calculate the gradient
print(x.grad)
print(x)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
However, it would be much more convenient if the dataset implemented the iterator protocol itself, so we could simply loop over samples with for sample in dataset.
We are losing a lot of features by using a simple for loop to iterate over the data. In particular, we are missing out on:
Batching the data
Shuffling the data
Load the data in parallel using multiprocessing workers.

Fortunately, this functionality is provided by the DataLoader class available via the torch.utils.data.DataLoader which is an iterator which provides all these features.
A DataLoader object takes a dataset object as argument and a number of options that configure the way samples are retrieved. For example, it is possible to load samples in parallel, using multiple processes. For this, the DataLoader constructor takes a num_workers argument.
Note that DataLoaders always return batches, whose size is set with the batch_size parameter.
List of important arguments for dataloader:
$$\begin{array}{|c|c|}
\hline
 Argument & Description \\hline
dataset \;(Dataset) & dataset\; object \;from\; which\; to\; load\; the\; data\\hline
batch_size (int, optional) & how \;many \;samples \;per \;batch \;to \;load (default: 1)\\hline
shuffle (bool, optional) & set \;to\; True\; to\; have\; the\; data\; reshuffled\; at\; every\; epoch\; (default: False).\\hline
num_workers (int, optional) &  how\; many\; subprocesses\; to\; use\; for\; data\; loading.\; 0\; means\; that\; the\; data\; will\; be\; loaded\; in\; the\; main\; process (default: 0)\\hline
collate_fn (callable, optional) & specify\; how\; exactly\; the\; samples\; need\; to\; be\; batched\; to\; form\; a\; mini-batch\;\\hline
\end{array}$$
Can you write Python code for it?

mydataloader = torch.utils.data.DataLoader(mydataset,batch_size=6,num_workers=2)
for i, batch in enumerate(mydataloader):
  print(i, batch)

mydataloader = torch.utils.data.DataLoader(mydataset,batch_size=6, shuffle=True, num_workers=2, drop_last=True)
for i, batch in enumerate(mydataloader):
  print(i, batch)

import math

class RangeDataset(torch.utils.data.Dataset):
  def __init__(self, start, end, step=1):
    self.start = start
    self.end = end
    self.step = step
  def __getitem__(self, index):
    value = self.start + index * self.step
    assert value < self.end
    return value
  def __len__(self):
    return math.ceil((self.end - self.start) / self.step)

dataset = RangeDataset(0, 10)
data_loader = torch.utils.data.DataLoader(dataset, batch_size=4, shuffle=True, num_workers=2, drop_last=True)

for i, batch in enumerate(data_loader):
  print(i, batch)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
If we don’t specify an axis to the argmax() method, it returns the index location of the max value from the flattened tensor, which in this case is indeed 11.
Argmax along Specific axes
Notice how the call to the max() method returns two tensors. The first tensor contains the max values and the second tensor contains the index locations for the max values along dim0.
The max values are determined by taking the element-wise maximum across each array running across the first axis.
Can you write Python code for it?

print(xa.max(dim=0))
print(xa.argmax(dim=0))
print(xa.max(dim=1))
print(xa.argmax(dim=1))


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Initializing parameters of the model

We can loop over the parameters of the module using model.parameters() and then initializing each with tensor functions such as exponential, uniform, fill etc.

Every module have an attribute definition .apply . We can call this apply on the module and pass it a function which handles the initialization for each of the parameter. Whenever .apply is called on a module, it is called on each of the module and parameter recursively.

We could use torch.nn.init module to initialize our parameters in more practical way. Suppose we have a parameter named m, and we need to initialize it using Xavier (Glorot) initialization, then we can do torch.nn.init.xavier_uniform(m) . Now, combining this with .apply we can use the module torch.nn.init to initialize parameters of all sort.

Applies function recursively to every submodule (as returned by .children()) as well as self. Typical use case includes initializing the parameters of a model (we also use torch-nn-init).
Can you write Python code for it?

import torch.nn as nn
def init_weights(m):
        print(m)
        if type(m) == nn.Linear:
            #m.weight.data.fill_(1.0)
            nn.init.xavier_uniform_(m.weight.data)
            print(m.weight.data)

net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
net.apply(init_weights)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Inside init we would usually configure some paths or change the set of samples ultimately returned. In len, we specify the upper bound for the index with which getitem may be called, and in getitem we return the actual sample, which could be an image or an audio snippet.
To iterate over the dataset we could, in theory, simply have a for i in range loop and access samples via getitem.
Can you write Python code for it?

for i in range(len(mydataset)):
  print(mydataset[i])
  if i == 3:
        break


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Mathematical operations on tensors can be performed element wise, and flat arrays can be returned which can used as masks.
Can you write Python code for it?

print(fancy>78)
print(fancy[fancy>78])

print(fancy.gt(78))


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Most times we deal with batches of color images and we don’t want to flatten the whole tensor. We only want to flatten the image tensors within the batch tensor leaving the batch dimension intact
$$(Batch Size,\; Channels,\; Height,\; Width) $$
We could thus do $flatten(start_dim=1)$ which tells the flatten() method which axis it should start the flatten operation.
The one here is an index, so it’s the second axis which is the color channel axis. We skip over the batch axis so to speak, leaving it intact.
Concatenating tensors
Can you write Python code for it?

t1 = torch.tensor([
    [1,2],
    [3,4]
])
t2 = torch.tensor([
    [5,6],
    [7,8]
])

# We can combine t1 and t2 row-wise (axis-0) in the following way:
print(torch.cat((t1, t2), dim=0))

# We can combine them column-wise (axis-1) like this:
print(torch.cat((t1, t2), dim=1))

print(torch.cat((t1, t2), dim=0).shape)

print(torch.cat((t1, t2), dim=1).shape)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Move to GPU
The main idea behind PyTorch’s GPU handling is that if an object (tensor, Variable, Parameter) is available on a device with id 0 , then after performing some operation (transformation) on it, the resultant object will also be stored on the same device 0. Further, operations cannot be performed on two objects residing on different devices, i.e., no a+b is allowed if a is on CPU and b is on GPU.
So, in order to run our training on GPU, it is enough to copy our model parameters and our input data on the GPU. This is done using .cuda() attribute available for all tensors and modules. For tensor, it is straightforward that this will copy the tensor to GPU, but for module (model), calling .cuda() will recursively copy all child modules and parameters to GPU.
We can verify if torch.cuda.is_available() returns True to check if a GPU is available on a machine.
cuda(device=None)
Access individual layer weights and gradients
Can you write Python code for it?

import torch

import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt

x = torch.randn(1, 2).cuda()
y = torch.randn(1, 1).cuda()

class net(torch.nn.Module):
  def __init__(self):
    super(net,self).__init__()
    self.linear1=torch.nn.Linear(2,1)
    self.linear2=torch.nn.Linear(1,2)

  def forward(self,x):
    h_relu = torch.nn.functional.relu(self.linear1(x))
    y_pred = self.linear2(h_relu)
    return y_pred

mymodel = net()

if torch.cuda.is_available():
  mymodel.cuda()

optimizer = torch.optim.SGD(mymodel.parameters(), lr=0.0009, momentum=0.9)

# Before training
print("predict (before training)", mymodel(x).data[0])

loss_col = []
epochs = 5
for epoch in range(epochs):
  optimizer.zero_grad()
  model_output = mymodel(x)
  loss = torch.nn.functional.mse_loss(model_output, y) # direct use of mse_loss from nn.functional
  loss_col.append(loss)
  #print(epoch, "{:.20f}".format(loss.item()))
  loss.backward()
  optimizer.step()
  print("progress:", epoch, loss.data.item())

plt.plot(loss_col)
plt.show()

#print(list(mymodel.parameters()))

# After training
hour_var = torch.Tensor([1.0,4.0]).cuda()
y_pred = mymodel(hour_var)
print("predict (after training)", y_pred.data)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Move to cpu
Moves all model parameters and buffers to the CPU
Can you write Python code for it?


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Multi-Dimensional Tensor Inputs
If we want to compute the derivatives, we call .backward() on a tensor.
If tensor is a scalar (i.e. it holds a one element data), we don’t need to specify any arguments to backward(), however if it has more elements, you need to specify a grad_output argument that is a tensor of matching shape.
I.e, if the Tensor contains one element, you don’t have to specify any parameters for the backward() function. If the Tensor contains more than one element, specify a gradient that’s a tensor of matching shape.
Basically to start the chain rule we need a gradient at the output, to get it going. In the event the output is a scalar loss function (which it usually is - normally we are beginning the backward pass at the loss variable ), its an implied value of 1.0
The function y.backward() is equivalent to doing  y.backward(torch.Tensor([1.0]))
Example3:
$$ f(x,y) = z =x^2 \cdot y$$
computing partial derivatives looks something like this:
$$\frac {\partial f}{\partial x} = \frac {\partial}{\partial x}x^2 y = 2 x y$$
$$\frac {\partial f}{\partial y} = \frac {\partial}{\partial y}x^2 y = x^2\cdot 1$$
Can you write Python code for it?

#!pip3 install torch torchvision
import torch
x = torch.tensor([1.0, 2.0, 3.0, 4.0], requires_grad=True)
y = torch.tensor([5.0, 6.0, 7.0, 8.0], requires_grad=True)
z = (x**2) * y
print(z)

z.backward(torch.FloatTensor([1, 0, 0, 0])) # do backward for first element of z
#z.backward()
print("The derivative of z_1 w.r.t to x: ",x.grad.data)
print("The derivative of z_1 w.r.t to y: ",y.grad.data)

x = torch.tensor([1.0, 2.0, 3.0, 4.0], requires_grad=True)
y = torch.tensor([5.0, 6.0, 7.0, 8.0], requires_grad=True)
z = (x**2) * y
z.backward(torch.FloatTensor([0, 1, 0, 0])) # do backward for second element of z
print("The derivative of z_2 w.r.t to x:, ",x.grad.data)
print("The derivative of z_2 w.r.t to y: ",y.grad.data)

# do backward for all elements of z, equal to the collection of partial derivatives z_1, z_2, z_3 and z_4
x = torch.tensor([1.0, 2.0, 3.0, 4.0], requires_grad=True)
y = torch.tensor([5.0, 6.0, 7.0, 8.0], requires_grad=True)
z = (x**2) * y
z.backward(torch.FloatTensor([1, 1, 1, 1]))
print(x.grad.data)
print(y.grad.data)

x = torch.tensor([1.0, 2.0, 3.0, 4.0], requires_grad=True)
y = torch.tensor([5.0, 6.0, 7.0, 8.0], requires_grad=True)
z = (x**2) * y
z.backward(gradient=torch.ones(z.size()))
print(x.grad.data)
print(y.grad.data)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Multiple Models in One File
When saving a model comprised of multiple torch.nn.Modules, such as a GAN, a sequence-to-sequence model, or an ensemble of models, you follow the same approach as when we are saving a general checkpoint.
In other words, save a dictionary of each model’s state_dict and corresponding optimizer. As mentioned before, we can save any other items that may aid us in resuming training by simply appending them to the dictionary.
A common PyTorch convention is to save these checkpoints using the .tar file extension.
To load the models, first initialize the models and optimizers, then load the dictionary locally using torch.load(). From here, you can easily access the saved items by simply querying the dictionary as you would expect.
Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results. If you wish to resuming training, call model.train() to set these layers to training mode.
Can you write Python code for it?

torch.save({
            'modelA_state_dict': modelA.state_dict(),
            'modelB_state_dict': modelB.state_dict(),
            'optimizerA_state_dict': optimizerA.state_dict(),
            'optimizerB_state_dict': optimizerB.state_dict(),
            ...
            }, PATH)

modelA = TheModelAClass(*args, **kwargs)
modelB = TheModelBClass(*args, **kwargs)
optimizerA = TheOptimizerAClass(*args, **kwargs)
optimizerB = TheOptimizerBClass(*args, **kwargs)

checkpoint = torch.load(PATH)
modelA.load_state_dict(checkpoint['modelA_state_dict'])
modelB.load_state_dict(checkpoint['modelB_state_dict'])
optimizerA.load_state_dict(checkpoint['optimizerA_state_dict'])
optimizerB.load_state_dict(checkpoint['optimizerB_state_dict'])

modelA.eval()
modelB.eval()
# - or -
modelA.train()
modelB.train()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Note PyTorch tensors can be created with the torch.Tensor constructor, which takes the tensor’s dimensions as input and returns a tensor occupying an uninitialized region of memory
Now, let’s create our tensors with each of these options 1-4, and have a look at what we get:
Can you write Python code for it?

o1 = torch.Tensor(datanp)
o2 = torch.tensor(datanp)
o3 = torch.as_tensor(datanp)
o4 = torch.from_numpy(datanp)

print(o1)
print(o2)
print(o3)
print(o4)

print(o1.dtype)
print(o2.dtype)
print(o3.dtype)
print(o4.dtype)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Notice how each type in the table has a CPU and GPU version. One thing to keep in mind about tensor data types is that tensor operations between tensors must happen between tensors with the same type of data.
The device, cpu in our case, specifies the device (CPU or GPU) where the tensor's data is allocated. This determines where tensor computations for the given tensor will be performed.
PyTorch supports the use of multiple devices, and they are specified using an index like so:
Can you write Python code for it?


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Notice that the load_state_dict() function takes a dictionary object, NOT a path to a saved object. This means that you must deserialize the saved state_dict before you pass it to the load_state_dict() function. For example, you CANNOT load using model.load_state_dict(PATH)
Save entire model
Saving a model in the below way will save the entire module using Python’s pickle module. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.
Can you write Python code for it?

# Save the entire model
torch.save(mymodel,"/content/rahule.pt")

# Load the entire model
mymodel_nf = torch.load("/content/rahule.pt")

# model.eval()
hour_var = torch.Tensor([1.0,4.0]).cuda()
y_pred_nf = mymodel_nf(hour_var)
print("predict (after training)", y_pred_nf.data)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Notice that we never explicitly call forward pass defined within the class. We always call the Module instance afterwards instead since it takes care of running the registered hooks while the former silently ignores them.
We can also obtain the number of trainable parameters of a model using in the last line of code.
torch.nn only supports mini-batches. The entire torch.nn package only supports inputs that are a mini-batch of samples, and not a single sample. For example, nn.Conv2d will take in a 4D Tensor of nSamples x nChannels x Height x Width. If you have a single sample, just use input.unsqueeze(0) to add a fake batch dimension.
This is motivated by the computational speed-up it induces. To evaluate a module on a sample, both the module’s parameters and the sample have to be first copied into cache memory, which is fast but small. For any model of reasonable size, only a fraction of its parameters can be kept in cache, so a module’s parameter have to be copied there every time it is used. These memory transfers are slower than the computation itself.This is the main reason for batch processing: it cuts down to one per module per batch the number of copies of parameters to the cache. It also cuts down the use of Python loops, which are awfully slow.
torch.nn vs torch.nn.functional
A nn.Module is actually a OO wrapper around the functional interface, that contains a number of utility methods, like eval() and parameters(), and it automatically creates the parameters of the modules for us.
We can use the functional interface whenever we want, but that requires us to define the weights by hand.
The difference between torch.nn and torch.nn.functional is very subtle. In fact, many torch.nn.functional have a corresponding equivalent in torch.nn.
In many code samples, we use torch.nn.functional for simpler operations that have no trainable parameters or configurable parameters. For example for ReLU, we do not require any learnable parameters to be called in forward() method hence it can be defined using the...
Can you write Python code for it?

import torch

class Model(torch.nn.Module):
    super().__init__()
    self.feature_extractor = nn.Sequential(
        Conv2d(3, 12, kernel_size=3, padding=1, stride=1),
        Conv2d(12, 24, kernel_size=3, padding=1, stride=1),)

    def forward(self, x):
        x = self.feature_extractor(x)
        return x


I am working on the file "Equation_Notes2.ipynb".
The first task is:
One thing to keep in mind about using multiple devices is that tensor operations between tensors must happen between tensors that exists on the same device.
Tensors have a torch.layout
The layout, strided in our case, specifies how the tensor is stored in memory.
As neural network programmers, we need to be aware of the following:

Tensors contain data of a uniform type (dtype).
Tensor computations between tensors depend on the dtype and the device.

Creating tensors using data
These are the primary ways of creating tensor objects (instances of the torch.Tensor class), with data (array-like) in PyTorch:

torch.Tensor(data)
torch.tensor(data)
torch.as_tensor(data)
torch.from_numpy(data)

We can use a Python list, or sequence, but numpy.ndarrays are going to be the more common option, so we’ll go with a numpy.ndarray like so:
Can you write Python code for it?

import numpy as np
datanp = np.array([1,2,3])
type(datanp)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Original Problem using pytorch
Can you write Python code for it?

import torch

# device = torch.device("cpu") # Uncomment this to run on CPU
dtype = torch.float
device = torch.device("cuda:0")

N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)  #64x1000
y = torch.randn(N, D_out, device=device, dtype=dtype) #64x10

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)  #1000 x 100
w2 = torch.randn(H, D_out, device=device, dtype=dtype) #100 x 10

learning_rate = 1e-6
for t in range(5):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2


I am working on the file "Equation_Notes2.ipynb".
The first task is:
PyTorch has another function called view() that does the same thing as the reshape() function. We can use -1 in the argments if we dont know the actual size of the tensor.
Can you write Python code for it?

#!pip3 install torch torchvision
import torch
v = torch.Tensor([9,8,7,6,5,4])
v_col= v.view(6,1)
print("\nView 6x1: \n",v_col)
v_col1= v.view(3,-1)
print("\nView 3x2: \n",v_col1)

print(v)
v_col1[0][0]=0
print(v_col1)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Reduction by Axis
Can you write Python code for it?

ra = torch.tensor([
    [1,1,1,1],
    [2,2,2,2],
    [3,3,3,3]], dtype=torch.float32)

print(ra.size())     # 3 x 4 rank-2 tensor
print(ra.sum(dim=0)) # Sum along the axis containing 3 elements i.e, column wise
print(ra.sum(dim=1)) # Sum along the axis containing 4 elements i.e, row wise


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Reduction operations
A reduction operation on a tensor is an operation that reduces the number of elements contained within the tensor.
Can you write Python code for it?

red = torch.tensor([
    [0,1,0],
    [2,0,2],
    [0,3,0]], dtype=torch.float32)

print(red.sum())
print("\nThe number of elements in original tensor",red.numel())

print("\nConcating Operations: ")
print(red.sum().numel())
print(red.sum().numel() < red.numel())

print(red.sum())
print(red.prod())
print(red.mean())
print(red.std())


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Retain graph
In PyTorch, the computation graph is created at the start of each iteration in an epoch and then subsequently freed to save memory at the end of that iteration.
In each iteration, we execute the forward pass, compute the derivatives of output w.r.t to the parameters of the network, and update the parameters to fit the given examples. After doing the backward pass, the graph will be freed to save memory. In the next iteration, a fresh new graph is created and ready for back-propagation
Because the computation graph will be freed by default after the first backward pass, you will encounter errors if you are trying to do backward on the same graph the second time.
We can specify retain_graph=True when calling backward the first time to make sure the graph is retained and the buffers are not freed till we finish the backward propogation through the graph a second time.
During the optimization step, we combine the chain rule and the graph to compute the derivative (partial) of the output w.r.t the learnable variable in the graph and update these variables to make the output close to what we want.
Can you write Python code for it?

#!pip3 install torch torchvision
import torch
f = torch.tensor([2.0,3.0],requires_grad=True)
print("Original Tensor: ",f)
g = f[0] * f[1]
g.backward(retain_graph=True)
#g.backward()
print("1st Backward Pass: ",f.grad)
g.backward()
print("2nd Backward Pass: ",f.grad)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Save and Load Models
You can save the models and load them back for inference. They are two ways of doing it:

Save the entire model and load it back - torch.save function saves a serialized object to disk. This function uses Python’s pickle utility for serialization. Models, tensors, and dictionaries of all kinds of objects can be saved using this function. Similarly torch.load function uses pickle’s unpickling facilities to deserialize pickled object files to memory. This function also facilitates the device to load the data into.

Save only model parameters - Neural network modules as well as optimizers have the ability to save and load their internal state using .state_dict(). With this we can continue training from previously saved state dicts and if needed
we'd just need to call .load_state_dict(state_dict).

torch.save(model.state_dict(), '/results/model.pth')

torch.save(optimizer.state_dict(), '/results/optimizer.pth')

A state_dict is simply a Python dictionary object that maps each layer with learnable parameters (convolutional layers, linear layers, etc.) to its parameter tensor (weights and biases accessed by accessed with model.parameters()).
Optimizer objects (torch.optim) also have a state_dict, which contains information about the optimizer’s state, as well as the hyperparameters used
Because state_dict objects are Python dictionaries, they can be easily saved, updated, altered, and restored, adding a great deal of modularity to PyTorch models and optimizers.
Can you write Python code for it?

# Print model's state_dict
print("Model's state_dict:")
for param_tensor in mymodel.state_dict():
    print("\n",param_tensor, "\t", mymodel.state_dict()[param_tensor].size())

# Print optimizer's state_dict
print("\nOptimizer's state_dict:")
for var_name in optimizer.state_dict():
    print("\n",var_name, "\t", optimizer.state_dict()[var_name])

# Save only model parameters
torch.save(mymodel.state_dict(), "/content/rahul.pth")

# Load only model parameters
# To re-read the parameters of the model, first we need to redefine the model once, then re-read the parameters
mymodel_n = net().cuda()
mymodel_n.load_state_dict(torch.load("/content/rahul.pth"))

print(mymodel_n)
# print(list(mymodel_n.parameters()))

# model.eval()
hour_var = torch.Tensor([1.0,4.0]).cuda()
y_pred_n = mymodel_n(hour_var)
print("predict (after training)", y_pred_n.data)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Save on CPU, Load on GPU
When loading a model on a GPU that was trained and saved on CPU, set the map_location argument in the torch.load() function to cuda:device_id. This loads the model to a given GPU device. Next, be sure to call model.to(torch.device('cuda')) to convert the model’s parameter tensors to CUDA tensors. Finally, be sure to use the .to(torch.device('cuda')) function on all model inputs to prepare the data for the CUDA optimized model. Note that calling my_tensor.to(device) returns a new copy of my_tensor on GPU. It does NOT overwrite my_tensor. Therefore, remember to manually overwrite tensors: my_tensor = my_tensor.to(torch.device('cuda')).
Can you write Python code for it?

torch.save(model.state_dict(), PATH)

device = torch.device("cuda")
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH, map_location="cuda:0"))  # Choose whatever GPU device number you want
model.to(device) # Model to GPU
# Make sure to call the code; input = input.to(device) on any input tensors that you feed to the model


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Save on GPU, Load on GPU
When loading a model on a GPU that was trained and saved on GPU, simply convert the initialized model to a CUDA optimized model using model.to(torch.device('cuda')). Also, be sure to use the .to(torch.device('cuda')) function on all model inputs to prepare the data for the model. Note that calling my_tensor.to(device) returns a new copy of my_tensor on GPU. It does NOT overwrite my_tensor. Therefore, remember to manually overwrite tensors: my_tensor = my_tensor.to(torch.device('cuda')).
Can you write Python code for it?

# Save
torch.save(model.state_dict(), PATH)

# Load
device = torch.device("cuda")
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.to(device) # Model to GPU
# Make sure to call the code; input = input.to(device) on any input tensors that you feed to the model


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Saving & Loading Model Across Devices

Save on GPU, Load on CPU
Save on GPU, Load on GPU
Save on CPU, Load on GPU

Save on GPU, Load on CPU
When loading a model on a CPU that was trained with a GPU, pass torch.device('cpu') to the map_location argument in the torch.load() function. In this case, the storages underlying the tensors are dynamically remapped to the CPU device using the map_location argument.
Can you write Python code for it?

# Save
torch.save(model.state_dict(), PATH)

# Load
device = torch.device('cpu')
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH, map_location=device)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Since the backward() function accumulates gradients, and you don’t want to mix up gradients between minibatches, you have to zero them out at the start of a new minibatch. This is exactly like how a general (additive) accumulator variable is initialized to 0 in code.
By the way, the best practice is to use the zero_grad() function on the optimizer.
Example 3:
The autograd graph is encoded through the
* fields grad_fn of Tensor s, and
* the fields next_functions of Function s.
Can you write Python code for it?

#!pip3 install torch torchvision
import torch

x = torch.tensor([ 1.0, -2.0, 3.0, -4.0 ]).requires_grad_()
a = x.abs()
s = a.sum()
print(s)
print(s.grad_fn.next_functions)
print(s.grad_fn.next_functions[0][0].next_functions)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Sometimes it gets difficult to plot our loss with respect to every parameter. As a result, we will store the loss in a list for each iteration.
We simply create an empty list, calculate the loss, and then store the loss value in the list. We plot out the loss for each iteration.
Can you write Python code for it?

print(mymodel.linear1.weight.grad.size())
print(mymodel.linear1.weight.data[0])
print(mymodel.linear1.weight.data.norm())       # norm of the weight
print(mymodel.linear1.weight.grad.data.norm())  # norm of the gradients


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Suppose that we have a computation graph shown above. The variable d and e is the output, and a is the input. The underlining computation is:
Can you write Python code for it?

import torch
a = torch.randn((1,4),requires_grad=True)
b = a**2
c = b*2
d = c.mean()
e = c.sum()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Tensor Functions
In practice, one will most often want to use one of PyTorch’s functions that return tensors initialized in a certain manner, such as:

torch.rand: values initialized from a random uniform distribution,
torch.randn: values initialized from a random normal distribution,
torch.eye(n): an $n×n$ identity matrix,
torch.from_numpy(ndarray): a PyTorch tensor from a NumPy ndarray,
torch.linspace(start, end, steps): a 1-D tensor with steps values spaced linearly between start and end,
torch.ones : a tensor with ones everywhere,
torch.zeros_like(other) : a tensor with the same shape as other and zeros everywhere,
torch.arange(start, end, step): a 1-D tensor with values filled from a range.

Tensor Operations
PyTorch tensors provide a very rich API for combination with other tensors as well as in-place mutation.
Also like NumPy, unary and binary operations can usually be performed via functions in the torch module, like torch.add(x, y), or directly via methods on the tensor objects, like x.add(y).
For the usual suspects, operator overloads like x + y exist. Furthermore, many functions have in-place alternatives that will mutate the receiver instance rather than creating a new tensor. These functions have the same name as the out-of-place variants, but are suffixed with an underscore, e.g. x.add_(y).
A selection of operations includes:

torch.add(x, y): elementwise addition,
torch.mm(x, y): matrix multiplication (not matmul or dot),
torch.mul(x, y): elementwise multiplication,
torch.exp(x): elementwise exponential,
torch.pow(x, power): elementwise exponentiation,
torch.sqrt(x): elementwise squaring,
torch.sqrt_(x): in-place elementwise squaring,
torch.sigmoid(x): elementwise sigmoid.
torch.cumprod(x): product of all values,
torch.sum(x): sum of all values,
torch.std(x): standard deviation of all values,
torch.mean(x): mean of all values.

Tensors support many of the familiar semantics of NumPy ndarray’s, such as broadcasting, advanced (fancy) indexing (x[x > 5]) and elementwise relational operators (x >...
Can you write Python code for it?

t = torch.tensor([
    [1,1,1,1],
    [2,2,2,2],
    [3,3,3,3]
], dtype=torch.float32)

print(t.size())
print(t.shape)  #shape is an alias for size()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Tensor Indexing and Slicing
Pytorch supports indexing and slicing of arrays just like sequences in python. Slices reference a subset of the array without copying the underlying data
We use rectangular brackets to access different elements of a tensor.
Depending on the number of dimensions of the tensor, we use the following notation for indexing
$$ tensor[dim_0, dim_1, dim_2,\cdots,dim_n]$$
Similary, slices are taken depending on the rank of the tensor.
All index values of a slice are optional, and the defaults values are the start of the sequence, the last item in the sequence, and a default step size of one where it cannot be zero. A negative step size can reverse the order of the output result.
$$tensor[slice_0,slice_1,slice_2,\cdots,slice_n]$$
where

[start: end: step] - Slice from [start] to [end-1] but step ahead based on step values

[ : : ] – All elements of the entire tensor

n representing the number of dimensions of original tensor
Can you write Python code for it?

#!pip3 install torch torchvision
import torch
a = torch.tensor([[9,8,7],[6,5,4],[3,2,1]])
#[ [a[0][0],a[0][1],a[0][2]], [a[1][0],a[1][1],a[1][2]],[a[2][0],a[2][1],a[2][2]] ]
print("\nGiven array: \n",a)

print("\nIndexing: ")
print(a[0,0])  # Single bracket notation for accessing individual elements
print(a[0][0]) # Double bracket notation for accessing individual elements
print(a[0])
print(a[1])

print("\nSlicing: ")
slice1 = a[0:3,0:2]
print(slice1)

print("\n")
slice1[0][0]=0
print(slice1)

print("\nMixing Indexing and Slicing:")
print(a[0,0:2])


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Tensor Types
Tensors are homogeneous as all the items within it are of the same data type. PyTorch tensors are instances of the torch.Tensor Python class. We can create a torch.Tensor object using the class constructor as below. They can also be created from torch.tensor.
Can you write Python code for it?

a = torch.tensor([0,1,2,3,4])

print("Type of entire array:")
type(a)

print("Datatype of tensor:")
a.type()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Testing
Can you write Python code for it?

# Test the model
# In test phase, we don't need to compute gradients (for memory efficiency)
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in testloader:
        images = images.reshape(-1, 28*28).cuda() # Flatten the data (n, 1, 28, 28)-> (n, 784)
        labels = labels.cuda()
        outputs = net1(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

# Save the model checkpoint
torch.save(net1.state_dict(), 'model.ckpt')


I am working on the file "Equation_Notes2.ipynb".
The first task is:
The DataLoader actually has some reasonably sophisticated logic to determine how to collate individual samples returned from our dataset’s getitem method into a batch, as returned by the DataLoader during iteration.
For example, if getitem returns a dictionary, the DataLoader will aggregate the values of that dictionary into a single mapping for the entire batch, using the same keys.
This means that if the Dataset’s getitem returns a dict(example=example, label=label), then the batch returned by the DataLoader will return something like dict(example=[example1, example2, ...], label=[label1, label2, ...]), i.e. unpacking the values of indidvidual samples and re-packing them into a single key for the batch’s dictionary.
To override this behavior, you can pass a function argument for the collate_fn parameter to the DataLoader object.

Note: To create a dataset class for most image datasets we will read the csv in init but leave the reading of images to getitem. This is memory efficient because all the images are not stored in the memory at once but read as required.

MNIST Example

MNIST is a collection 28 x 28 pixel images of handwritten digits.

Every image can be thought of as an array of numbers between 0 and 1 describing how dark each pixel is (intensity of pixel).

Every MNIST data point has two parts:
1. Image of a handwritten digit
2. Corresponding label (number between 0 and 9) representing the digit drawn in the image.
The label will be used to compare the predicted digit (by the model) with the true digit (given by the data).

Each of the 28 x 28 pixel images of handwritten digits can be represented as an array of 784 numbers each corresponding to respective pixels.

Every image in MNIST is of a handwritten digit between 0 and 9

So there are only 10 possible things that an image can be

For each image, we want to give the probabilities for it being each digit.

Hence, we want to normalize the evidences, so that all the evidences for one image sum up to 1 and each element
Is limited...
Can you write Python code for it?

#!pip3 install torch torchvision
import torch
import torch.nn as nn
import torch.nn.functional as F

class mymnist(nn.Module):
    def __init__(self):
        super(mymnist, self).__init__()
        self.fc1 = nn.Linear(28 * 28, 200)
        self.fc2 = nn.Linear(200, 200)
        self.fc3 = nn.Linear(200, 10)
    def forward(self, x):
      x = F.relu(self.fc1(x))
      x = F.relu(self.fc2(x))
      x = self.fc3(x)
      return F.log_softmax(x,dim=1)

net1 = mymnist().cuda()
print(net1)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
The above step creates an instance of the network architecture. We then setup an optimizer and a loss criteria. In PyTorch, the optimizer knows how to optimize any attribute of type Parameter.
Can you write Python code for it?

# create a stochastic gradient descent optimizer
optimizer = torch.optim.SGD(mymodel.parameters(), lr=0.0009, momentum=0.9)
# create a loss function
#criterion = nn.NLLLoss()
criterion = torch.nn.MSELoss(reduction='sum')


I am working on the file "Equation_Notes2.ipynb".
The first task is:
The dimension corresponds to the number of nested list sets
Can you write Python code for it?

print("The dimension of tensors s1d, s2d, s3d,s4d are:")
print(s1d.ndimension(),s2d.ndimension(),s3d.ndimension(),s4d.ndimension())


I am working on the file "Equation_Notes2.ipynb".
The first task is:
The first two o1 and o2 still have the original value of 1 for index 0, while the second two o3 and o4 have the new value of 0 for index 0.
This happens because torch.Tensor() and torch.tensor() copy their input data while torch.as_tensor() and torch.from_numpy() share their input data in memory with the original input object.
This sharing just means that the actual data in memory exists in a single place. As a result, any changes that occur in the underlying data will be reflected in both objects.
Sharing data is more efficient and uses less memory than copying data because the data is not written to two locations in memory.
This establishes that torch.as_tensor() and torch.from_numpy() both share memory with their input data.
The torch.from_numpy() function only accepts numpy.ndarrays, while the torch.as_tensor() function accepts a wide variety of Python array-like objects including other PyTorch tensors. For this reason, torch.as_tensor() is the winning choice in the memory sharing game.
Given all of these details, these two are the best options:

torch.tensor()
torch.as_tensor()

The torch.tensor() call is the sort of go-to call, while torch.as_tensor() should be employed when tuning our code for performance.
Rank, Axes and Shape of Tensors
The concepts of rank, axes, and shape are the tensor attributes that will concern us most in deep learning.

Rank
Axes
Shape

These concepts build on one another starting with rank, then axes, and building up to shape, so keep any eye out for this relationship between these three.
Rank of a tensor
The rank of a tensor refers to the number of dimensions present within the tensor. Suppose we are told that we have a rank-2 tensor. This means all of the following:

We have a matrix
We have a 2d-array
We have a 2d-tensor

We are introducing the word rank here because it is commonly used in deep learning when referring to the number of dimensions present within a given tensor.
The rank of a tensor tells us how many indexes are required to access (refer to) a specific...
Can you write Python code for it?


I am working on the file "Equation_Notes2.ipynb".
The first task is:
The special method forward() will automatically get called when the layer is calculated and its gonna get passed the data from previous layer.
Using Custom Class
Can you write Python code for it?

# -*- coding: utf-8 -*-
import torch

class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        #h_relu = self.linear1(x).clamp(min=0)
        h_relu = torch.nn.functional.relu(self.linear1(x))
        y_pred = self.linear2(h_relu)
        return y_pred

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Construct our model by instantiating the class defined above
model = TwoLayerNet(D_in, H, D_out)

# Construct our loss function and an Optimizer. The call to model.parameters()
# in the SGD constructor will contain the learnable parameters of the two
# nn.Linear modules which are members of the model.
criterion = torch.nn.MSELoss(reduction='sum')

optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)

for t in range(5):
    # Forward pass: Compute predicted y by passing x to the model
    y_pred = model(x) #Construct the graph and return the output of forward

    # Compute and print loss, can directly call nn.functional.mse_loss(out, y)since it is a non-parametric function
    loss = criterion(y_pred, y)
    print(t, "{:.20f}".format(loss.item()))

    # Zero gradients, perform a backward pass, and update the weights.
    optimizer.zero_grad()
    loss.backward()

    optimizer.step()

sum([p.numel() for p in model.parameters()])


I am working on the file "Equation_Notes2.ipynb".
The first task is:
There are many methods available for each module to access its children —
* model.modules() , model.named_modules() , model.parameters() model.named_parameters() , model.children() and model.named_children().
But the most used is model.parameters() , as this is used to access all the parameters recursively and hence can be used to pass to an optimizer for updating weights/bias.
Parameters are of the type torch.nn.Parameter which is a Tensor with requires_grad to True, and known to be a model parameter by various utility functions, in particular torch.nn.Module.parameters()
LOSS/COST
torch.nn has loss/cost functions called criteria.

L1Loss, MSELoss, CrossEntropyLoss, CTCLoss, NLLLoss, PoissonNLLLoss, KLDivLoss, BCELoss, BCEWithLogitsLoss, MarginRankingLoss, HingeEmbeddingLoss, MultiLabelMarginLoss, SmoothL1Loss, SoftMarginLoss, MultiLabelSoftMarginLoss, CosineEmbeddingLoss, MultiMarginLoss, TripletMarginLoss

The general syntax is

loss = torch.nn.MSELoss()
output = loss(input, target)
output.backward()

Note: Criteria/Loss do not accept a tensor with requires_grad set to True for target.
The first parameter of a loss is traditionally called the input and the second the target. These two quantities may be of different dimensions or even types for some losses (e.g.for classification).
OPTIMIZATION/WEIGHT UPDATES
torch.optim is a package implementing various optimization algorithms

ASGD, Adadelta, Adagrad, Adam, Adamax, LBFGS, RMSprop, Rprop, SGD, SparseAdam

To use torch.optim we construct an optimizer object, that will hold the current state and will update the parameters based on the computed gradients.
We give it an iterable containing the parameters to optimize such as the learning rate, weight decay which are optimizer specific.

optimizer = optim.SGD(model.parameters(), lr = 0.01, momentum=0.9)
optimizer = optim.Adam([var1, var2], lr = 0.0001)

model.parameters() returns an iterator over our model’s parameters (weights and biases).
If we need to move a model to GPU via .cuda(), we need to do so...
Can you write Python code for it?

#!pip3 install torch torchvision
import torch
import torch.optim

x = torch.randn((5, 5))
w = torch.randn((5, 5),requires_grad=True)

z = w.mm(x).mean() # Perform an operation

opt = torch.optim.Adam([w], lr=0.1, betas=(0.5, 0.999)) # Define the optimizer
z.backward() # Calculate gradients
print(w.data) # Print the weights

opt.step() # Update w according to Adam's gradient update rules
print(w.data) # Print updated weights after training step

opt.step()
print(w.data)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
There are some other ways to refer to element-wise operations, and all of these mean the same thing:

Element-wise
Component-wise
Point-wise
Can you write Python code for it?

print(t.abs())
print(t.sqrt())
print(t.neg())
print(t.neg().abs())


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Thus torch.nn.Sequential expects a list of the layers that we want in the neural network. in our case our list has two things, we want a linear layer (y=wx+b) whose input is a vector of some length along with a non-linear activation function relu followed another linear layer with sigmoid activation function - this is a two layer neuron model
There is also the implicit input layer which is understood.
Can you write Python code for it?

import torch

#device = torch.device("cpu") # Uncomment this to run on CPU
dtype = torch.float
device = torch.device("cuda:0")

N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)  #64x1000
y = torch.randn(N, D_out, device=device, dtype=dtype) #64x10

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)  #1000 x 100
w2 = torch.randn(H, D_out, device=device, dtype=dtype) #100 x 10

# Use the nn package to define our model as a sequence of layers. nn.Sequential
# is a Module which contains other Modules, and applies them in sequence to
# produce its output. Each Linear Module computes output from input using a
# linear function, and holds internal Tensors for its weight and bias.
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
    torch.nn.Sigmoid())

model = model.cuda()

# The nn package also contains definitions of popular loss functions; in this
# case we will use Mean Squared Error (MSE) as our loss function.
loss_function = torch.nn.MSELoss(reduction='sum').cuda()

# Set the learning rate value
learning_rate = 1e-4

for t in range(5):
  # Forward pass: compute predicted y by passing x to the model. Module objects
  # override the __call__ operator so you can call them like functions. When
  # doing so you pass a Tensor of input data to the Module and it produces
  # a Tensor of output data. Basically construct the graph
  y_pred=model(x)

  # Compute and print loss. We pass Tensors containing the predicted and true
  # values of y, and the loss function returns a Tensor containing the
  # loss.
  loss = loss_function(y_pred, y)
  print(t, "{:.20f}".format(loss.item()))

  # Zero the gradients before running the backward pass.
  model.zero_grad()

  # Backward pass: compute gradient of the loss with respect to all the learnable
  # parameters of the model. Internally, the parameters of each Module are stored
  # in Tensors with...


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Training
The outer training loop is the number of epochs, whereas the inner training loop runs through the entire training set in batch sizes which are specified in the code as batch_size.
The MNIST input data-set which is supplied in the torchvision package has the size (batch_size, 1, 28, 28) when extracted from the data loader – this 4D tensor is more suited to convolutional neural network architecture, and not so much our fully connected network. Therefore we need to flatten out the (1, 28, 28) data to a single dimension of 28 x 28 =  784 input nodes.
The .view() function operates on PyTorch tensor to reshape them. If we want to be agnostic about the size of a given dimension, we can use the “-1” notation in the size definition. So by using data.view(-1, 28*28) we say that the second dimension must be equal to 28 x 28, but the first dimension should be calculated from the size of the original data variable. In practice, this means that data will now be of size (batch_size, 784). We can pass a batch of input data like this into our network and the magic of PyTorch will do all the hard work by efficiently performing the required operations on the tensors.
On the next line, we run optimizer.zero_grad() – this zeroes / resets all the gradients in the model, so that it is ready to go for the next back propagation pass. In other libraries this is performed implicitly, but in PyTorch we have to remember to do it explicitly.
The next line is where we pass the input data batch into the model – this will actually call the forward() method in our Net class.
After this line is run, the tensor outputs will now hold the log softmax output of our neural network for the given data batch.
That’s one of the great things about PyTorch, we can activate whatever normal Python debugger we usually use and instantly get a gauge of what is happening in your network. This is opposed to other deep learning libraries such as TensorFlow and Keras which require elaborate debugging sessions to be setup before you can check out what...
Can you write Python code for it?

# Hyper-parameters
num_epochs = 1
learning_rate = 0.001

# Train the model
total_step = len(trainloader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(trainloader):
        # Move tensors to the configured device
        images = images.reshape(-1, 28*28).cuda() # Flatten the data (n, 1, 28, 28)-> (n, 784)
        labels = labels.cuda()

        # Forward pass
        outputs = net1(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Typically, after we know a tensor’s shape, we can deduce a couple of things. First, we can deduce the tensor's rank. The rank of a tensor is equal to the length of the tensor's shape.
We can also deduce the number of elements contained within the tensor. The number of elements inside a tensor (12 in our case) is equal to the product of the shape's component values.
Can you write Python code for it?

print(len(t.shape))
print(torch.tensor(t.shape).prod())
print(t.numel()) #In PyTorch, there is a dedicated function for determining the number of elements inside a tensor


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Up to this point we have updated the weights of our models by manually mutating the Tensors holding learnable parameters (with torch.no_grad() or .data to avoid tracking history in autograd). This is not a huge burden for simple optimization algorithms like stochastic gradient descent, but in practice we often train neural networks using more sophisticated optimizers like AdaGrad, RMSProp, Adam, etc.
The optim package in PyTorch abstracts the idea of an optimization algorithm and provides implementations of commonly used optimization algorithms.
In this example we will use the nn package to define our model as before, but we will optimize the model using the Adam algorithm provided by the optim package:
Can you write Python code for it?

# -*- coding: utf-8 -*-
import torch

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Use the nn package to define our model and loss function.
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out))

loss_fn = torch.nn.MSELoss(reduction='sum')

# Use the optim package to define an Optimizer that will update the weights of
# the model for us. Here we will use Adam; the optim package contains many other
# optimization algoriths. The first argument to the Adam constructor tells the
# optimizer which Tensors it should update.
learning_rate = 1e-4

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for t in range(5):
    # Forward pass: compute predicted y by passing x to the model.
    y_pred = model(x) # Compute the graph

    # Compute and print loss.
    loss = loss_fn(y_pred, y)
    print(t, "{:.20f}".format(loss.item()))

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables it will update (which are the learnable
    # weights of the model). This is because by default, gradients are
    # accumulated in buffers( i.e, not overwritten) whenever .backward()
    # is called. Checkout docs of torch.autograd.backward for more details.
    optimizer.zero_grad()

    # Backward pass: compute gradient of the loss with respect to model
    # parameters
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its
    # parameters
    optimizer.step()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Using Class
In PyTorch when using classes, we define the Models as subclasses of torch.nn.Module.
The two functions required while defining (inherenting) any model using class are:

init - We define all the parametric (and sometimes non-parametric) layers. This function has to always be inherited first, then we define parameters of the layer such as the class variables i.e. self.x
forward  - We connect all the layers and other functions to the input to form a graph.

In the _init_ function,  we initialize the layers we want to use and Pytorch goes more low level and we have to specify the sizes of our network so that everything matches.
In the forward method, we specify the connections of your layers. This means that we will use the layers we already initialized, in order to re-use the same layer for each forward pass of data we make.
For layers that do not have trainable weights, we can use either the layer form (from torch.nn) or connection form (from torch.nn.functional),
Can you write Python code for it?

#!pip3 install torch torchvision
import torch

import torch.nn as nn
import torch.nn.functional as F

class net(torch.nn.Module):
  def __init__(self):
    super(net,self).__init__() # Can also use torch.nn.Module.__init__(self)
    self.linear1=torch.nn.Linear(D_in,H)
    self.linear2=torch.nn.Linear(H,D_out)

  def forward(self,x):
    h_relu = torch.nn.functional.relu(self.linear1(x))
    #h_relu = self.linear1(x).clamp(min=0)
    y_pred = self.linear2(h_relu)
    return y_pred


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Using pytorch built in functions.
Can you write Python code for it?

import torch
N, D_in, H, D_out = 64, 1000, 100, 10

x = torch.randn(N, D_in,requires_grad=False)
y = torch.randn(N, D_out,requires_grad=False)

w1 = torch.randn(D_in, H,requires_grad=True)
w2 = torch.randn(H, D_out,requires_grad=True)

learning_rate = 1e-6

for t in range(5):
    # Do forward
    y_pred = x.mm(w1).clamp(min=0).mm(w2)

    # Compute Loss
    loss = (y_pred - y).pow(2).sum()
    print(t, loss.detach().item())

    # Backprop to compute gradients (partial derivatives) of w1 and w2 with respect to loss
    loss.backward()

    # Update weights using gradient descent
    w1.data -= learning_rate * w1.grad.data
    w2.data -= learning_rate * w2.grad.data

    # Zero them for next iteration, as we have already update the weights with gradient data
    w1.grad.data.zero_()
    w2.grad.data.zero_()


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Usually after a backpropagation you process the next batch so you don’t need the gradients of the previous batch anymore.
retain_variables argument has been deprecated in favor of retain_graph
A real use case that you want to backward through the graph for more than once is multi-task learning where you have multiple losses at different layers. Suppose that you have 2 losses: loss1 and loss2 and they reside in different layers. In order to back-prop the gradient of loss1 and loss2 w.r.t to the learnable weight of your network independently. You have to use retain_graph=True in backward() method in the first back-propagated loss.
Can you write Python code for it?

#loss1.backward(retain_graph=True) # suppose you first back-propagate loss1, then loss2 (you can also do it in reverse order)
#loss2.backward() # now the graph is freed, and next process of batch gradient descent is ready
#optimizer.step() # update the network parameters


I am working on the file "Equation_Notes2.ipynb".
The first task is:
We can

Combine fancy indexing with simple indices

Combine fancy indexing with slicing

Combine fancy indexing with masking
Can you write Python code for it?

r = torch.tensor([[0.0,1.0,2.0,3.0],
       [4.0,5.0,6.0,7.0],
       [8.0,9.0,10.0,11.0]])

print(r)
print("\nCombine fancy indexing and simple indices :", r[2, [2, 0, 1]])
print("\nCombine fancy indexing with slicing: ", r[1:, [2, 0, 1]])
#print("\nCombine fancy indexing with masking: ",)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
We often want to display a grid of images to show samples for the training or testing images. torchvision.utils.make_grid makes a grid to be displayed as known as Image grid. It takes input a 4D mini-batch Tensor of shape (B x C x H x W) or a list of images all of the same size.
matplotlib.pyplot.imshow() needs a 2D array, or a 3D array with the third dimension being of shape 3 or 4!
Thus iff X is an array, it can have the following shapes and types:
    MxN – values to be mapped (float or int)
    MxNx3 – RGB (float or uint8)
    MxNx4 – RGBA (float or uint8)

The value for each component of MxNx3 and MxNx4 float arrays should be in the range 0.0 to 1.0. MxN arrays are mapped to colors based on the norm (mapping scalar to scalar) and the cmap (mapping the normed scalar to a color).
Can you write Python code for it?

im = torchvision.utils.make_grid(images)
print(im.size())
plt.imshow(np.transpose(im.numpy(), (1, 2, 0)))
#plt.imshow(np.transpose(im.numpy(), (2, 1, 0)))


I am working on the file "Equation_Notes2.ipynb".
The first task is:
What is a tensor?
The inputs, outputs, and transformations within neural networks are all represented using tensors, and as a result, neural network programming utilizes tensors heavily.
The concept of a tensor is a mathematical generalization of other more specific concepts. Let’s look at some specific instances of tensors.
Specific instances of tensors
Each of these examples are specific instances of the more general concept of a tensor:

number
scalar
array
vector
2d-array
matrix

Let’s organize the above list of example tensors into two groups:

number, array, 2d-array
scalar, vector, matrix

The first group of three terms (number, array, 2d-array) are terms that are typically used in computer science, while the second group (scalar, vector, matrix) are terms that are typically used in mathematics.
Tensors can be created using the following syntax.
Can you write Python code for it?


I am working on the file "Equation_Notes2.ipynb".
The first task is:
Wrap-up

The backward() function made differentiation very simple
For non-scalar tensor, we need to specify grad_tensors
If you need to backward() twice on a graph or subgraph, you will need to set retain_graph to be true
Note that grad will accumulate from excuting the graph multiple times

There are several attributes related to gradients that every tensor has:
grad: A property which holds a tensor of the same shape containing computed gradients.
is_leaf: True, if this tensor was constructed by the user and False, if the object is a result of function transformation.
requires_grad: True if this tensor requires gradients to be calculated. This property is inherited from leaf tensors, which get this value from the tensor construction step (zeros() or torch.tensor() and so on). By default, the constructor has requires_grad=False, so if you want gradients to be calculated for your tensor, then you need to explicitly say so.
Homework
Check if operations and functions in pytorch support broadcasting and which of them share memory and dont share memory (create new copies).
Pytorch Example
The network will have a single hidden layer, i.e, a two-layer network and will be trained with gradient descent to fit random data by minimizing the Euclidean distance between the network output and the true output. This is adapter from module1 but implemented using pytorch without helper functions.
Can you write Python code for it?

# Original Problem
# -*- coding: utf-8 -*-
import numpy as np

N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)

learning_rate = 1e-6
for t in range(5):
    # Forward pass: compute predicted y
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Zero the grads is not required as we are not using torch backward() function which accumulates buffers

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)

    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2


I am working on the file "Equation_Notes2.ipynb".
The first task is:
buffers(recurse=True)
Returns an iterator over module buffers. If  recurse=True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module
Can you write Python code for it?

for buf in net.buffers():
  print(type(buf.data), buf.size())


I am working on the file "Equation_Notes2.ipynb".
The first task is:
named_buffers(prefix='', recurse=True)
Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
Children( modules)
Returns an iterator over immediate children modules
Can you write Python code for it?

for child in net.children():
  print(child)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
named_modules(memo=None, prefix='')
Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Duplicate modules are returned only once.
Can you write Python code for it?

l = nn.Linear(2, 2)
net = nn.Sequential(l, l)
for idx, m in enumerate(net.named_modules()):
  print(idx,":",m)


I am working on the file "Equation_Notes2.ipynb".
The first task is:
named_parameters(prefix='', recurse=True)
Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
Can you write Python code for it?

for name, param in net.named_parameters():
    print(name,": ",param.size())


I am working on the file "Equation_Notes2.ipynb".
The first task is:
parameters(recurse=True)
Returns an iterator over module parameters. This is typically passed to an optimizer.
Can you write Python code for it?

for param in net.parameters():
  print(param,":",param.size())


I am working on the file "Equation_Notes2.ipynb".
The first task is:
torch.cat() concatenate tensors along an existing dimension and torch.stack() stack tensors along a new dimension.
torch.cat and torch.stack requires the arguments to have the same shape (except along the concatenating dimension). They don't implement broadcasting. Pytorch don't enforce this sufficiently, which leads to the incorrect values.
Can you write Python code for it?

import torch

a = torch.randn(1, 4)
b = torch.randn(1, 4)
print(a)
print(b)

print("\n")
abcat = torch.cat((a, b), 0)
print(abcat)
print(abcat.shape)

print("\n")
abstack = torch.stack((a, b),0) # The last element is the dimension to insert. Has to be between 0 and the number of dimensions of concatenated tensors (inclusive)
print(abstack)
print(abstack.shape)

tensor_one = torch.tensor([[1,2,3],[4,5,6]])
print(tensor_one.shape)

tensor_two = torch.tensor([[7,8,9],[10,11,12]])
print(tensor_two.shape)

tensor_tre = torch.tensor([[13,14,15],[16,17,18]])
print(tensor_tre.shape)

tensor_list = [tensor_one, tensor_two, tensor_tre]

stacked_tensor = torch.stack(tensor_list)
print(stacked_tensor.shape)


I am working on the file "Equation_To_Object_Instructions.ipynb".
The first task is:
Making A Parameter Grid
The equation we came up with is called a parametric equation.  It maps the parameters t and u, which can be thought of as the coordinates of points on a two-dimensional surface, to the coordinates x y and z in a three-dimensional space.  To make things concrete, we have to come up with a set of specific parameters.  The mgrid object from the library numpy is a way to do that.  We give it a start, an end and an increment in each of the two dimensions.  It gives us two two-dimensional matrices, one with t coordinates, and one with u coordinates.  Execute the cell below to create those matrices.
Can you write Python code for it?


I am working on the file "Equation_To_Object_Instructions.ipynb".
The first task is:
So now we have a representation of the shape of our equation that we can work with using Python tools.  Now let's take a look at a more complex example, and see how we can make it ready for a 3D printer.
A More Complex Example
We won't work through creating this equation in Grapher, or how to translate it to Python.  It's just here to give us an interesting shape to send to the 3D printer.  Execute each of the code cells in order.
This equation:
$$
\left[ \begin{array}{c} x \ y \ z \end{array} \right]=\left[ \begin{array}{c} \frac{\sin \left( u\; \cos \left( 6v \right) \right)}{\cos \left( 6v \right)} \ v\; +\; \left( \frac{u\; \sin \left( 12v \right)}{2^{3}} \right) \ \frac{1-\cos \left( u\; \cos \left( 6v \right) \right)}{\cos \left( 6v \right)} \end{array} \right]
$$
can be translated into the Python function below:
Can you write Python code for it?

def ruffle(u, v):
    x = sin(u * cos(v*6)) / cos(v*6)
    y = v + (u * sin(12*v)/2**3)
    z = (1 - cos(u * cos(v*6))) / cos(v*6)
    return (x, y, z)


I am working on the file "Equation_To_Object_Instructions.ipynb".
The first task is:
The Equation As a Python Function
Our equation can be thought of as a relationship between any two numbers t and u and a list of three numbers x y and z.  That is, if you choose specific numbers for t and u, you can find out corresponding values for x y and z.  That concept can be expressed as a Python function that takes two arguments t and u and returns a tuple of x y and z.  Take a look at the cell below, then execute it.
Can you write Python code for it?

def saddle(t, u):
    x = sin(t)
    y = sin(u)
    z = cos(u) - cos(t)
    return (x, y, z)


I am working on the file "Equation_To_Object_Instructions.ipynb".
The first task is:
This function will wrap another function in a circle around the z axis:
Can you write Python code for it?

def wrap(x, y, z):
    x1 = x * cos(y)
    y1 = x * sin(y)
    z1 = z
    return (x1, y1, z1)


I am working on the file "Equation_To_Object_Instructions.ipynb".
The first task is:
This will let us scale the size of the figure (for instance, to scale from millimeters to centimeters):
Can you write Python code for it?

def scale(x, y, z, scale_factor=1.0):
    x1 = x * scale_factor
    y1 = y * scale_factor
    z1 = z * scale_factor
    return (x1, y1, z1)


I am working on the file "Equation_To_Object_Instructions.ipynb".
The first task is:
We set up the parameter grid.
I used t and u in the simple example because that's what Grapher uses by default, and I used u and v in this example because it's what I'm used to.  Sorry if that's confusing.  Also, we are re-using variable names, which a good programmer doesn't do, but this is just messing around in an iPython session, so it doesn't matter.
Can you write Python code for it?

inner = pi/2**5
outer = pi/2

du, dv = 0.05*pi, 0.01*pi
u, v = mgrid[inner:outer+du:du, 0:2*pi+dv:dv]


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Agonist Only
To calculate dose-response relation in the case of agonist only, we use general dose-response relation equation described previously. The function is shown below.
Can you write Python code for it?

# Calculate dose-response relation (DRR) for agonist only
# c     : Drug concentration(s) in nanomolar (nM)
# EC_50 : 50% effective concentration in nanomolar (nM)
# F     : Efficacy (unitless)
# n_H   : Hill coefficients (unitless)
def calc_drr(c, EC_50 = 20, F = 1, n_H = 1):
    r = (F * (c ** n_H) / ((c ** n_H) + (EC_50 ** n_H)))
    return r


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Agonist Plus Competitive Antagonist
Compatitive antagonist, as the name sugest, competes with agonist molecules to sit in the same pocket. It makes the binding harder for agonist as well as to trigger the activation. Therefore, higher agonist concentration is required to reach both full and partial (like $EC_{50}$) activation. New $EC_{50}$ value, called $EC_{50}'$ ($EC_{50}$ prime) is calculated using following formula.
$$EC_{50}' = EC_{50} * \left(1 + \frac{c_i}{K_i}\right)$$
It depends on inhibitor concentration ($c_i$) and dissociation constant of the inhibitor ($K_i$).
Following is a new function to calculate drug response of agonist with competitive antagonist. It shows new $EC_{50}$ value (EC_50_prime) replacing agonist only $EC_{50}$ value (EC_50).
Can you write Python code for it?

# Calculate dose-response relation (DRR) for agonist plus competitive antagonist
#  - Agonist
#    c     : Drug concentration(s) in nanomolar (nM)
#    EC_50 : 50% effective concentration in nanomolar (nM)
#    F     : Efficacy (unitless)
#    n_H   : Hill coefficients (unitless)
#  - Antagonist
#    K_i   : Dissociation constant of inhibitor in nanomolar (nM)
#    c_i   : Inhibitor concentration in nanomolar (nM)
def calc_drr_agonist_cptv_antagonist(c, EC_50 = 20, F = 1, n_H = 1, K_i = 5, c_i = 25):
    EC_50_prime = EC_50 * (1 + (c_i / K_i))
    r = calc_drr(c, EC_50_prime, F, n_H)
    return r


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Dose-Response Relations $^{[1]}$
Generally, dose-response relations can be written as follow. In which the dose is represented as concentration ($c$), while the formula returns the response ($r$).
$$r = \frac{F.c^{n_H}}{{c^{n_H} + EC_{50}^{n_H}}}$$
Other terms like $EC_{50}$ is the effective concentration achieved at 50% of maximum response. Normally, efficacy ($F$) is normalized to one so that it is easier to make comparison among different drugs. Furthermore, if full agonist is defined to have efficacy equal to one, anything lower than one is treated to be partial agonist. Finally, Hill coefficients ($n_H$) defines the number of drug molecules needed to activate target receptor.
Drug Concentartion
Both linearly and logarithmically increased concentrations are used to study dose-response relations.

Linearly increased concentration (c_lin):
Can you write Python code for it?

c_lin = np.linspace(0,100,101) # Drug concentration in nanomolar (nM)
plot_concentration(c_lin)


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Following result shows drug response of agonist only to the linearly increased concentrations.
Can you write Python code for it?

c = c_lin   # Drug concentration(s) in nanomolar (nM)
EC_50 = 20  # 50% effective concentration in nanomolar (nM)
F = 1       # Efficacy (unitless)
n_H = 1     # Hill coefficients (unitless)
r = calc_drr(c, EC_50, F, n_H)
plot_dose_response_relation(c, r, "Agonist")


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Following result shows drug response of agonist only to the logarithmically increased concentrations.
Can you write Python code for it?

c = c_log   # Drug concentration(s) in nanomolar (nM)
EC_50 = 20  # 50% effective concentration in nanomolar (nM)
F = 1       # Efficacy (unitless)
n_H = 1     # Hill coefficients (unitless)
r = calc_drr(c, EC_50, F, n_H)
plot_dose_response_relation(c, r, "Agonist", log_flag = True)


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Following result shows drug response of agonist with competitive antagonist to the linearly increased concentrations.
Can you write Python code for it?

c = c_lin   # Drug concentration(s) in nanomolar (nM)
EC_50 = 20  # 50% effective concentration in nanomolar (nM)
F = 1       # Efficacy (unitless)
n_H = 1     # Hill coefficients (unitless)
r_a = calc_drr(c, EC_50, F, n_H)

K_i = 5     # Dissociation constant of inhibitor in nanomolar (nM)
c_i = 25    # Inhibitor concentration in nanomolar (nM)
r_aca = calc_drr_agonist_cptv_antagonist(c, EC_50, F, n_H, K_i, c_i)

plot_dose_response_relation(c, r_a, "Agonist Only", r_aca, "Plus Antagonist")


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Following result shows drug response of agonist with competitive antagonist to the logarithmically increased concentrations.
Can you write Python code for it?

c = c_log   # Drug concentration(s) in nanomolar (nM)
EC_50 = 20  # 50% effective concentration in nanomolar (nM)
F = 1       # Efficacy (unitless)
n_H = 1     # Hill coefficients (unitless)
r_a = calc_drr(c, EC_50, F, n_H)

K_i = 5     # Dissociation constant of inhibitor in nanomolar (nM)
c_i = 25    # Inhibitor concentration in nanomolar (nM)
r_aca = calc_drr_agonist_cptv_antagonist(c, EC_50, F, n_H, K_i, c_i)

plot_dose_response_relation(c, r_a, "Agonist Only", r_aca, "Plus Antagonist", log_flag = True)


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Housekeeping Functions
Following function plots different concentrations used to visualize dose-response relation.
Can you write Python code for it?

def plot_concentration(c):
    fig = plt.figure()

    sp111 = fig.add_subplot(111)
    # Display grid
    sp111.grid(True, which = 'both')
    # Plot concentration
    len_c = len(c)
    sp111.plot(np.linspace(0,len_c-1,len_c), c, color = 'gray', linewidth = 2)
    # Label
    sp111.set_ylabel('Concentration (nM)')

    # Set X axis within different concentration
    plt.xlim([0, len_c-1])

    plt.show()


I am working on the file "Equations/Dose-Response Relations/Dose-Response Relations.ipynb".
The first task is:
Logarithmically increased concentration (c_log):
Can you write Python code for it?

c_log = np.logspace(0,5,101)   # Drug concentration in nanomolar (nM)
plot_concentration(c_log)


I am working on the file "Erin - Midterm.ipynb".
The first task is:
Features class
Can you write Python code for it?

'''
Features class:
    Static vars:
        * traitCounts - an array with length equal to the feature count. Each
            element holds an integer representing the number of possible traits
            for the corresponding feature.

    Object vars:
        * curTraits - an array with length equal to the feature count. Each
            element holds the current trait value for the corresponding feature.
            For the purposes of this model, curTraits[0] represets the binary
            trait for the prisionization feature.

    Static methods:
        * init(traitCounts, prisPct) - sets the feature count, traitCounts,
            and initial relative prisionization.

    Object methods:
        * randomizeTraits - sets random traits for each feature in the object
        * setTrait - sets the trait of a selected feature.
'''
class Features(object):

    # Initialize the feature count and the trait ranges for those features
    @staticmethod
    def init(traitCounts):
        Features.count = len(traitCounts)
        Features.traitCounts = traitCounts

    def __init__(self):
        # Initialize an empty array with a location for each current trait
        self.curTraits = [0 for i in range(Features.count)]
        self.randomizeTraits()
        self.setTrait(0,0)

    def randomizeTraits(self):
        for i in range(1, Features.count):
            self.curTraits[i] = random.randint(0, self.traitCounts[i]-1)

    def setTrait(self, which, val):
        self.curTraits[which] = val