Intro to pytorch I: Stochastic gradient
By: Michael Ruggiero
Tags: Linear Regression, Matrix, Python3, Pytorch, Tensor

1. Foundations

Careful with empty arrays, they contain random junk in memory. For true zeros use torch.zeros:

import torch #input

e = torch.empty(2,2)

print(e) #output

tensor([[ 0.0000e+00, -1.0842e-19],

[-5.9247e-01, 3.6902e+19]]) #input

z = torch.zeros(2,2)

print(z) #output

tensor([[0., 0.],

[0., 0.]])

numpy and pytorch play nice with each other:

import numpy as np #input: from numpy to tensor

nump = torch.tensor(np.linspace(0,3,5))

print(nump) #output

tensor([0.0000, 0.7500, 1.5000, 2.2500, 3.0000], dtype=torch.float64) #input: from torch to numpy

nump.numpy() #output

array([0. , 0.75, 1.5 , 2.25, 3. ])

Explicit tensor creation is a cake walk:

#Explicit creation of 2x2x2 tensor:

t3 = torch.tensor([[[1,2],[3,4]],[[5,6],[7,8]]], dtype = torch.int) # this method creates a new tensor "y" that has the same datatype as # the original tensor t3

y = t3.new_ones(2,1) #Cross Product

t3 @ y #output

tensor([[[ 3],

[ 7]],



[[11],

[15]]], dtype=torch.int32)

Dot product:

#input

t3 * t3 #output

tensor([[[ 1, 4],

[ 9, 16]],



[[25, 36],

[49, 64]]], dtype=torch.int32) #The `*_` method works like `inplace=True` in pandas #input

t3.add_(t3) #output

tensor([[[ 2, 4],

[ 6, 8]],



[[10, 12],

[14, 16]]], dtype=torch.int32)

2. Autograd

Autograd allows automatic differentiation for all operations on Tensors.

# Create a tensor and set requires_grade=True

# to track computation with it

x = torch.ones(3,2, requires_grad=True) # Because y was created as a result of an operation,

# it also has a grad_fn method

y = 3*x - 4

print(y) #output

tensor([[-1., -1.],

[-1., -1.],

[-1., -1.]], grad_fn=<SubBackward0>) # We can do more to y (and thus x) and calculate its derivatives

z = 2*y**2

w = z.mean() print("z: {},



w: {}".format(z, w)) #output

z: tensor([[2., 2.],

[2., 2.],

[2., 2.]], grad_fn=<MulBackward0>),



w: 2.0 #Computes the gradient of current tensor

w.backward()

x.grad #output

tensor([[-2., -2.],

[-2., -2.],

[-2., -2.]])

3. LR With Stochastic Gradient Descent

The starting point for much of this notebook is based notes I transcribed from Dr. Alvarez’s excellent ODSC presentation: “Understanding the PyTorch Framework with Applications to Deep Learning.”

Step #1: Basic Definitions of Data Space

n_data = 10_000

X = torch.ones(n_data,2) #populate random points into X

X[:,0].uniform_(-1.15,1.15) actual = torch.tensor([np.random.uniform(-14,14),

np.random.uniform(-14,14)]) y = X @ actual + torch.randn(n_data)/3

Step #2: Train/Test Split

There is no auto-covariance in this toy data dataset, therefore a simple train test split will suffice.

#Train test split

train_size = int(n_data * .75)

test_size = n_data - train_size X_train, X_test = X[0 : train_size, :], X[train_size : n_data, :]

y_train, y_test = y[0 : train_size], y[train_size : n_data] import matplotlib.pyplot as plt plt.scatter(X_train[:,0], y_train, s = .1);

Step #3: Define Scoring Function

def mean_standard_error(y, y_pred):

return ((y - y_pred) ** 2).mean()

Step #4: Start with a arbitrary model to begin training

#A random prediction

model = torch.tensor([np.random.uniform(-14,14),

np.random.uniform(-14,14)]) y_hat = X_train @ model

mean_standard_error(y_hat, y_train) plt.scatter(X_train[:,0],y_train, s = .1)

plt.scatter(X_train[:,0],y_hat, s = .1)

Step #5: Parameterize the model

import torch.nn as nn model = nn.Parameter(model)

model #output

Parameter containing:

tensor([8.6251, 6.5764], requires_grad=True)

Bringing Step 1–5 together

#A glorified stochastic gradient descent machine

def gentle_linear_torch(data, target, learning_rate, iterations):

answer = {}

def mean_standard_error(y_actual, y_pred):

return ((y_actual - y_pred) ** 2).mean()



#Random Initial Model

model = nn.Parameter(torch.tensor([np.random.uniform(-14,14),

np.random.uniform(-14,14)])) y_hat = data @ model

mean_standard_error(y_hat, target) for time in range(iterations):

y_hat = data @ model

loss = mean_standard_error(target, y_hat)



if time > 0 and last_loss == loss:

print("

convergence of SGD at step {}".format(time))

return(model)

break



if time > 0 and time % 5 == 0:

print("step: {:4} MSE: {:5}".format(time,

float(last_loss.data - loss.data)))

plt.scatter(data[:,0], target, s = .1, alpha = .05);

plt.scatter(data[:,0], y_hat.detach().numpy(), s = .1);



loss.backward()

with torch.no_grad():

#These *_ set values like inplace = True

model.sub_(learning_rate * model.grad)

model.grad.zero_()



last_loss = loss



return(model) sgd = gentle_linear_torch( X_train, y_train, .1, 1000)



y_hat = X_test @ sgd

float(mean_standard_error(y_hat, y_test)) #MSE output

0.10991530865430832 plt.scatter(X_test[:,0], y_test, s = .1, alpha = .5);

plt.scatter(X_test[:,0], y_hat.detach().numpy(), s = .1);

Plot on test set

Normal Equation Solution

Calculating the linear regression with Ordinary least squares (OLS) estimator:

theta = (torch.t(X_train) @ X_train).inverse() @ torch.t(X_train) @ y_train

print(theta)

print(sgd) #output

tensor([0.8584, 1.9531])

Parameter containing:

tensor([0.8580, 1.9532], requires_grad=True)

Done!

And there you have it. Let me know if you like these less verbose posts. Check out the next story in this series here.