Initialise the parameters of a NN and then train it

username: fx12345
cooked: <p>Hi,</p>
<p>I read about an architecture of ReLU NN in Error bounds for approximations with deep ReLU networks and want to implement the neural network that is proposed in the paper, and then train it to see how the parameters of the neural network would change. But the parameters does not change after training, no matter what I initialise the parameters to. shouldn’t the parameters converge to a local minimum of the loss function?</p>
<p>I have pasted my code below</p>
<pre><code class="lang-auto">#this is the architecture i implemented in python
class yarotskyNet(nn.Module):
    def __init__(self, deg, matrix, vec):
        super(yarotskyNet, self).__init__()
        #changeble no. of hidden layers
        self.input =nn.Linear(1, 3)
        self.input.bias = nn.Parameter(torch.tensor([0,-0.5,-1.0], requires_grad=True))
        self.input.weight = nn.Parameter(torch.tensor([[1.0],[1.0],[1.0]], requires_grad=True))
        self.linears = nn.ModuleList([])
        for i in range(1, deg-1):
            linear = nn.Linear(3,3)
            w = [[2.0, -4.0, 2.0],
                [2.0, -4.0, 2.0],
                [2.0, -4.0, 2.0]]
            a = [[84.2, 33.02, 19.389],
                [35.0, 41.0, 54.0],
                [91.0, -33.0, 28.0]]
            b = [0,-0.5,-1.0]
            c = [1.3948,-8.2,3.048]
            linear.weight = nn.Parameter(torch.tensor(a), requires_grad=True)
            linear.bias = nn.Parameter(18.943*torch.tensor(c))
            self.linears.extend([linear])
        self.output = nn.Linear(3, 1)
        self.output.bias = nn.Parameter(torch.tensor([0.0]))
        self.deg = deg
        #w = [1.0, 1.0, 1.0]
        #self.output.weight = nn.Parameter(torch.tensor(w))
        self.output.weight = nn.Parameter(torch.tensor([[2.0, -4.0, 2.0]]))


    def forward(self, x):
        summation = x
        x = F.relu(self.input(x))
        i = 1
        for fc in self.linears:
           cst = 2**(2*i)
           i = i+1
           y = self.output(x)
           y = y/cst
           summation = summation - y
           x = F.relu(fc(x))
        x = self.output(x)
        x = x/2**(2*self.deg)
        return summation - x
   
``
``
def trainNet(epoch, sample_size, test_size, net):
    X = []
    y = []
    for i in range(sample_size):
        X.append([np.float64(random.random())])
        y.append(X[i][0]*X[i][0])
    X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=test_size)
    train_size = len(X_train)
    X_train = torch.Tensor(X_train)
    X_test = torch.Tensor(X_test)
    y_train = torch.Tensor(y_train)
    y_test = torch.Tensor(y_test)
    y_train = y_train.view(-1, 1)
    y_test = y_test.view(-1, 1)
    criterion = nn.MSELoss()
    optimiser = torch.optim.Adam(net.parameters())
    for j in range(epoch):
        y_res = net(X_train)
        loss = criterion(y_res, y_train)
        optimiser.zero_grad()
        loss.backward()
        optimiser.step()
    return net
``</code></pre>

username: paulk
cooked: <p>when i add some prints to your training function is appears to be training.</p>
<pre><code class="lang-auto">def trainNet(epoch, sample_size, test_size, net):
    X = []
    y = []
    for i in range(sample_size):
        X.append([np.float64(random.random())])
        y.append(X[i][0]*X[i][0])
    X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=test_size)
    train_size = len(X_train)
    X_train = torch.Tensor(X_train)
    X_test = torch.Tensor(X_test)
    y_train = torch.Tensor(y_train)
    y_test = y_train.view(-1, 1)
    y_train = y_train.view(-1, 1)
    
    criterion = nn.MSELoss()
    optimiser = torch.optim.Adam(net.parameters())
    
    before_weights = {name: param.clone() for name, param in net.named_parameters()}

    for j in range(epoch):
        y_res = net(X_train)
        loss = criterion(y_res, y_train)
        optimiser.zero_grad()
        loss.backward()
        optimiser.step()
        
        if j % 100 == 0:
            print(f"Epoch {j}, Loss: {loss.item()}")
            
    after_weights = {name: param.clone() for name, param in net.named_parameters()}
    for name in before_weights:
        print(f"Parameter {name} changed: {not torch.allclose(before_weights[name], after_weights[name])}")
            
    return net

net = yarotskyNet(deg=3, matrix=None, vec=None)
trained_net = trainNet(epoch=1000, sample_size=1000, test_size=0.2, net=net)
</code></pre>
<pre><code class="lang-auto">Epoch 0, Loss: 30.96532440185547
Epoch 100, Loss: 18.746919631958008
Epoch 200, Loss: 11.290495872497559
Epoch 300, Loss: 6.906811714172363
Epoch 400, Loss: 4.311870574951172
Epoch 500, Loss: 2.7773687839508057
Epoch 600, Loss: 1.8449206352233887
Epoch 700, Loss: 1.2643393278121948
Epoch 800, Loss: 0.9039990901947021
Epoch 900, Loss: 0.6738611459732056
Parameter input.weight changed: True
Parameter input.bias changed: True
Parameter linears.0.weight changed: True
Parameter linears.0.bias changed: True
Parameter output.weight changed: True
Parameter output.bias changed: True
</code></pre>
<p>also in your code y_test = y_train, seems to be a typo, maybe you want to change that once you start using y_test</p>
<p>hope that helps</p>

username: Dhia-naouali
cooked: <p>Hiii</p>
<p>the parameters does actually change using your code, yet it’s minimal changes for a few reasons,<br>
it seems like the gradient flow in this architecture isn’t the best (judging from the operatiosn in the <code>forward</code> method), the relatively too small default learning rate for the small model (small weights)</p>
<p>I’ve put together a bit cleaner code based on yours, included comments for the suggested changes and printed all parameter values after training, you can observe the changes there, it you want to monitor the parameter changes during training, just move the print calls inside the loop (it will be a bit messy tho, there are better ways to do so)</p>
<p>note: I used a learning rate <code>0.05</code> which I suggest is a good value for the model side &amp; adam, to see more variation in the parameters after training, consider using a higher value (<code>01</code> would do), you’ll end up with a higher loss at the end of the trianing loop tho</p>
<pre data-code-wrap="python"><code class="lang-python">

import random
import numpy as np
from sklearn.model_selection import train_test_split

import torch
from torch import nn
import torch.nn.functional as F

torch.set_default_dtype(torch.float64)

class yarotskyNet(nn.Module):
    # only the deg arg is being used
    def __init__(self, deg, **kwargs):
        super().__init__()
        self.deg = deg

        self.input = nn.Linear(1, 3)
        # the underlying tensor is accessable using .data 
        # and the weight / bias tensor is already created with requires_grad=True
        # no need to create a new weight / bias nn.Param override the already created one
        self.input.weight.data = torch.ones(3, 1)
        self.input.bias.data = torch.tensor([0., -.5, -1.])

        self.linears = nn.ModuleList()
        # assuming you wanted the linears list to be of length deg
        # range deg will return [0, 1, 2, ..., deg-1] which is of length deg
        for i in range(deg):
            linear = nn.Linear(3, 3)
            
            linear.weight.data = torch.tensor([
                [84.2, 33.02, 19.389],
                [35.0, 41.0, 54.0],
                [91.0, -33.0, 28.0],
            ])
            linear.bias.data = torch.tensor([1.3948,-8.2,3.048]) * 18.943

            # a more clean way to add a module / layer to your nn.ModuleList
            self.linears.append(linear)


        self.output = nn.Linear(3, 1)
        self.output.bias.data = torch.tensor([0.0])
        self.output.weight.data = torch.tensor([[2.0, -4.0, 2.0]])


    # kept it as it is
    def forward(self, x):
        summation = x
        x = F.relu(self.input(x))
        i = 1
        for fc in self.linears:
           cst = 2**(2*i)
           i = i+1
           y = self.output(x)
           y = y/cst
           summation = summation - y
           x = F.relu(fc(x))
        x = self.output(x)
        x = x/2**(2*self.deg)
        return summation - x
   
   
def trainNet(epoch, sample_size, test_size, net):

    # a better way to create random values is using torch.randn or np.random.randn
    # using torch.randn will get you torch tensors from initialization
    # no need to create numpy arrays or lists then convert them
    # plus, you'll create them in the correct shape
    x = torch.randn(sample_size, 1)
    y = x[:, 0] * x[:, 0] # shaped (sample_size,)
    y = y.unsqueeze(1) # shaped (sample_size, 1)

    x_train, x_test, y_train, y_test = train_test_split(x,y, test_size=test_size)

    print(f"x_train: {x_train.shape}")
    print(f"y_train: {y_train.shape}")
    print(f"x_test: {x_test.shape}")
    print(f"y_test: {y_test.shape}")

    criterion = nn.MSELoss()
    optimiser = torch.optim.Adam(net.parameters(), lr=5e-2)

    # let's print the loss each 10% of the training epochs, eventually 10 times during training
    print_loss_every = epoch // 10
    for j in range(epoch):
        y_res = net(x_train)
        loss = criterion(y_res, y_train)
        optimiser.zero_grad()
        loss.backward()
        optimiser.step()
        if j % print_loss_every == 0:
            print(f"loss at step {j+1:5}: {loss.item()}")

    return net


net = yarotskyNet(deg=3, matrix=None, vec=None)
trained_net = trainNet(epoch=1000, sample_size=1000, test_size=0.2, net=net)



end="\n"*4
print(end)

print(f"input.weight: {net.input.weight.data}")
print(f"input.bias: {net.input.bias.data}", end=end)


print("linears: ")
for i in range(net.deg):
    layer = net.linears[i]
    print(f"\t layer_{i}.weight: {layer.weight.data}")
    print(f"\t layer_{i}.bias: {layer.bias.data}", end="\n"*2)

print(end)

print(f"output.weight: {net.output.weight.data}")
print(f"output.bias: {net.output.bias.data}", end=end)


# import IPython; IPython.embed()

</code></pre>

username: fx12345
cooked: <p>Thank you very much for the reply!: D Seems like the training sectio works fine. Then it’s probably something wrong with the way I designed the neural network.n</p>

username: fx12345
cooked: <p>: D Thank you soooo much for the detailed explanation and code! So this means if I would like to see greater change in the parameters, then the forward method may need be written differently?</p>

username: Dhia-naouali
cooked: <p>the forward method depends on the network you’re trying to implement<br>
few things to point at in the code snippet :</p>
<ul>
<li>you’re training on random inputs (i.e training the netwro to overfit / memorize some random values</li>
<li>since you’re experimenting you’re using small layers / weights</li>
<li>the initial values you used, tho worked here, aren’t really the best state to start with (sampling from a random distribution (N(0, 1), or even better a propoer initialization (he, kaiming, …) for instance would yield better results / more visible change during training</li>
</ul>
<p>does that solve your issue ?</p>