Multinomial logit - loss oscillates

Aurelie_Sotura: <p>Hi everyone,</p>
<p>I am relatively new to Pytorch. I try to estimate parameters of a multinomial logit. I use Pytorch because I have a lot of observations and my problem is rather complex. I found strange results so I decided to test my code with a really simple logit problem with simulated data. However even with these simulated data and this totally simple probleme my loss still oscillates and my weight does not converge to the true value of my simulated data… I tested my data with Stata a statistical software package and it works perfectly fine (Stata finds the true value in less than one second). So I know the problem dont come from the simulated data.</p>
<p>here is my code:</p>
<p>import pandas as pd<br>
import numpy as np<br>
import os<br>
from torch.utils.data import Dataset<br>
import torch<br>
from torch.utils.data import DataLoader<br>
from torch import nn<br>
from torch.nn import functional as F<br>
from time import time<br>
from torch.optim import Adam, SGD , ASGD , lr_scheduler<br>
from tqdm.autonotebook import tqdm, trange<br>
from torch.utils.data import DataLoader<br>
import matplotlib.pyplot as plt<br>
from torch.utils.tensorboard import SummaryWriter</p>
<h3>Hyperparameters</h3>
<p>LEARNING_RATE = 0.1<br>
OPTIMIZER_CONSTRUCTOR = SGD</p>
<p>BATCH_SIZE = 32<br>
WORKERS = 0</p>
<p>path =[…]</p>
<p>class RebateData(Dataset):<br>
def <strong>init</strong>(<br>
self, file<br>
):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.file = torch.tensor(pd.read_csv(file).values , dtype = int)
    
def __getitem__(self, i):

    typeo = self.file[i]
    return [typeo]


def __len__(self):
    return len(self.file)
</code></pre>
<p>class Model(nn.Module):</p>
<pre><code>def __init__(self):
    super().__init__()
    self.dwelling_bias = nn.Parameter(torch.tensor([-0.5], dtype = torch.float) )


def forward(self, typeo):
    utility = (torch.cat([torch.zeros(1), self.dwelling_bias])[None, :]* torch.ones(typeo.size()[0])[:,None])[:,:,None]
    prob = F.softmax(utility, dim=1)

    avg_prob_choice = prob[torch.arange(len(typeo[:,0])), typeo[:,0],:].mean(-1)
    return -torch.log(avg_prob_choice ).mean()
</code></pre>
<p>dataset = RebateData( file = path + “try.csv” )<br>
dataloader = DataLoader(dataset, batch_size=BATCH_SIZE,  num_workers=WORKERS,<br>
shuffle=True, pin_memory=True, )<br>
model = Model()</p>
<p>optimizer = OPTIMIZER_CONSTRUCTOR(model.parameters() , lr=LEARNING_RATE)<br>
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer,<br>
‘max’ , verbose = True , patience = 1 )<br>
t = 0</p>
<p>log_name = f"{LEARNING_RATE}, {OPTIMIZER_CONSTRUCTOR.<strong>name</strong>}"<br>
runs = “C:\Users\l\Desktop\runs”<br>
writer = SummaryWriter(log_dir=f"{runs}{log_name}")</p>
<p>try:<br>
for epoch in range(10):<br>
for batch in dataloader:<br>
optimizer.zero_grad()<br>
loss = model(*batch)<br>
loss.backward()<br>
print(model.dwelling_bias.grad)<br>
optimizer.step()<br>
scheduler.step(loss)<br>
print(“lr”,epoch,optimizer.param_groups[0][‘lr’])<br>
if ( optimizer.param_groups[0][‘lr’] &lt; 0.001) :<br>
raise StopIteration<br>
t+=1<br>
##        if t  % 20 == 0 :<br>
##            print("dwelling type : ancien ",model.dwelling_bias.detach().numpy())<br>
##            print(“sigma”,model.sigma.detach().numpy())<br>
##            print("loss ",loss)<br>
##        writer.add_scalar(‘sigma’, model.sigma.detach().numpy(), global_step=t)<br>
##        print(writer)<br>
writer.add_scalar(‘dwelling type : ancien’, model.dwelling_bias.detach().numpy(), global_step=t)<br>
writer.add_scalar(‘learning rate’, optimizer.param_groups[0][‘lr’], global_step=t)<br>
writer.add_scalar(‘loss’, loss, global_step=t)</p>
<p>except StopIteration: pass<br>
writer.close()</p>
<p>The true value of dwelling bias in my data is -1, but Pytorch cant find it.</p>
<p>If anyone can help me on this,</p>
<p>Thanks a lot in advance…</p>

ccook: <p>This is an old post and it’s a bit hard to read the code in this format, but usually oscillations in simple models come from a learning rate that is too large. You may also want to use a learning rate scheduler to decrease the learning rate over time.</p>