Improve my training iteration

shwe87: <p>Hello,</p>
<p>I have created the following model:</p>
<pre><code class="lang-auto">def __init__(self, input_size, output_size, hidden_dim, n_layers, n_feats, drop_prob=0.5):
        super(MySpeechRecognition, self).__init__()
        #output_dim = will be the alphabet + '' and space = 28 chars
        self.input_size = input_size
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers
        self.drop_prob = drop_prob
        self.output_dim = output_size
        

        # GRU Layer --&gt; input (batch, channel*features, time)
        # Input size = number of features
        self.gru = nn.GRU(input_size, hidden_dim, n_layers, batch_first=True, dropout=drop_prob)
        # shape output (batch, channel*features, time * hidden_size)
        self.layer_norm = nn.LayerNorm(n_feats)
        # (batch, channel, features, time)
        #Fully Connected 
        self.fc1 = nn.Linear(self.hidden_dim,512)
        self.fc2 = nn.Linear(512, self.output_dim)
        self.dropout = nn.Dropout(0.2)
</code></pre>
<pre><code class="lang-auto">input_size = 128
n_classes = 29
hidden_dim = 250
n_layers = 2
n_feats = 128
batch_size = 32
</code></pre>
<p>And this is my training loop:</p>
<pre><code class="lang-auto">def train(n_epochs, train_loader, valid_loader, model, optimizer, criterion, clip, save_path):
    data_len = len(train_loader.dataset)
    t0 = time.time()
    model.train()
    for e in range(n_epochs):
        #Initialize hidden state
        
        running_losses = 0.0
        test_losses = 0.0
        loss_values = []
        test_loss_values = []



        h = model.init_hidden(batch_size)

        #batch loop
        for batch_idx, _data in enumerate(train_loader):

            specs, labels, input_lengths, label_lengths = _data
            #print("1")
            if (train_on_gpu):
                specs, labels = specs.cuda(), labels.cuda()
            # Break if it is the last batch or the length is not the same as batch size because otherwise it will get error.
            if (len(specs) != batch_size):
                break
            # Creating new variables for the hidden state, otherwise
            # we'd backprop through the entire training history
            h = h.data
            # zero accumulated gradients
            optimizer.zero_grad()

            # get the output from the model

            output, h = model(specs, h)
            output = F.log_softmax(output, dim=2)
            output = output.transpose(0,1)
            # calculate the loss and perform backprop
            loss = criterion(output, labels.float(), input_lengths, label_lengths)
            loss.backward()
            # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
            nn.utils.clip_grad_norm_(model.parameters(), clip)
            optimizer.step()
            scheduler.step()


            # loss stats
            running_loss =+ loss.item()*specs.size(0)


            if batch_idx % 100 == 0 or batch_idx == data_len:
                # Get Validation loss

                val_losses = []
                model.eval()
                val_h = model.init_hidden(batch_size)

                for batch_idx_v, _data in enumerate(valid_loader):

                    val_h = val_h.data

                    specs, labels, input_lengths, label_lengths = _data

                    #print(len(specs))
                    if (len(specs) != batch_size):
                        break
                    if(train_on_gpu):
                        specs, labels = specs.cuda(), labels.cuda()

                    with torch.no_grad():
                        output, h = model(specs, h)
                        output = F.log_softmax(output, dim=2)
                    output = output.transpose(0,1)
                    val_loss = criterion(output, labels.float(), input_lengths, label_lengths)
                    test_losses =+ val_loss.item()*specs.size(0)
                model.train()

                print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}\Valid Loss: {:.6f}'.format(
                        e+1, batch_idx * len(specs), data_len,
                        100. * batch_idx / len(train_loader), loss.item(), val_loss.item()))

        loss_values.append(running_loss/data_len)
        test_loss_values.append(test_losses)
        torch.save(model.state_dict(), save_path)
        print('Epoch {} + took {} seconds'.format(e+1, time.time() - t0))
    return (model, loss_values, test_loss_values)
</code></pre>
<p>I am not sure if I can improve the training iteration in any way. It is taking too long even for 10 epochs…</p>
<p>I tried both with CPU and GPU on kaggle.</p>
<p>Thanks.</p>

ptrblck: <p>You could try to profile the code and determine where the bottleneck is at the moment.<br>
E.g. if the data loading is the bottleneck, you could potentially increase the number of workers, if that’s not already done. I don’t know what kind of systems are used on Kaggle, so I cannot comment on the data loading speed etc.</p>
<p>Unrelated to this issue, but you should not use the <code>.data</code> attribute, but instead call <code>.detach()</code> on the tensor you would like to detach from the computation graph.</p>

shwe87: <p>Thank you so much!</p>
<p>I didn’t know about .detach().</p>

Chirath_Pansilu: <p>Sir , is it necessary to  use previous batch’s hidden state as , initial hidden state to the next batch .<br>
Should not we use typically all zeros as the initial hidden state ?</p>
<p>something like this --&gt;</p>
<pre><code class="lang-auto">output, h = model( specs, None) 
</code></pre>
<p>Thank You.</p>

shwe87: <aside class="quote no-group" data-username="shwe87" data-post="1" data-topic="84770">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shwe87/48/25270_2.png" class="avatar"> shwe87:</div>
<blockquote>
<p><code>h = model.init_hidden(batch_size)</code></p>
</blockquote>
</aside>
<p>I initialized the hidden weights with zeros at first:</p>
<p>h = model.init_hidden(batch_size)</p>
<p>The init_hidden function is inside the model’s class<br>
if (train_on_gpu):<br>
hidden = (torch.zeros(self.n_layers<em>self.n_direction, batch_size, self.hidden_dim).zero_()).cuda()<br>
else:<br>
hidden = (torch.zeros(self.n_layers</em>self.n_direction, batch_size, self.hidden_dim).zero_())</p>

shwe87: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have another question for detach() and data</p>
<p>For initializing weights, should I also use detach() rather than data?</p>
<p>xavier(m.weight.detach())</p>
<p>or<br>
xavier(m.weight.data)</p>

ptrblck: <p>You should initialize the parameters directly via <code>xavier(m.weight)</code>. There is no need to call detach, as this operation shouldn’t be tracked by Autograd.</p>