Problem of Vanishing Gradients in GRU in Seq2Seq model

username: Mehran_tgn
cooked: <p>Hi,<br>
I was training an encoder-decoder model where the encoder model has only a GRU, and the decoder is a simple MLP which is given the hidden state of GRU as the input. My data is stock market prices (Open, High, Low, Close). The inputs to the encoder are sequences of OHLC prices (say for example 20 OHLCs where 20 is the window size). The sequence in each window is independent of other windows. During the training of the model, after some iterations, all the gradients become zero (gradients of the encoder) and the hidden states become all the same. I initialize the hidden state to zero before feeding the input to the encoder for each input. Here is my encoder model:</p>
<pre><code class="lang-auto">class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, device):
        super(EncoderRNN, self).__init__()
        self.device = device
        self.hidden_size = hidden_size
        self.gru = nn.GRU(input_size, hidden_size)

    def forward(self, x):
        if len(x.shape) &lt; 3:
            x = x.unsqueeze(1)

        hidden = self.initHidden(x.shape[1])

        output, hidden = self.gru(x, hidden)
        return output, hidden

    def initHidden(self, batch_size):
        return torch.zeros(1, batch_size, self.hidden_size, device=self.device)

</code></pre>
<p>I feed data to the model in batches, and x.shape[1] shows the batch size.<br>
I would be appreciated if someone can help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Usama_Hasan
cooked: <p>Hello, If it’s a gradient vansihing problem, this can be solved using clipping gradient.<br>
You can do this using by registering a simple backward hook.</p>
<pre><code class="lang-auto">clip_value = 0.5
for p in model.parameters():
    p.register_hook(lambda grad: torch.clamp(grad, -clip_value, clip_value))
</code></pre>

username: Mehran_tgn
cooked: <p>Thank you so much for your answer<br>
Actually I clip gradients in each iteration. Here is the code:</p>
<pre><code class="lang-auto">        for param in model.parameters():
            param.grad.data.clamp_(-1, 1)
</code></pre>
<p>Can the problem be related to the input data?</p>

username: Usama_Hasan
cooked: <p>Hey will you try initializing weights of the model.</p>

username: Mehran_tgn
cooked: <p>Sorry, Usama_Hasan but how can it help?</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/mehran_tgn">@Mehran_tgn</a>, it’s better to have weights initialized for better convergence</p>

username: Mehran_tgn
cooked: <p>Hi <a class="mention" href="/u/usama_hasan">@Usama_Hasan</a><br>
I have initialized my weights using a Normal distribution but that didn’t work either. My model diverges after some iterations. I think it’s because of the loss function. I used Smooth L1 Loss. I should try it to make sure if the problem is with loss function or not. This problem became so ambiguous to me.</p>

username: Mehran_tgn
cooked: <p>I have solved the problem :))<br>
It was because I haven’t normalized my input data. In this problem, I have multiple models including MLP, Convolutional and LSTM/GRU. The MLP and Convolutional models’ performance was good without normalization. That’s why I haven’t normalized my input data before.</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/mehran_tgn">@Mehran_tgn</a>, It’s good to hear that you have solved your problem, it’s sometimes really hard to solve a problem or find a fix without all the details. My advice try to make a post with all the possible things which can cause such a behaviour. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Mehran_tgn
cooked: <p>Hi <a class="mention" href="/u/usama_hasan">@Usama_Hasan</a><br>
Thank you so much<br>
I have tried some other stuff which can be related to this one. I summarize all the stuff I have done here:</p>
<ol>
<li>Sometimes, your data consists of independent sequences, meaning that you have a window that moves over your data. The data inside the window is a sequence, and sequences of data in different windows are independent. This way, you should <code>detatch_()</code> the hidden part in LSTM or GRU so that the history would be cut. Otherwise, in backpropagation through time, the backprop would go all the way to the beginning of the input data history.</li>
<li>If your data contains very different values, such as Stock Market prices, try to normalize your data first. You can do it by, for example, <code>MinMaxScaler()</code> in <code>sklearn</code>  library.</li>
<li>Don’t forget to do gradient clipping in your model when using sequential data</li>
</ol>

username: RachelShalom
cooked: <p>hey , thank you for your elaborated answer. I also have the vanishing gradient problem with a network of 2 CNN layers by two stacked GRus. my data is sensors and fed to the network in windows each window has 4 channels and length 1450 ( sensors)<br>
How did you normalise your data?<br>
is it normalising each window using the values of only this window or relatively to all samples in the the daaa?</p>