Adding More Linear Layers in the RNN

username: Ahx
cooked: <p>Dear Friends</p>
<p>I’m following the <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="nofollow noopener">tutorial</a></p>
<p>In the Exercise section, you will see:<br>
-&gt; Add more linear layers</p>
<p>Does adding more linear layer means, creating hidden-to-hidden layers?</p>
<p>For instance, below is the tutorial code:</p>
<pre><code class="lang-auto">class RNN(Module):
    def __init__(self, input_size, hidden_size,
                 output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        size_sum = input_size + hidden_size
        self.i2h = Linear(size_sum, hidden_size)
        self.h2o = Linear(size_sum, output_size)
        self.softmax = LogSoftmax(dim=1)

    def forward(self, input_, hidden_):
        combined = cat(tensors=(input_, hidden_), dim=1)
        hidden_ = self.i2h(input=combined)
        output = self.h2o(input=combined)
        output = self.softmax(input=output)
        return output, hidden_

    def init_hidden(self):
        return zeros(1, self.hidden_size)
</code></pre>
<p>If I add a hidden-to-hidden layer, does it mean I’m adding more linear layers or am I mistaken?</p>
<pre><code class="lang-auto">class RNN(Module):
   def __init__(self, input_size, hidden_size,
                output_size):
       super(RNN, self).__init__()
       self.hidden_size = hidden_size
       size_sum = input_size + hidden_size
       self.i2h = Linear(size_sum, hidden_size)
       self.h2h = Linear(hidden_size, hidden_size)
       self.h2o = Linear(hidden_size, output_size)
       self.softmax = LogSoftmax(dim=1)

   def forward(self, input_, hidden_):
       combined = cat(tensors=(input_, hidden_), dim=1)
       hidden_ = self.i2h(input=combined)
       hidden_ = self.h2h(input=hidden_)
       hidden_ = self.h2h(input=hidden_)
       hidden_ = self.h2h(input=hidden_)
       hidden_ = self.h2h(input=hidden_)
       hidden_ = self.h2h(input=hidden_)
       output = self.h2o(input=hidden_)
       output = self.softmax(input=output)
       return output, hidden_

   def init_hidden(self):
       return zeros(1, self.hidden_size)
</code></pre>
<p>The reason I’m asking is when I run the second RNN Module my accuracy is 8% lower than the previous module. I may be interpreting adding a more linear layer statement wrong.</p>
<p>Thanks</p>

username: vdw
cooked: <p>You call <code> hidden_ = self.h2h(input=hidden_)</code> multiple times in your <code>forward()</code> method. I don’t think that make sense. Try doing it only once.</p>
<p>But yes, “adding more linear layers” is rather ambiguous. A arguably harmless addition would be to add a linear layer between <code>self.h2o</code> and <code>self.softmax</code></p>
<p>Also note that you need an non-linear activation function between linear layers (e.g., <code>ReLU</code>). I mean it works without it, of course, but you essentially give up the benefits of additional layers.</p>

username: Ahx
cooked: <p>Dear Chris</p>
<p>From what I gather, I need to code between the output -&gt; softmax layers</p>
<pre><code class="lang-auto">class RNN(Module):
    def __init__(self, input_size, hidden_size,
                 output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        size_sum = input_size + hidden_size
        self.i2h = Linear(size_sum, hidden_size)
        self.h2o = Linear(hidden_size, output_size)
        self.o2h = Linear(output_size, hidden_size)
        self.softmax = LogSoftmax(dim=1)

    def forward(self, input_, hidden_):
        combined = cat(tensors=(input_, hidden_), dim=1)
        hidden_ = self.i2h(input=combined)
        output = self.h2o(input=hidden_)
        hidden_ = self.o2h(input=output)
        hidden_ = relu(input=hidden_)
        output = self.h2o(input=hidden_)
        return output, hidden_

    def init_hidden(self):
        return zeros(1, self.hidden_size)
</code></pre>
<p>When I made the network like above, it is not learning anything. (0% accuracy)</p>
<p>I think, I need more time to think about where I’m doing it wrong.</p>
<p>Thanks for the answer.</p>

username: vdw
cooked: <p>Everything works when you use the code given in the tutorial?</p>
<p>I cannot see any main issue why adding a layer would fail. Well you seem to miss</p>
<pre><code>output = self.softmax(output)
</code></pre>
<p>in your <code>forward()</code> method.</p>
<p>In practice, an accuracy of 0% can’t be right. The tutorial shows a classification task, so just by chance some predictions should be correct. Without any training, the accuracy of a classification model us usually around <code>1/#num_classes</code> which more or less represents guessing.</p>