Why is Pytorch OK with this wierd operation and not fail?

username: Shisho_Sama
cooked: <p>Please take a moment and have a look at the notebook <a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/recurrent-neural-networks/char-rnn/Character_Level_RNN_Solution.ipynb" rel="nofollow noopener">here</a>. This is a simple tutorial concerning LSTMs taught at Udemy’s Pytorch Course.<br>
There are two sections in this IPython notebook that confuses me greatly.</p>
<ol>
<li>Why is it necessary to use contiguous() when using an LSTM?<br>
and more importantly :</li>
<li>Why doesnt the training procedure fail! becasue of the following code snippet :</li>
</ol>
<pre><code class="lang-auto">    counter = 0
    n_chars = len(net.chars)
    for e in range(epochs):
        # initialize hidden state
        h = net.init_hidden(batch_size)
        
        for x, y in get_batches(data, batch_size, seq_length):
            counter += 1
            
            # One-hot encode our data and make them Torch tensors
            x = one_hot_encode(x, n_chars)
            inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
            
            if(train_on_gpu):
                inputs, targets = inputs.cuda(), targets.cuda()

            # Creating new variables for the hidden state, otherwise
            # we'd backprop through the entire training history
            h = tuple([each.data for each in h])

            # zero accumulated gradients
            net.zero_grad()
            
            # get the output from the model
            output, h = net(inputs, h)
            print(f'output.shape: {output.shape}')
            print(f'y.shape :{targets.shape}')
            print(targets[0,:])
            # calculate the loss and perform backprop
            loss = criterion(output, targets.view(batch_size*seq_length).long())
            loss.backward()
            # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
            nn.utils.clip_grad_norm_(net.parameters(), clip)
            opt.step()
            
            # loss stats
            if counter % print_every == 0:
</code></pre>
<p>What I’m specifically refering to is this line:</p>
<pre><code class="lang-auto">loss = criterion(output, targets.view(batch_size*seq_length).long())

</code></pre>
<p>basically, here the author is using a one-hot encoded output with the shape <code>(batch, sequence_length, features)</code> with a normal not-one-hot encoded target tensor of shape <code>(batchsize, sequence_length)</code>!<br>
why does it not fail? how is <code>crossentropy</code> doing its job when the two tensors are not both one-hot  encoded?!<br>
if you go and one-hot encode the targets as well, you will face the error :<br>
<code>RuntimeError: multi-target not supported at C:/w/1/s/windows/pytorch/aten/src\THCUNN/generic/ClassNLLCriterion.cu:15</code></p>
<p>The usage of contiguous() seems not to do any good and the only way to get this to work seems like this!</p>
<p>also have a side question, what does  <code>weight = next(self.parameters()).data</code> mean?<br>
Why did t he author do  :</p>
<pre><code class="lang-auto">weight = next(self.parameters()).data
hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                weight.new(self.n_layers, batch_size, self.n_hidden).zero_())
</code></pre>
<p>whats this weight.new? how does he/she know what parameters to send? why did not the author simply use tensor.zeros() instead and do :</p>
<pre><code class="lang-auto">hidden_state = torch.zeros(num_layer*direction, batch_size, hidden_size).to(device)
cellstate = torch.zeros_like(hidden_state).to(device)
hiddenstates = (hidden_state,cellstate)
</code></pre>
<p>Can anyone please explain to me what is happening here?<br>
I grealy appreciate it</p>

username: sonsus
cooked: <aside class="quote no-group" data-username="Shisho_Sama" data-post="1" data-topic="47651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/48/6926_2.png" class="avatar"> Shisho_Sama:</div>
<blockquote>
<p>basically, here the author is using a one-hot encoded output with the shape <code>(batch, sequence_length, features)</code> with a normal not-one-hot encoded target tensor of shape <code>(batchsize, sequence_length)</code> !</p>
</blockquote>
</aside>
<p>For this part what I can tell you is that <code>nn.CrossEntropyLoss()</code> does take <code>output</code> as one-hot encoded and <code>targets</code> with indices that isn’t encoded into one-hot. Internally it converts it into one-hot encoding and computes the loss.</p>
<p>You can find this easily on documentation and its source code.</p>