Does reshaping/slicing a tensor with np.split break autograd graph?

username: kekgle
cooked: <p>Hi! I am working on a personal NLP project that involves word and character embeddings.</p>
<p>I want to merge together pre-computer word embeddings and word-embeddings built from character embeddings, through concatenation. The dimension of the pre-computed embeddings is [batch_size; sentence_length; 200], where batch_size is the number of sentences. The dimension for the char-based word embeddings is [number_of_words; word_length; 60], because when computing char embeddings I have to concatenate all words of all sentences in the batch to make the input 3D.</p>
<p>I also have, for char embeddings, the indexes that I can use to separate the words back into sentences. I reshape the output of the char embedding layer into [batch_size; sentence_length; 60] by doing this:</p>
<pre><code class="lang-auto">x_char = self.char_embeddings(x_char, x_char_mask)   # shape: [number_of_words; word_length; 60]
x_char = pad_sequence(np.split(x_char, splits[:-1]), batch_first=True)  # shape: [batch_size; sentence_length; 60]
</code></pre>
<p>In the snippet above, variable <code>splits</code> contains the position of the last token of each sentence. I am wondering if using <code>np.split</code> to slice the tensor will tamper with the computational graph built by autograd, blocking backprop to the char embedding LSTM. Can anybody help?</p>

username: phan_phan
cooked: <aside class="quote no-group" data-username="kekgle" data-post="1" data-topic="100125">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kekgle/48/30091_2.png" class="avatar"> kekgle:</div>
<blockquote>
<p>because when computing char embeddings I have to concatenate all words of all sentences in the batch to make the input 3D.</p>
</blockquote>
</aside>
<p>Are you sure you need to make the character input 3d ?<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#embedding" rel="noopener nofollow ugc">nn.Embedding</a> is supposed to work with any input shape.</p>

username: kekgle
cooked: <p>Hi phan_phan! I’m sorry I was not clear enough, to create the char-based word embeddings I first use the <code>nn.Embedding</code> module to encode each character into a vector; then I take all the characters of a word and feed it into a RNN (LSTM to be precise) to obtain the word embedding from the character embeddings. The RNN needs the input to be 3D so I have to “concatenate all the sentences” along the same dimension and then separate them again to merge the char-based word embeddings, with, say, GloVe word embeddings that I get from a separate layer.</p>

username: albanD
cooked: <aside class="quote no-group" data-username="kekgle" data-post="1" data-topic="100125">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kekgle/48/30091_2.png" class="avatar"> kekgle:</div>
<blockquote>
<p>I am wondering if using <code>np.split</code> to slice the tensor will tamper with the computational graph built by autograd</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>As a general rule, any op that is not provided by pytorch will break the graph (should ideally fail to run if the Tensor requires grad).<br>
But here you can simply use <code>torch.split</code> or <code>torch.chunk</code> and all will be differentiable</p>

username: kekgle
cooked: <p>Thanks! torch.split is exactly what I was looking for. Just a follow-up question based on your response. Is slicing via <code>[ ]</code> safe for autograd?</p>
<pre><code class="lang-auto">my_tensor[new_indices, :, :]  # can I re-order a 3D tensor like this? or should I use torch.index_select?
</code></pre>

username: albanD
cooked: <p>Yes it is.</p>
<p>In general in pytorch, all the differentiable function will return correct gradients or raise an error. So if something is differentiable and runs, you are safe and it computes the right thing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>