RuntimeError: inplace operation

divyesh_rajpura: <p>Getting an below error when I execute my code. I have given the snippet of the code (from my loss function) which is causing an error. I think changing the value of <code>batch_spk_centroid[index1]</code> in <code>calc_cosine_sim_div()</code> is the main cause of an error. <code>batch_spk_centroid</code> has <code>require_grad=True</code>, do we really need <code>require_grad=True</code> for intermediate local variables of functions? Can someone suggest solution for this to handle this?</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [8, 256]], which is output 0 of torch::autograd::CopySlices, is at version 128; expected version 127 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<pre><code class="lang-auto">def forward(self, batch_embedding, batch_label):
        batch_embedding = batch_embedding.reshape((self.hp.train.num_spk_per_batch, self.hp.train.num_utt_per_spk, batch_embedding.size(1)))
        cos_sim_matrix = self.calc_cosine_sim_div(batch_embedding, batch_label)
        torch.clamp(self.w, 1e-6)
        torch.clamp(self.b, 1e-6)
        cos_sim_matrix = cos_sim_matrix * self.w + self.b
        loss = self.embed_loss(batch_embedding, cos_sim_matrix)
        return loss.sum()

    def calc_cosine_sim_div(self, batch_embedding, batch_label):
        batch_spk_centroid = torch.mean(batch_embedding, dim = 1)
        batch_spk_centroid = batch_spk_centroid.detach()
        cos_sim_matrix = []
        for index1, spk_embedding in enumerate(batch_embedding):
            for index2, utt_embedding in enumerate(spk_embedding):
                spk_centroid_org = copy.deepcopy(batch_spk_centroid[index1].data)
                batch_spk_centroid[index1] = ((batch_spk_centroid[index1] * self.hp.train.num_utt_per_spk ) - batch_embedding[index1][index2] ) / (self.hp.train.num_utt_per_spk - 1)
                cos_sim_matrix.append(self.cos_sim(utt_embedding.repeat(batch_embedding.shape[0],1), batch_spk_centroid))
                # batch_spk_centroid[index1] = ((batch_spk_centroid[index1] * (self.hp.train.num_utt_per_spk-1) ) + batch_embedding[index1][index2] ) / self.hp.train.num_utt_per_spk
                batch_spk_centroid[index1] = spk_centroid_org
        cos_sim_matrix = torch.stack(cos_sim_matrix).reshape(batch_embedding.shape[0],batch_embedding.shape[1],batch_embedding.shape[0])
        return cos_sim_matrix
</code></pre>

JuanFMontesinos: <pre><code class="lang-auto">copy.deepcopy(batch_spk_centroid[index1].data)
</code></pre>
<p>dont use deep copy<br>
tensor has a method called clone</p>
<p>Besides,</p>
<pre><code class="lang-auto">batch_spk_centroid[index1] = ((batch_spk_centroid[index1] * self.hp.train.num_utt_per_spk ) - batch_embedding[index1][index2] ) / (self.hp.train.num_utt_per_spk - 1)

</code></pre>
<p>I think this is in-place as you are modifying a tensor value based on its own value, you may need to clone that</p>
<pre><code class="lang-auto">batch_spk_centroid[index1] = ((batch_spk_centroid[index1].clone() * self.hp.train.num_utt_per_spk ) - batch_embedding[index1][index2] ) / (self.hp.train.num_utt_per_spk - 1)

</code></pre>

divyesh_rajpura: <p><a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>, Thank You for your reply. I removed deepcopy operation now. I tried .clone() previously, but it was giving error. It is still showing an error.</p>
<pre><code class="lang-auto">def calc_cosine_sim_div(self, batch_embedding, batch_label):
        batch_spk_centroid = torch.mean(batch_embedding, dim = 1)
        batch_spk_centroid = batch_spk_centroid.detach()
        cos_sim_matrix = []
        for index1, spk_embedding in enumerate(batch_embedding):
            for index2, utt_embedding in enumerate(spk_embedding):
                batch_spk_centroid[index1] = ((batch_spk_centroid[index1].clone() * self.hp.train.num_utt_per_spk ) - batch_embedding[index1][index2] ) / (self.hp.train.num_utt_per_spk - 1)
                cos_sim_matrix.append(self.cos_sim(utt_embedding.repeat(batch_embedding.shape[0],1), batch_spk_centroid))
                batch_spk_centroid[index1] = ((batch_spk_centroid[index1].clone() * (self.hp.train.num_utt_per_spk-1) ) + batch_embedding[index1][index2] ) / self.hp.train.num_utt_per_spk
        cos_sim_matrix = torch.stack(cos_sim_matrix).reshape(batch_embedding.shape[0],batch_embedding.shape[1],batch_embedding.shape[0])
        return cos_sim_matrix
</code></pre>

JuanFMontesinos: <pre><code class="lang-auto">self.cos_sim(utt_embedding.repeat(batch_embedding.shape[0],1)
</code></pre>
<p>What does self.cos_sim do?<br>
Besides it may happen because spk_embedding comes from batch_embeeding<br>
Think that utt_embedding comes from batch_emmbeding so if you modify it, it counts also as an in-place operation.</p>

divyesh_rajpura: <p>As batch_spk_centroid is local variable to the function calc_cosine_sim_div, I think maintaining the computational graph for the batch_spk_centroid is not required (please correct me if I am wrong.) So I have added with torch.no_grad() before the for loop begins. Below code is working perfectly.</p>
<pre><code class="lang-auto">def calc_cosine_sim_div(self, batch_embedding, batch_label):
        batch_spk_centroid = torch.mean(batch_embedding, dim = 1)
        cos_sim_matrix = []
        with torch.no_grad():
            for index1, spk_embedding in enumerate(batch_embedding):
                for index2, utt_embedding in enumerate(spk_embedding):
                    batch_spk_centroid[index1] = ((batch_spk_centroid[index1].clone() * self.hp.train.num_utt_per_spk ) - batch_embedding[index1][index2] ) / (self.hp.train.num_utt_per_spk - 1)
                    cos_sim_matrix.append(self.cos_sim(utt_embedding.repeat(batch_embedding.shape[0],1), batch_spk_centroid))
                    batch_spk_centroid[index1] = ((batch_spk_centroid[index1].clone() * (self.hp.train.num_utt_per_spk-1) ) + batch_embedding[index1][index2] ) / self.hp.train.num_utt_per_spk
        cos_sim_matrix = torch.stack(cos_sim_matrix).reshape(batch_embedding.shape[0],batch_embedding.shape[1],batch_embedding.shape[0])
        return cos_sim_matrix
</code></pre>

divyesh_rajpura: <p>self.cos_sim is nn.CosineSimilarity() function. batch_embeddings is actual output by my model and I am not modifying any value of batch_embeddings in my code.</p>

JuanFMontesinos: <p>As far as I understand batch_spk_centroid is local but cos_sim_matrix is not.</p>
<pre><code class="lang-auto">def calc_cosine_sim_div(self, batch_embedding, batch_label):
        batch_spk_centroid = torch.mean(batch_embedding, dim = 1)
        cos_sim_matrix = []
        with torch.no_grad():
            for index1, spk_embedding in enumerate(batch_embedding):
                for index2, utt_embedding in enumerate(spk_embedding):
                    batch_spk_centroid[index1] = ((batch_spk_centroid[index1].clone() * self.hp.train.num_utt_per_spk ) - batch_embedding[index1][index2] ) / (self.hp.train.num_utt_per_spk - 1)
                    cos_sim_matrix.append(self.cos_sim(utt_embedding.repeat(batch_embedding.shape[0],1), batch_spk_centroid))
                    batch_spk_centroid[index1] = ((batch_spk_centroid[index1].clone() * (self.hp.train.num_utt_per_spk-1) ) + batch_embedding[index1][index2] ) / self.hp.train.num_utt_per_spk
        cos_sim_matrix = torch.stack(cos_sim_matrix).reshape(batch_embedding.shape[0],batch_embedding.shape[1],batch_embedding.shape[0])
        return cos_sim_matrix
</code></pre>
<p>So<br>
gradient computation of</p>
<pre><code class="lang-auto">cos_sim_matrix.append(self.cos_sim(utt_embedding.repeat(batch_embedding.shape[0],1), batch_spk_centroid))
</code></pre>
<p>is lost there.<br>
Hmmm I don’t find any other issue which is cleary in-place to my but you can try to abuse of .clone to catch it.</p>
<pre><code class="lang-auto">batch_embedding[index1][index2]
</code></pre>
<p>is a suggestion as it later on becomes part of batch_spk_centroid</p>

divyesh_rajpura: <p>I also see that, after applying torch.no_grad() time for backward pass reduced too much. So I think adding torch.no_grad() we may miss some computation and might not be better approach. <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> Thank you for your time and suggestions. At last my quick ques is that do we really need to keep track of this kind of local variables in computation graph??</p>

JuanFMontesinos: <p>Hi,<br>
I depends on what you want.<br>
Let’s put an example<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d71e328cd52481d0658548c969735138c0b56f03.png" alt="image" data-base62-sha1="uH1lWquxMMjf0qbK6FqgShjonV9" width="436" height="371"><br>
Here the error wrt i1 depends (at some point given chain rule) of w1 and w3, which depends on h1 and h2. So if you don’t track gradients it is equivalent to delete the edge which goes from h1 to i1.</p>
<p>Is it convinient? Depends on your model.</p>

divyesh_rajpura: <p><a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>  Surely, I agree model require to keep track of all the operations, weights etc. I was confuse about loss function how it contributes in backward pass. But, yes loss function would also be major component in computing derivatives. I think I need to revise some of the concepts. Thank You.</p>