BatchNorm1d backward error

username: jetcai1900
cooked: <p>I am using BatchNorm1d proceeding a linear layer and the input to BatchNorm1d is 2-dimensional. But I get the error message “torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: tensor does not have a device”</p>
<p>The pytorch version is 1.4.0.</p>
<p>However, this code works with pytorch 0.1.12. Anyone has any ideas on fixing this bug? Thanks a lot.</p>

username: ptrblck
cooked: <p>Could you update PyTorch to <code>1.5.1</code> and rerun the code?<br>
If you are still seeing this issue, could you post a code snippet to reproduce it, please?</p>

username: jetcai1900
cooked: <p>Thanks for your reply. Still the same error “RuntimeError: tensor does not have a device (device at /opt/conda/conda-bld/pytorch_1591914880026/work/c10/core/TensorImpl.h:463)<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7f178facab5e in /anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> at::Tensor::options() const + 0x1ef (0x7f17bce2675f in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#2:</span> at::native::empty_like(at::Tensor const&amp;, c10::TensorOptions const&amp;, c10::optional<a>c10::MemoryFormat</a>) + 0x36 (0x7f17b7c5e556 in anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#3:</span>  + 0xe912d1 (0x7f17b7f6b2d1 in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#4:</span>  + 0xee45d3 (0x7f17b7fbe5d3 in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#5:</span>  + 0x26c805a (0x7f17925ec05a in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#6:</span> std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt; at::native::batch_norm_backward_cuda_template&lt;float, float, int&gt;(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, bool, double, std::array&lt;bool, 3ul&gt;) + 0x17f (0x7f179260d77f in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#7:</span> at::native::batch_norm_backward_cuda(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, bool, double, std::array&lt;bool, 3ul&gt;) + 0x2ce (0x7f17925eea2e in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#8:</span>  + 0xde3c3c (0x7f1790d07c3c in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#9:</span>  + 0xe21143 (0x7f17b7efb143 in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#10:</span>  + 0x2884df4 (0x7f17b995edf4 in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#11:</span>  + 0xe21143 (0x7f17b7efb143 in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#12:</span>  + 0x25f0429 (0x7f17b96ca429 in anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#13:</span> torch::autograd::generated::NativeBatchNormBackward::apply(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;&amp;) + 0x338 (0x7f17b96ca9a8 in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#14:</span>  + 0x2ae7df5 (0x7f17b9bc1df5 in /anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#15:</span> torch::autograd::Engine::evaluate_function(std::shared_ptr<a>torch::autograd::GraphTask</a>&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) + 0x16f3 (0x7f17b9bbf0f3 in anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#16:</span> torch::autograd::Engine::thread_main(std::shared_ptr<a>torch::autograd::GraphTask</a> const&amp;, bool) + 0x3d2 (0x7f17b9bbfed2 in/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#17:</span> torch::autograd::Engine::thread_init(int) + 0x39 (0x7f17b9bb8549 in anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#18:</span> torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f17bd108638 in/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#19:</span>  + 0xb8408 (0x7f17bf958408 in /anaconda3/lib/python3.7/site-packages/torch/lib/…/…/…/…/./libstdc++.so.6)<br>
frame <span class="hashtag">#20:</span>  + 0x7e65 (0x7f17e71c6e65 in /lib64/libpthread.so.0)<br>
frame <span class="hashtag">#21:</span> clone + 0x6d (0x7f17e6eef88d in /lib64/libc.so.6)”</p>
<p>my code is a little bit long.</p>

username: jetcai1900
cooked: <p>I am running this code.</p>
<p>Not sure whether we need to register parameter?<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/hyqneuron/pytorch-avitm/blob/master/pytorch_model.py#L42" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/hyqneuron/pytorch-avitm/blob/master/pytorch_model.py#L42" target="_blank" rel="nofollow noopener">hyqneuron/pytorch-avitm/blob/master/pytorch_model.py#L42</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="32" style="counter-reset: li-counter 31 ;">
<li>    self.register_buffer('prior_mean',    prior_mean)</li>
<li>    self.register_buffer('prior_var',     prior_var)</li>
<li>    self.register_buffer('prior_logvar',  prior_logvar)</li>
<li>    # initialize decoder weight</li>
<li>    if ac.init_mult != 0:</li>
<li>        #std = 1. / math.sqrt( ac.init_mult * (ac.num_topic + ac.num_input))</li>
<li>        self.decoder.weight.data.uniform_(0, ac.init_mult)</li>
<li>    # remove BN's scale parameters</li>
<li>    self.logvar_bn .register_parameter('weight', None)</li>
<li>    self.mean_bn   .register_parameter('weight', None)</li>
<li class="selected">    self.decoder_bn.register_parameter('weight', None)</li>
<li>    self.decoder_bn.register_parameter('weight', None)</li>
<li>
</li><li>def forward(self, input, compute_loss=False, avg_loss=True):</li>
<li>    # compute posterior</li>
<li>    en1 = F.softplus(self.en1_fc(input))                            # en1_fc   output</li>
<li>    en2 = F.softplus(self.en2_fc(en1))                              # encoder2 output</li>
<li>    en2 = self.en2_drop(en2)</li>
<li>    posterior_mean   = self.mean_bn  (self.mean_fc  (en2))          # posterior mean</li>
<li>    posterior_logvar = self.logvar_bn(self.logvar_fc(en2))          # posterior log variance</li>
<li>    posterior_var    = posterior_logvar.exp()</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: jetcai1900
cooked: <p>I think “self.logvar_bn.register_parameter(‘weight’, None)” is the fault. Could you explain why after declaring logvar_bn = nn.BatchNorm1d, we need to register_parameter to None? Thanks. Does this mean we are not going to use batch norm?</p>

username: ptrblck
cooked: <p>The affine parameters won’t be used, if you register the <code>weight</code> and <code>bias</code> with <code>None</code>.<br>
The PyTorch <code>nn.BatchNorm</code> implementation does the same in <a href="https://github.com/pytorch/pytorch/blob/5f6c6ed157378cece43c71b73cceedd6967dbb97/torch/nn/modules/batchnorm.py#L41-L46">these lines of code</a>.</p>
<p>I’m not familiar with the linked repository, but could you replace it with a valid parameter (or just remove these lines), if you think they might cause the issue?</p>