An error will occur if the batch size is set to 1 or more

skerlet_flandorle: <p>I’m doing reinforcement learning<br>
Batch size cannot be greater than 1<br>
I’m trying a batch size of 32</p>
<p>I can’t find anything strange when I look at Ross<br>
I’m in trouble</p>
<blockquote>
<p>tensor([[1.3092e-03, 2.1749e-03, 2.6177e-06, 2.2517e-03, 2.1730e-03, 4.7358e-05,<br>
1.3836e-03, 8.5324e-04, 1.0955e-03, 1.9888e-03, 1.3575e-03, 1.6514e-05,<br>
2.5816e-04, 1.8731e-04, 1.9047e-03, 5.3802e-04, 6.7010e-04, 8.9404e-04,<br>
4.5198e-04, 1.1572e-04, 3.0101e-04, 1.6938e-03, 4.7421e-04, 6.1655e-04,<br>
3.7556e-03, 1.9652e-04, 3.1410e-04, 6.3015e-05, 3.7714e-04, 2.9124e-08,<br>
6.6303e-04, 2.7347e-05]], device=‘cuda:0’, grad_fn=)</p>
</blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
427             trin.Double_R2D2_IQN_pioritized_Nstep_NAF_replay(batch_size, gamma,step=episode,<br>
428                                                              state_size=state_,action_size=acthon,<br>
 → 429                                                              multireward_steps=multireward_steps)<br>
430<br>
431         if done or t==max_number_of_steps + 1:</p>
<p> in Double_R2D2_IQN_pioritized_Nstep_NAF_replay(self, batch_size, gamma, step, state_size, action_size, multireward_steps)<br>
263             loss=loss*weights<br>
264             print(loss)<br>
 → 265         loss.backward() # ここを変更<br>
266         optimizer.step()<br>
267         self.Rs=[0 for _ in range(multireward_steps)]</p>
<p>~\Anaconda3\envs\pyflan\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
196                 products. Defaults to <code>False</code>.<br>
197         “”"<br>
 → 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
199<br>
200     def register_hook(self, hook):</p>
<p>~\Anaconda3\envs\pyflan\lib\site-packages\torch\autograd_<em>init</em>_.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
92         grad_tensors = list(grad_tensors)<br>
93<br>
—&gt; 94     grad_tensors = _make_grads(tensors, grad_tensors)<br>
95     if retain_graph is None:<br>
96         retain_graph = create_graph</p>
<p>~\Anaconda3\envs\pyflan\lib\site-packages\torch\autograd_<em>init</em>_.py in _make_grads(outputs, grads)<br>
33             if out.requires_grad:<br>
34                 if out.numel() != 1:<br>
—&gt; 35                     raise RuntimeError(“grad can be implicitly created only for scalar outputs”)<br>
36                 new_grads.append(torch.ones_like(out, memory_format=torch.preserve_format))<br>
37             else:</p>
<p>RuntimeError: grad can be implicitly created only for scalar outputs</p>
<p>The code is</p>
<pre><code>    targets= torch.zeros(batch_size,device='cuda:0')
    trin_x=list(range(batch_size))
    weights = torch.ones(batch_size,device='cuda:0')
    idx=np.random.choice(np.arange(len(memory_TDerror.buffer)), size=batch_size,
                         replace=False,p = memory_TDerror.buffer/np.sum(memory_TDerror.buffer))
    
    for i in idx:
        inputs=(torch.cat([memory.buffer[i-multireward_steps][0]]),
                           torch.cat([memory.buffer[i-multireward_steps][1]]))
    for _,i in enumerate(idx):
        with torch.no_grad():
            targets[_]=memory.buffer[i][2].to("cuda:0")+(gamma ** multireward_steps)*targetQN.forward(memory.buffer[i][0],"net_v")
        
        priority = rank_sum(memory_TDerror.buffer[i], self.alpha)
        weights[_]=(len(memory.buffer) * priority) ** (-self.beta)
    weights = weights / max(weights)   
    #リプレイとおなじ
    optimizer.zero_grad()
    output = mainQN.forward(inputs,"net_q")
    if self.IQN==True:
        self.loss_IQN(target,output,weights)
    else:
        loss = huber(output-targets)
        loss=loss*weights
        print(loss)
    loss.backward() # ここを変更
    optimizer.step()
    self.Rs=[0 for _ in range(multireward_steps)]
</code></pre>

ptrblck: <p>Based on the error message it seems that <code>loss</code> is not a scalar value but a tensor with multiple values.<br>
You could either reduce it to a scalar (e.g. with <code>torch.mean</code>, if that would fir your use case) or you could pass the gradients to <code>backward</code> e.g. via <code>loss.backward(torch.ones_like(loss))</code>.</p>