RuntimeError mat1 dim 1 must match mat2 dim 0

_TV: <p>I see a couple of such posts in forum but I have hardtime generalizing it to my own problem. Here’s the error:</p>
<p>Result:</p>
<pre><code class="lang-auto">File "Pmain.py", line 244, in Pmain
    optimize_model()
  File "Pmain.py", line 114, in optimize_model
    state_action_values = Policy_Net(state_batch).gather(1, action_batch)
  File "Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "OriNet.py", line 59, in forward
    h1 = F.relu(self.fc1(x)) 
  File "Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "Programs\Python\Python37\lib\site-packages\torch\nn\modules\linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "Programs\Python\Python37\lib\site-packages\torch\nn\functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: mat1 dim 1 must match mat2 dim 0
</code></pre>
<p>Network</p>
<pre><code class="lang-auto">class DQN(nn.Module):
    
    def __init__(self, num_states, num_actions):
        super(DQN, self).__init__()
        self.fc1 = nn.Linear(num_states, 32)
        self.fc2 = nn.Linear(32, 32) 
        self.fc3 = nn.Linear(32,32)  
        self.fc4 = nn.Linear(32,12)   
        self.fc5 = nn.Linear(12, num_actions)

    
    def forward(self, x):
        h1 = F.relu(self.fc1(x)) 
        h2 = F.relu(self.fc2(h1))
        h3 = F.relu(self.fc3(h2))
        h4 = F.relu(self.fc4(h3))
        output = self.fc5(h4)
        return output

</code></pre>
<p>optimize_model</p>
<pre><code class="lang-auto">def optimize_model():
        if len(memory) &lt; BATCH_SIZE:
            return
        transitions = memory.sample(BATCH_SIZE)
        batch = Transition(*zip(*transitions))

        state_batch = torch.cat(batch.state)
        action_batch = torch.cat(batch.action)
        reward_batch = torch.cat(batch.reward)
        
#        state_batch = torch.unsqueeze(state_batch,1)
#        state_batch = torch.unsqueeze(state_batch,0)
#        state_batch = torch.unsqueeze(state_batch,0)

        state_action_values = Policy_Net(state_batch).gather(1, action_batch)

        next_state_values = torch.zeros(BATCH_SIZE, device=device)

        non_final_mask = torch.ByteTensor(
            tuple(map(lambda s: s is not None, batch.next_state)))
        non_final_next_states = torch.cat([s for s in batch.next_state
                                                    if s is not None])
        next_state_values[non_final_mask] = Target_Net(
            non_final_next_states).max(1)[0].detach()

        expected_state_action_values = (next_state_values * GAMMA) + reward_batch

        loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

        optimizer.zero_grad()
        loss.backward()
        for param in Policy_Net.parameters():
            param.grad.data.clamp_(-1, 1)
        optimizer.step()
</code></pre>
<p>DQN input: [A,B,C,D,E,F,G,H] ( FloatTensor 8 variable,1dimension)</p>
<p>state_batch:</p>
<pre><code class="lang-auto">tensor([  0., 181., 171.,  ..., 181., 196., 171.], device='cuda:0')
</code></pre>
<p>action_batch:</p>
<pre><code class="lang-auto">tensor([[0],
        [0],
        [0],
        [0],
    ~
        [0],
        [0],
        [0]], device='cuda:0')
</code></pre>
<p>I tryed “torch.unsqueeze”, but this program returned same error.<br>
Can anyone help to solve this problem ?</p>

ptrblck: <p>Assuming <code>num_states</code> is set to <code>8</code>, the posted model expects an input in the shape <code>[batch_size, 8]</code>.<br>
Could you check the input shape and make sure it matches the expected one?</p>

_TV: <p>Thank you for advice.<br>
I’m not very good at English, so I apologize if I misinterpreted your sentence.<br>
num_states is set to 8.batch_size is set to 128.</p>
<p>print(state_batch.size())</p>
<blockquote>
<p>torch.Size([1024])</p>
</blockquote>
<p>“input” is model’s declaration?<br>
InputNumber is set to 8, AttackNumber is set to 6.</p>
<pre><code class="lang-auto">
Policy_Net = OriNet.DQN(InputNumber,AttackNumber).to(device)
Target_Net = OriNet.DQN(InputNumber,AttackNumber).to(device)
Target_Net.load_state_dict(Policy_Net.state_dict())
Target_Net.eval()
optimizer = optim.Adam(Policy_Net.parameters(), lr=0.001)
memory = OriNet.ReplayMemory(10000)
</code></pre>

_TV: <p>I understand that that output “torch.Size([1024])” has an error, It is needed to become "torch.Size([128,8])</p>
<p>it appears be flatten.</p>
<p>I don’t use  the gym , PLE ,etc.<br>
I refer to tutorial(<a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html#training" class="inline-onebox" rel="noopener nofollow ugc">Reinforcement Learning (DQN) Tutorial — PyTorch Tutorials 1.7.1 documentation</a>),</p>
<p>inputs is equal to tutorial’s state.</p>
<pre><code class="lang-auto">inputs =  [A,B,C,D,E,F,G,H]
inputs = np.array(inputs)
inputs = torch.from_numpy(inputs).type(torch.FloatTensor).to(device)
</code></pre>
<p>I can’t run my progam yet.<br>
Please Help me.</p>

_TV: <p>Replay Memory module is same as Tutorial’s ReplayMemory.</p>
<pre><code class="lang-auto">class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """Saves a transition."""
        if len(self.memory) &lt; self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)
</code></pre>

ptrblck: <p>I’m still unsure how the input is created. Are you seeing the same issue if you run the tutorial directly?<br>
If not, could you post an executable code snippet with random values, which would reproduce this issue?</p>

_TV: <p>I can run tutorial.</p>
<p>I made a simple programs, it outputs the same error.</p>
<pre><code class="lang-auto">import Battle
import re
import numpy as np
import os
import copy
import random
from copy import deepcopy
import matplotlib.pyplot as plt
import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from collections import namedtuple
import math
import OriNet


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

BATCH_SIZE = 128
GAMMA = 0.999
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 200
TARGET_UPDATE = 500
InputNumber = 8
AttackNumber = 6
steps_done = 0
Transition = namedtuple('Transition',
                        ('state', 'action', 'next_state', 'reward'))

AttackList = [1,2,3,4,5,6]

def snippet():
    EPISODE = 2000
    n_actions = 6
    Policy_Net = OriNet.DQN(InputNumber,AttackNumber).to(device)
    Target_Net = OriNet.DQN(InputNumber,AttackNumber).to(device)
    Target_Net.load_state_dict(Policy_Net.state_dict())
    Target_Net.eval()
    optimizer = optim.Adam(Policy_Net.parameters(), lr=0.001)
    memory = OriNet.ReplayMemory(10000)

    def select_action(state):
        global steps_done
        sample = random.random()
        eps_threshold = EPS_END + (EPS_START - EPS_END) * \
            math.exp(-1. * steps_done / EPS_DECAY)
        steps_done += 1
        if sample &gt; eps_threshold:
            with torch.no_grad():
#                print(Policy_Net(state)[0])
                return Policy_Net(state)
        else:
            return torch.tensor([[random.randrange(n_actions)]], device=device, dtype=torch.long)


    def optimize_model():
        if len(memory) &lt; BATCH_SIZE:
            return
        transitions = memory.sample(BATCH_SIZE)
        batch = Transition(*zip(*transitions))

        state_batch = torch.cat(batch.state)
        action_batch = torch.cat(batch.action)
        reward_batch = torch.cat(batch.reward)

#        state_batch = torch.unsqueeze(state_batch,1)


        state_action_values = Policy_Net(state_batch).gather(1, action_batch)

        next_state_values = torch.zeros(BATCH_SIZE, device=device)
#        next_state_values[non_final_mask] = Target_Net(non_final_next_states).max(1)[0].detach()

        non_final_mask = torch.ByteTensor(
            tuple(map(lambda s: s is not None, batch.next_state)))
        non_final_next_states = torch.cat([s for s in batch.next_state
                                                    if s is not None])
        next_state_values[non_final_mask] = Target_Net(
            non_final_next_states).max(1)[0].detach()

        expected_state_action_values = (next_state_values * GAMMA) + reward_batch

        loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

        optimizer.zero_grad()
        loss.backward()
        for param in Policy_Net.parameters():
            param.grad.data.clamp_(-1, 1)
        optimizer.step()
        print('aaa')

    for i in range(EPISODE):
        inputs = [100,200,300,400,500,600,700,800]
        inputs = np.array(inputs)
        inputs = torch.from_numpy(inputs).type(torch.FloatTensor).to(device)

        Action = select_action(inputs)
        AttackNum = Action[0].to('cpu').detach().numpy().copy()
        Attack = AttackList[np.argmax(AttackNum)]

#      this time, program updates status in this game. 
        NowState = [101,201,301,401,501,601,701,801]
        NowState = np.array(NowState)
        NowState = torch.from_numpy(NowState).type(torch.FloatTensor).to(device)

        reward = 1
        reward = torch.tensor([reward],device = device)

        AttackNum = torch.tensor([[Attack]],device = device)

        memory.push(inputs,AttackNum,NowState,reward)
        optimize_model()



snippet()
</code></pre>
<p>DQN model</p>
<pre><code class="lang-auto">class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """Saves a transition."""
        if len(self.memory) &lt; self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

class DQN(nn.Module):
    
    def __init__(self, num_states, num_actions):
        super(DQN, self).__init__()
        self.fc1 = nn.Linear(num_states, 32)
        self.fc2 = nn.Linear(32, 32) 
        self.fc3 = nn.Linear(32,32)  
        self.fc4 = nn.Linear(32,12)   
        self.fc5 = nn.Linear(12, num_actions)

    
    def forward(self, x):
        h1 = F.relu(self.fc1(x)) 
        h2 = F.relu(self.fc2(h1))
        h3 = F.relu(self.fc3(h2))
        h4 = F.relu(self.fc4(h3))
        output = self.fc5(h4)
        return output


</code></pre>

ptrblck: <p>Thanks for the code snippet.<br>
The error is raised, as <code>torch.cat(batch_state)</code> and <code>torch.cat([s for s in batch.next_state if s is not None])</code> will return the flattened tensors. Use <code>torch.stack</code> instead and the code should be working fine.<br>
I haven’t checked the exact differences between your code and the tutorial, but guess your states are containing multiple state values, which would yield this error.</p>

_TV: <p>Thank you for your advices.<br>
My program appears to be working.<br>
Writing for the knowledge of younger students, I have corrected this:</p>
<p>before</p>
<pre><code class="lang-auto">state_batch = torch.cat(batch.state) 
non_final_next_states = torch.cat([s for s in batch.next_state
                                                    if s is not None])
</code></pre>
<p>after</p>
<pre><code class="lang-auto">state_batch = torch.stack(batch.state) 
non_final_next_states = torch.stack([s for s in batch.next_state

                                                    if s is not None])
</code></pre>

_TV: <p>My program ran once, but returned an error when I ran it again.</p>
<blockquote>
<p>non_final_next_states).max(1)[0].detach()<br>
RuntimeError: CUDA error: device-side assert triggered</p>
</blockquote>
<p>When I tried to use the CPU, it returned the following error.</p>
<blockquote>
<p>state_action_values = Policy_Net(state_batch).gather(1, action_batch)<br>
RuntimeError: index 8 is out of bounds for dimension 1 with size 6</p>
</blockquote>

_TV: <pre><code class="lang-auto">def optimize_model():
        if len(memory) &lt; BATCH_SIZE:
            return
        transitions = memory.sample(BATCH_SIZE)
        batch = Transition(*zip(*transitions))

        non_final_mask = torch.ByteTensor(
            tuple(map(lambda s: s is not None, batch.next_state)))
        non_final_next_states = torch.stack([s for s in batch.next_state
                                                if s is not None])
        
        state_batch = torch.stack(batch.state)
        action_batch = torch.cat(batch.action)
        reward_batch = torch.cat(batch.reward)

        state_action_values = Policy_Net(state_batch).gather(1, action_batch)

        next_state_values = torch.zeros(BATCH_SIZE, device=device)
        next_state_values[non_final_mask] = Target_Net(non_final_next_states).max(1)[0].detach()

        expected_state_action_values = (next_state_values * GAMMA) + reward_batch

        loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

        optimizer.zero_grad()
        loss.backward()
        for param in Policy_Net.parameters():
            param.grad.data.clamp_(-1, 1)
        optimizer.step()

</code></pre>