RuntimeError: mat1 and mat2 shapes cannot be multiplied in actor-critic

username: shgh
cooked: <p>following error <code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x1 and 3x256) </code></p>
<p>happens at this line: <code>x = F.relu(self.linear1(state))</code></p>
<pre><code class="lang-auto">class Critic(nn.Module):
    def __init__(self, state_size, action_size, hidden_size, output_size):
        super(Critic, self).__init__()
        self.linear1 = nn.Linear(state_size, hidden_size)
        self.linear2 = nn.Linear(hidden_size+action_size, hidden_size)
        self.linear3 = nn.Linear(hidden_size, output_size)

    def forward(self, state, action):
        x = F.relu(self.linear1(state))
        x = torch.cat([x, action], dim = 1)
        x = F.relu(self.linear2(x))
        #x = F.relu(self.linear2(x))
        x = self.linear3(x)

        return x
</code></pre>
<p>and that 64 depends on the <code>batch_size</code> below</p>
<pre><code class="lang-auto">  def sample(self, batch_size):
        state_batch = []
        action_batch = []
        reward_batch = []
        next_state_batch = []
        done_batch = []

        batch = random.sample(self.buffer, batch_size)

        for experience in batch:
            state, action, reward, next_state, done = experience
            state_batch.append(state)
            action_batch.append(action)
            reward_batch.append(reward)
            next_state_batch.append(next_state)
            done_batch.append(done)

        return state_batch, action_batch, reward_batch, next_state_batch, done_batch
</code></pre>
<p>does anyone know how this error can be fixed?</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="shgh" data-post="1" data-topic="116903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d78d45/48.png" class="avatar"> shgh:</div>
<blockquote>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x1 and 3x256</p>
</blockquote>
</aside>
<p>So, this mis-match would indicate that <code>state_size</code> is different from what you expected. I assume that <code>state_batch</code> is a <code>Tensor</code> of shape 64x1? But expect it to be 64x3?</p>
<aside class="quote no-group" data-username="shgh" data-post="1" data-topic="116903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d78d45/48.png" class="avatar"> shgh:</div>
<blockquote>
<pre><code class="lang-auto">for experience in batch:
            state, action, reward, next_state, done = experience
            state_batch.append(state)
            action_batch.append(action)
            reward_batch.append(reward)
            next_state_batch.append(next_state)
            done_batch.append(done)

        return state_batch, action_batch, reward_batch, next_state_batch, done_batch
</code></pre>
</blockquote>
</aside>
<p>Also, if you’re expecting a <code>Tensor</code> to be passed in, shouldn’t you use <code>torch.stack</code> to covert the list of Tensors to a single <code>Tensor</code> which can be passed in batch? Could you check the <code>type</code> and <code>shape</code> of <code>state_batch</code>?</p>

username: shgh
cooked: <pre><code class="lang-auto">type of `state_batch` is `list` with a length of `64`.
</code></pre>
<p>And the complete code for the <code>state_batch</code> is here:</p>
<pre><code class="lang-auto">class Memory:
    def __init__(self, max_size):
        self.max_size = max_size
        self.buffer = deque(maxlen=max_size)

    def push(self, state, action, reward, next_state, done):

        experience = (state, action, np.array([reward]), next_state, done)
        self.buffer.append(experience)

    def sample(self, batch_size):
        state_batch = []
        action_batch = []
        reward_batch = []
        next_state_batch = []
        done_batch = []

        batch = random.sample(self.buffer, batch_size)

        for experience in batch:
            state, action, reward, next_state, done = experience
            state_batch.append(state)
            print(f"state_batch in sample method is: {type(state_batch)}")
            action_batch.append(action)
            reward_batch.append(reward)
            next_state_batch.append(next_state)
            done_batch.append(done)

        return state_batch, action_batch, reward_batch, next_state_batch, done_batch
</code></pre>

username: AlphaBetaGamma96
cooked: <p>What are the contents of the list? Are they Tensors? If so, what shape? If the elements are of shape <code>[1,3]</code> and you have a list of <code>64</code> items, you can <code>stack</code> the list of Tensors into a single Tensor which might solve the mis-match error!</p>
<p>Could you print what <code>state_batch[0].shape</code> returns?</p>

username: shgh
cooked: <p>elements of <code>state_batch</code> list are <code>state</code>. They are numbers (scalars). Below is the full stack error. And part of <code>update</code> function that is referring to:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 34, in &lt;module&gt;
    agent.update(batch_size)
  File "ddpg.py", line 62, in update
    Qvals = self.critic.forward(states, actions)
  File "model.py", line 56, in forward
    x = F.relu(self.linear1(state))

RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x1 and 3x128)

</code></pre>
<pre><code class="lang-auto">    def update(self, batch_size):
        states, actions, rewards, next_states, _ = self.memory.sample(batch_size)
        states = torch.FloatTensor(states)
        states = states.unsqueeze(1).expand(-1, 1)
        actions = torch.FloatTensor(actions)
        rewards = torch.FloatTensor(rewards)
        next_states = torch.FloatTensor(next_states)

        # Critic loss
        Qvals = self.critic.forward(states, actions)
        next_actions = self.actor_target.forward(next_states)
        next_Q = self.critic_target.forward(next_states, next_actions.detach())
        Qprime = rewards + self.gamma * next_Q
        critic_loss = self.critic_criterion(Qvals, Qprime)

</code></pre>

username: AlphaBetaGamma96
cooked: <p>So, if <code>state</code> is a batch of scalars, it makes sense why you’re getting a mis-match within <code>x = F.relu(self.linear1(state))</code>, perhaps change <code>self.linear1</code> to be of size <code>[1, 128]</code>? rather than <code>[3,128]</code>?</p>

username: shgh
cooked: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="6" data-topic="116903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>perhaps change <code>self.linear1</code> to be of size <code>[1, 128]</code>? rather than <code>[3,128]</code>?</p>
</blockquote>
</aside>
<p>how can this change be made?</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="shgh" data-post="1" data-topic="116903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d78d45/48.png" class="avatar"> shgh:</div>
<blockquote>
<pre><code class="lang-auto">        self.linear1 = nn.Linear(state_size, hidden_size)
</code></pre>
</blockquote>
</aside>
<p>change the <code>state_size</code> to 1 rather than 3 will make it pass through that layer. If you’re sure the <code>state_size</code> is 3, then the issue lies with <code>state</code> instead</p>

username: shgh
cooked: <p>I am sure about <code>state_size</code> being 3. but <code>state</code> issue seems unsolvable!</p>

username: AlphaBetaGamma96
cooked: <aside class="quote no-group" data-username="shgh" data-post="5" data-topic="116903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d78d45/48.png" class="avatar"> shgh:</div>
<blockquote>
<p>elements of <code>state_batch</code> list are <code>state</code>. They are numbers (scalars)</p>
</blockquote>
</aside>
<p>So, <code>state_batch</code> is a list of 64 elements which are purely scalars? So, <code>state_batch[0].shape</code> would return <code>1</code>? If that’s true, and you’re sure <code>state_size</code> is <code>3</code> then something’s wrong with the <code>state_batch</code> variable!</p>

username: shgh
cooked: <p><code>state_bacth[0].shape</code> returns ().<br>
the <code>state_bacth[0]</code> returns a scalar.</p>

username: AlphaBetaGamma96
cooked: <p>So, that explains the mis-match issue. Either <code>state_batch</code> should be of size <code>[64,3]</code> or <code>state_size</code> should be of size 1.</p>