A3c in pytorch.Does normal a3c with multinomial/categorical sampling works well in continuous STATE with discrete action as well?

granth_jain: <p>Hi,</p>
<p>I am trying to develop A3C reinforcement learning in pytorch.</p>
<p>But I am getting same action being taken on multinomial as well as categorical sampling of actions.</p>
<p>Below is my code…I need help in understanding whether simple A3C works well in continuous state space as well.</p>
<p>Thanks,<br>
Granth</p>
<pre><code>    while count&lt;max_timesteps-1:
        episode_length += 1
        if done:
            cx = Variable(torch.zeros(1, params.state_dim))
            hx = Variable(torch.zeros(1, params.state_dim))
        else:
            cx = Variable(cx.data)
            hx = Variable(hx.data)
        values = []
        log_probs = []
        rewards = []
        entropies = []
        while count&lt;max_timesteps-1:
            value, action_values, (hx, cx) = model((Variable(state.unsqueeze(0)), (hx, cx)))
            prob = F.softmax(action_values,dim = -1)
            log_prob = F.log_softmax(action_values, dim=-1)
            entropy = -(log_prob * prob).sum(1, keepdim=True)
            entropies.append(entropy)
            cdist = categorical.Categorical(prob)
            action = cdist.sample()
            log_prob = log_prob[0, Variable(action)].data
            state, reward, done = env.step(action)
            done = (done or count == max_timesteps-2)
            reward = max(min(reward, 1), -1)
            
            
            count +=1
            
            if done:
                episode_length = 0
                state = env.reset()
                
            
            values.append(value)
            log_probs.append(log_prob)
            rewards.append(reward)
            print(ticker," action:",action, "reward ",reward)

            if done:
                break
            
        R = torch.zeros(1, 1)
        if not done:
            value, _, _ = model((Variable(state.unsqueeze(0)), (hx, cx)))
            R = value.data
        values.append(Variable(R))
        policy_loss = 0
        value_loss = 0
        R = Variable(R)
        gae = torch.zeros(1, 1)
        for i in reversed(range(len(rewards))):
            R = params.gamma * R + rewards[i]
            advantage = R - values[i]
            value_loss = value_loss + 0.5 * advantage.pow(2)
            TD = rewards[i] + params.gamma * values[i + 1].data - values[i].data
            gae = gae * params.gamma * params.tau + TD
            policy_loss = policy_loss - log_probs[i] * Variable(gae) - 0.01 * entropies[i]

        optimizer.zero_grad()
        (policy_loss + 0.5 * value_loss).backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 40)
        optimizer.step()</code></pre>

iffiX: <p>A3C works in continuous space, but its unstable (even more than A2C), and produce poorer results than PPO / IMPALA(distributed)</p>
<p>For correct implementation, consider reference:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/iffiX/machin/blob/b1a8b0ce99be9e4d47e132c1325e1aaedb87e0a4/machin/frame/algorithms/a2c.py#L312" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/iffiX/machin/blob/b1a8b0ce99be9e4d47e132c1325e1aaedb87e0a4/machin/frame/algorithms/a2c.py#L312" target="_blank" rel="nofollow noopener">iffiX/machin/blob/b1a8b0ce99be9e4d47e132c1325e1aaedb87e0a4/machin/frame/algorithms/a2c.py#L312</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="302" style="counter-reset: li-counter 301 ;">
<li>will be cleared after update is finished.</li>
<li>
</li><li>Args:</li>
<li>    update_value: Whether update the Q network.</li>
<li>    update_policy: Whether update the actor network.</li>
<li>    concatenate_samples: Whether concatenate the samples.</li>
<li>
</li><li>Returns:</li>
<li>    mean value of estimated policy value, value loss</li>
<li>"""</li>
<li class="selected">sum_act_loss = 0</li>
<li>sum_value_loss = 0</li>
<li>self.actor.train()</li>
<li>self.critic.train()</li>
<li>for _ in range(self.actor_update_times):</li>
<li>    # sample a batch</li>
<li>    batch_size, (state, action, advantage) = \</li>
<li>        self.replay_buffer.sample_batch(self.batch_size,</li>
<li>                                        sample_method="random_unique",</li>
<li>                                        concatenate=concatenate_samples,</li>
<li>                                        sample_attrs=[</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/iffiX/machin/blob/b1a8b0ce99be9e4d47e132c1325e1aaedb87e0a4/machin/frame/algorithms/a3c.py#L131" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/iffiX/machin/blob/b1a8b0ce99be9e4d47e132c1325e1aaedb87e0a4/machin/frame/algorithms/a3c.py#L131" target="_blank" rel="nofollow noopener">iffiX/machin/blob/b1a8b0ce99be9e4d47e132c1325e1aaedb87e0a4/machin/frame/algorithms/a3c.py#L131</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="121" style="counter-reset: li-counter 120 ;">
<li>    if self.is_syncing:</li>
<li>        self.critic_grad_server.pull(self.critic)</li>
<li>    return super(A3C, self)._criticize(state)</li>
<li>
</li><li>def update(self,</li>
<li>           update_value=True,</li>
<li>           update_policy=True,</li>
<li>           concatenate_samples=True,</li>
<li>           **__):</li>
<li>    # DOC INHERITED</li>
<li class="selected">    org_sync = self.is_syncing</li>
<li>    self.is_syncing = False</li>
<li>    super(A3C, self).update(update_value, update_policy,</li>
<li>                            concatenate_samples)</li>
<li>    self.is_syncing = org_sync</li>
<li>    self.actor_grad_server.push(self.actor)</li>
<li>    self.critic_grad_server.push(self.critic)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


granth_jain: <p>Hi,</p>
<p>Thanks for the answer…could you please help to let know if there is a version of ppo with lstm.</p>

iffiX: <p>If you just need ppo + lstm with 1-step back brop, there are. But it would be more tricky if you would like to have the BPTT (back-propagate through time feature)</p>

granth_jain: <p>Hi,</p>
<p>I am unable to figure it out how PPO is to be implemented with LSTM.</p>
<p>Also,given the main objective of PPO is to have an update near to the current policy…is it same to clip the gradient in A3C to a small value and then expect the results to be same as PPO??</p>
<p>may I get help in how to implement this and if there a correct implementation of ppo+lstm can you please help to share.</p>
<p>Thanks</p>