Continuous action A3C
Category: reinforcement-learning

AjayTalati: Hi, I wonder if anyone has got A3C working with continuous actions? I guessed it would be a good idea to ask first before trying to do it, as there’s probably a good reason why no one’s got it to work yet? So for example Open AIs pendulum, as only got a state/observation vector of 3, so there’s no need for any conv’s in the Actor-Critic module, basically I’m trying, <SCODE>lstm_out = 256
enc_in = 3 # for pendulum
enc_hidden = 200
enc_out = lstm_out

class ActorCritic(nn.Module):

    def __init__(self , lstm_in ):
        super(ActorCritic, self).__init__( )  
        self.fc_enc_in  = nn.Linear(enc_in,enc_hidden) # enc_input_layer
        self.fc_enc_out  = nn.Linear(enc_hidden,enc_out) # enc_output_layer             
        self.lstm = nn.LSTMCell(lstm_in, lstm_out)
        self.actor_mu = nn.Linear(lstm_out, 1)
        self.actor_sigma = nn.Linear(lstm_out, 1)
        self.critic_linear = nn.Linear(lstm_out, 1)
        self.train()

    def forward(self, inputs):
        
        x, (hx, cx) = inputs

        x = F.relu(self.fc_enc_in(x))
        x = self.fc_enc_out(x)

        hx, cx = self.lstm(x, (hx, cx))
        x = hx

        return self.critic_linear(x), self.actor_mu(x), self.actor_sigma(x), (hx, cx)
<ECODE> The initialisation code in main.py, then looks like, <SCODE>env = gym.envs.make("Pendulum-v0")
lstm_in = 3    
global_model = ActorCritic( lstm_in )
global_model.share_memory()
local_model = ActorCritic( lstm_in )
<ECODE> And the training code is where I get confused (as usual) ???, <SCODE>env = gym.envs.make("Pendulum-v0")
s0 = env.reset()
done = True
state = torch.from_numpy(s0).float().unsqueeze(0) 
value, mu, sigma, (hx, cx) = local_model((Variable(state), (hx, cx)))

#mu = mu.clamp(-1, 1) # constain to sensible values 
Softplus=nn.Softplus()     
sigma = Softplus(sigma + 1e-5) # constrain to sensible values
normal_dist = torch.normal(mu, sigma) 

prob = normal_dist
log_prob = torch.log(prob)
entropy = 0.5 * (torch.log(2. * np.pi * sigma ) + 1.)

##--------------------------------------------------------------
# TODO Calculate the Gaussian neg log-likelihood, log(1/sqrt(2sigma^2pi)) - (x - mu)^2/(2*sigma^2)
# See - https://www.statlect.com/fundamentals-of-statistics/normal-distribution-maximum-likelihood
#
log_prob = torch.log(torch.pow( torch.sqrt(2. * sigma * np.pi) , -1)) - (normal_dist - mu)*(normal_dist - mu)*torch.pow((2. * sigma), -1)
##--------------------------------------------------------------

action = Variable( prob.data )

#action=[0,]
state, reward, done, _ = env.step([action.data[0][0]])

<ECODE> References,

AjayTalati: How do you do a logarithm, in PyTorch? <SCODE>>>> nnlog = nn.Log()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch.nn' has no attribute 'Log'
<ECODE>

jekbradbury: 

AjayTalati: 

AjayTalati: Well, I came back to this after a few days, and I’m still stuck. So any advice will make you a genius in my view? Here’s a post of my code as simple as I could make it as a big blob, I keep getting this error, <SCODE>File "main_single.py", line 174, in <module>
value_loss = value_loss + advantage.pow(2)
AttributeError: 'numpy.ndarray' object has no attribute 'pow'
<ECODE> Any ideas what I’ve got wrong? Thanks a lot for your help, Best, Ajay

jekbradbury: reward is probably returned from gym as a numpy object (I guess a scalar?) so I think you have to convert it?

AjayTalati: What seems to help a little is, changing the code to <SCODE>    for t in reversed(range(len(rewards))):
        R = torch.mul(R, args.gamma)  
        R = torch.add(R, rewards[t])
        advantage = R - values[t]
        value_loss = value_loss + advantage.pow(2)
<ECODE> Now I get the error, <SCODE>  File "main_single.py", line 185, in <module>
    (policy_loss + 0.5 * value_loss).backward()
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/autograd/variable.py", line 158, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/torch/autograd/stochastic_function.py", line 13, in _do_backward
    raise RuntimeError("differentiating stochastic functions requires "
RuntimeError: differentiating stochastic functions requires providing a reward
<ECODE>

apaszke: 

AjayTalati: … this is padding to make this post 20 characters or more

andrewliao11: 

AjayTalati: I never managed to get it working very well, (I tried it on non-mujoco stuff), so went back to experimenting with the discrete actions version. Do you plan on experimenting with shared RMSProp? A3C is a great tool - you can apply it to a lot of stuff - it should be really helpful to you in the future! Kind regards, Ajay

andrewliao11: 

alexis-jacq: <SCODE>action = (mu + sigma_sq.sqrt()*Variable(eps)).data
<ECODE>