Asynchronous parameters updating?
Category: reinforcement-learning

pengsun: <SCODE>-- in main thread: shared parameters
params, _ = sharedNet:getParameters()

-- in worker thread: its own gradParameters
tNet = sharedNet:clone()
_, gradParams = tNet:getParameters()

-- in worker thread: stuff

-- in worker thread: updating shared parameters with its own gradParameters
function feval() return nil, gradParams end
optim.rmsprop(feval, params, sharedStates)
<ECODE> Thanks in advance!

apaszke: I have already implemented A3C in pytorch, and it works just fine.  When you get a copy with everything shared in the subprocess just do this to break the gradient sharing, and use the optimizer as you’d normaly do: <SCODE>for param in model.parameters():
    param.grad.data = param.grad.data.clone()
<ECODE>

pengsun: <SCODE># after every several steps (e.g., 5 or 20)
for t_param, shared_param in zip(t_model.parameters(), shared_model.parameters()):
    t_param.data.copy_(shared_param.data)
<ECODE> Or did you find it not critical to the accuracy in your implementation? (Previously, I strictly follow the paper and I can reproduce the scoring curve for breakout as in the paper’s figure)

apaszke: <SCODE>t_model.load_state_dict(shared_state_dict)
<ECODE>

pengsun: 

Anurup_Arvind: 

apaszke: 

Ilya_Kostrikov: It converges on PongDeterministic-v3 in 10 minutes with 16 threads. However, it work poorly on PongDeterministic-v3. Could you please take a look? I wonder whether I got asynchronous updates in PyTorch right.

apaszke: I don’t think I’d recommend reusing the same grad tensors in multiple Variables, but apart form that I can’t see anything wrong at a glance.

Ilya_Kostrikov: The problem was in the architecture that I used initially, seems to work on Breakout now. Reusing the same grad tensor might cause some problems?

apaszke: 

AjayTalati: AttributeError: ‘float’ object has no attribute ‘backward’ Any guesses what’s caused it? Apart from my custom environment, the only difference to your code is I’ve remove the conv layers, and used a 2 layer LSTM - which is the model I’m using in TensorFlow. As a performance improvement have you tried concatenating the previous action, reward, and a timestep counter onto the end of the state as 3 scalars - I noticed a significant improvement in my TF implementation when I do this.

apaszke: 

AjayTalati: Which I’m working through, which seems closer to my TF implementation. Hopefully I’ll figure it out soon. At the moment my environment code, (I guess like yours), is stuck in the middle of a larger project, and is not stand alone, yet. This is actually the reason why I’m moving away from TF. Hopefully, I’ll be able to wrap my environment into something like an OpenAI env, and be able to open source it on Github within a few days, and get back to you. It’s a nice example of graph optimisation/travelling salesman, so it should be of interest to quite a few people. All the best, Aj

apaszke: About Variables, you only need to use the for parts that will require differentiation, and it’s best to not use them for everything else. If you have any questions about any specific cases, I’ll be happy to answer.

onlytailei: you mean right?

denizs: 

Hung_Nguyen: