Running average of parameters

Seungyoung_Park: I would like to use the running average of parameters instead of using the parameters from the training directly at the test session. To do this, I initialized the running average parameters from the network as <SCODE>avg_param = torch.cat([param.view(-1) for param in model.parameters()],0)
<ECODE> Then, I performed the running average at each training iteration as <SCODE>avg_param = 0.9*avg_param + 0.1*torch.cat([param.data.view(-1) for param in model.parameters()],0)
<ECODE> Finally, at the test session, I loaded the parameters as <SCODE>i = 0
for param in model.parameters():
    param = avg_param[i:i+param.nelement()].resize(*param.size())
    i = i+param.nelement()
<ECODE> Is this process correct ?

apaszke: There are a few problems I can see: This would be correct: <SCODE>def flatten_params():
    return torch.cat([param.data.view(-1) for param in model.parameters()], 0)

def load_params(flattened):
    offset = 0
    for param in model.parameters():
        param.data.copy_(flattened[offset:offset + param.nelement()]).view(param.size())
        offset += param.nelement()

avg_param = flatten_params() # initialize

def train():
    ...
    avg_param = 0.9 * avg_param + 0.1 * flatten_params()

def test():
    original_param = flatten_params() # save current params
    load_params(avg_param) # load the average
    ...
    load_params(original_param) # restore parameters
    
<ECODE>

Seungyoung_Park: Thanks for your reply. Before employing the running average, my code occupied only half of video memory. But, when I tried as you suggested, it didn’t proceed after 2 iterations due to ‘out of memory’. The error is shown below. <SCODE>CompleteTHCudaCheck FAIL file=/data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.9_1487349287443/work/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
Traceback (most recent call last):
  File "inception_ae_resume_ra.py", line 251, in <module>
train_iter_loss, avg_param = train(config, epoch, avg_param)
  File "inception_ae_resume_ra.py", line 166, in train
avg_param = 0.9*avg_param + 0.1*flatten_params()
  File "/home/sypark/anaconda2/envs/py36/lib/python3.6/site-packages/torch/tensor.py", line 320, in __mul__
return self.mul(other)
RuntimeError: cuda runtime error (2) : out of memory at /data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.9_1487349287443/work/torch/lib/THC/generic/THCStorage.cu:66<ECODE>

apaszke: Your model must have a lot of parameters. Instead of flattening them into a single big tensor, you process it in parts: <SCODE>from copy import deepcopy

avg_param = deepcopy(list(p.data for p in model.parameters()))

def train():
    ...
    for p, avg_p in zip(model.parameters(), avg_param):
        avg_p.mul_(0.9).add_(0.1, p.data)
<ECODE> Not sure if you’ll manage to fit another copy of the params in memory, so you can restore them after testing.

Seungyoung_Park: Thanks for your reply. <SCODE>param.data.copy_(flattened[offset:offset + param.nelement()]).view(param.size())
RuntimeError: copy from list to FloatTensor isn't implemented
<ECODE> I think the load_param function should be modified due to the list.

apaszke: 

Seungyoung_Park: I modified the functions as below. <SCODE>def load_params(avg_param):
    for p, avg_p in zip(model.parameters(), avg_param):
        p.data = deepcopy(avg_p)

  def flatten_params():
    flatten = deepcopy(list(p.data for p in model.parameters()))
    return flatten

def load_params(flattened):
    for p, avg_p in zip(model.parameters(), flattened):
        p.data = deepcopy(avg_p)
<ECODE>

apaszke: This would be better: <SCODE>def load_params(flattened):
    for p, avg_p in zip(model.parameters(), flattened):
        p.data.copy_(avg_p)
<ECODE> Also, note that they’re no longer flattened, so you might want to change the name.

Seungyoung_Park: Thanks. I will change them.

pinocchio: <SCODE>W = Variable(w_init, requires_grad=True)
W_avg = Variable(torch.FloatTensor(W).type(dtype), requires_grad=False)
for i in range(nb_iterations):
    #some GD stuff...
    W_avg = (1/nb_iter)*W + W_avg<ECODE>