Change:
Hi, I encountered this bug: The code skeleton is like: It seems the loaded are , and I've tried a workaround to move to , but it still has the same bug.
Could you provide a full script to reproduce the problem?
maybe you can try like this, () (beta1).add_(1 - beta1, ())
Sorry, I missed the reply email. I am afraid that I am unable to provide a reproducer now. It is a work I am doing for the OpenNMT-py , trying to use for doing lr update. And I encoutered this problem when testing the case. So I factored out the code skeleton about this problem above. I've tried several methods, including tricks like what suggests, but it still happens. Per my analysis, it is because the previous training was done on gpu, so when saving the , the stored states(tensors) are of version. During resuming, when we load the saved optimizer, loads this version to cpu(the model() can be moved to gpu easily, but seems lacking this ability?) , so this problem emerges.
Try moving optimizer state to the GPU memory manually after loading it from the checkpoint. I agree that having an method for this operation would be nice.
thanks. My work was suspended due to other problems, when resumed, I will try your method.
I'm afraid solution won't work. It will make the error go away, but your optimizer will no longer be training the model. You should recreate optimizers after casting modules to a different type or device, and you can use to restore the state from a previous copy. This currently doesn't work, but we should fix it (by copying from the data from the state dict, instead of using the tensors directly - this allows for cross-device or cross-type updates).
, yep, your method is what I currently use, it works. But I will wait for upstream to fix this problem though. Thanks for your great works!
Ah, my bad. I forgot to update the line where the optimizer is recreated. But otherwise, the following should do the job, right?
ah, right. That should work
Except that you should use instead of
I had a similar problem. When I save the optimizer state from a GPU other than GPU 0 and then load the state it still loads everything to GPU 0. Specifying in didn't work either. 's solution solves this though.
Hi guys, I have a very similar problem as the one in this thread, here's my code: And then once I resume, I got KeyErrors on my optimizer: Do you guys know how to fix this issue? BTW, I have 8 GPUs used, I'm guessing if this issue was because of that?
were you able to solve your issue with multiple-GPUs?
Thanks for asking. Yeah, I'm able to, turns out the issue is because I used an early version of PyTorch, after I updated the version, it works like a charm!
Just a comment, this problem is caused by makes all state tensor are moved into GPU0 so by moving the state of an optimizer to specific GPU will fix this problem.
Hi how do you know the specific GPU location of different tensors in advance?
Would a feature where all () calls always makes use of an automatically generated CPU version be feasible ? And then at resume the () would make use of the "current" device being used (or any better strategy). At the moment it seems we need a lot of boilerplater code to ensure saving and loading are consistents across devices for models/optimizer/scheduler/etc.
I have met similar problem, I recreated Adam optimizer without () after reloading model, () and DataParallel(model) according to solution.
thanks, it work!
Hi, as you say that: every time moving model to other device, we should build optimizer, but, if we move the model to other device and move back, should we build the optimizer again? here is an example code: does optimizer run as expected? also, if doing , should we rebuild optimizer ?
Is there a problem if you switch the order to something like this? Meaning moving the model to 'cuda' but only loading it's state dict from checkpoint after loading the optimizer's state dict first?
The problem can be concluded that the optimizer's state will be loaded to the device as same as the model. You must load the model to GPU at first, and then load the optimizer's state. So that both the model and the optimizer's state are loaded in GPU.
Instead of moving optimizer to cuda after loading it in cpu, you could load the checkpoint directly in cuda:
I've independently rediscovered that this works :) Should read until the end of the thread next time
I find my codes still have the problem. I tried my best to range the modules as the examples shown in the above. can anyone give me some hints? I also tried this
For me, doesn't solve the issue, but the one by works. Any idea why? I create the optimizer, load the state (with map location to CUDA), pass to the train loop, where the mode is pushed to device (though I assume it's not needed if the loaded model is pushed to cuda by maplocation). Specifically, I would get stuck at call, throwing the device mismatch error.
Still it doesn't work for me. Could you please help me?

Discussion:
def fn_base(optimizer, weight, bias): optimizer.zero_grad() <del> loss = (weight.mv(input) + bias).pow(2).sum() </del> <ins> i = input_cuda if weight.is_cuda else input loss = (weight.mv(i) + bias).pow(2).sum() </ins> loss.backward() return loss

# Make sure state dict wasn't modified self.assertEqual(state_dict, state_dict_c) <ins> # Check that state dict can be loaded even when we cast parameters # to a different type and move to a different device. if not torch.cuda.is_available(): return input_cuda = Variable(input.data.float().cuda()) weight_cuda = Variable(weight.data.float().cuda(), requires_grad=True) bias_cuda = Variable(bias.data.float().cuda(), requires_grad=True) optimizer_cuda = constructor(weight_cuda, bias_cuda) fn_cuda = functools.partial(fn_base, optimizer_cuda, weight_cuda, bias_cuda) state_dict = deepcopy(optimizer.state_dict()) state_dict_c = deepcopy(optimizer.state_dict()) optimizer_cuda.load_state_dict(state_dict_c) # Make sure state dict wasn't modified self.assertEqual(state_dict, state_dict_c) for i in range(20): optimizer.step(fn) optimizer_cuda.step(fn_cuda) self.assertEqual(weight, weight_cuda) self.assertEqual(bias, bias_cuda) </ins> def _test_basic_cases(self, constructor, ignore_multidevice=False): self._test_state_dict( torch.randn(10, 5),

line_search_fn = group['line_search_fn'] history_size = group['history_size'] <del> state = self.state['global_state'] </del> <ins> # NOTE: LBFGS has only global state, but we register it as state for # the first param, because this helps with casting in load_state_dict state = self.state[self._params[0]] </ins> state.setdefault('func_evals', 0) state.setdefault('n_iter', 0)

<del> from collections import defaultdict </del> <ins> from collections import defaultdict, Iterable </ins> import torch from copy import deepcopy

id_map = {old_id: p for old_id, p in zip(chain(*(g['params'] for g in saved_groups)), chain(*(g['params'] for g in groups)))} <del> state = defaultdict( dict, {id_map.get(k, k): v for k, v in state_dict['state'].items()}) </del> <ins> def cast(param, value): """Make a deep copy of value, casting all tensors to device of param.""" if torch.is_tensor(value): # Floating-point types are a bit special here. They are the only ones # that are assumed to always match the type of params. if any(tp in type(param.data).__name__ for tp in {'Half', 'Float', 'Double'}): value = value.type_as(param.data) value = value.cuda(param.get_device()) if param.is_cuda else value.cpu() return value elif isinstance(value, dict): return {k: cast(param, v) for k, v in value.items()} elif isinstance(value, Iterable): return type(value)(cast(param, v) for v in value) else: return value # Copy state assigned to params (and cast tensors to appropriate types). # State that is not assigned to params is copied as is (needed for # backward compatibility). state = defaultdict(dict) for k, v in state_dict['state'].items(): if k in id_map: param = id_map[k] state[param] = cast(param, v) else: state[k] = v </ins> # Update parameter groups, setting their 'params' value def update_group(group, new_group):