Can I deepcopy a model?

ONTDave: <p>There is some chatter online that I can’t deepcopy a model… Is this right?<br>
Additionally, is there a way after loading a model to move it between cpu and gpu?</p>

ptrblck: <p>You can deepcopy a model:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
model_copy = copy.deepcopy(model)

with torch.no_grad():
    model.weight.fill_(1.)

print(model.weight)
&gt; Parameter containing:
tensor([[10.]], requires_grad=True)

print(model_copy.weight)
&gt; Parameter containing:
tensor([[-0.5596]], requires_grad=True)
</code></pre>
<p>To move a model, just call:</p>
<pre><code class="lang-python">model.to('cuda:0')  # moves model (its parameters) to GPU0
model.to('cpu')  # moved model to CPU
</code></pre>

ONTDave: <p>Hey! Many thanks for the quick reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Pleased about the deepcopy.<br>
So the model.to syntax…<br>
I was under the impression this didn’t move the model, just changed its formatting as the documentation here (<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a>) suggests:<br>
“converts the initialized  <code>model</code>  to a CUDA optimized model using  <code>model.to(torch.device('cuda'))</code> .”<br>
It sounds like the map_location bit in torch.load(PATH, map_location=device) specifies where the model is?</p>

ptrblck: <p>The <code>map_location</code> argument specifies where to put the loaded parameters.<br>
E.g. if you saved the <code>model.state_dict()</code> of a model, which was pushed to the GPU, and would like to load this <code>state_dict</code> on a CPU-only machine, you could specify <code>map_location='cpu'</code> to restore the parameters.</p>

ONTDave: <p>Does that mean that if I load a model from file (saved using torch.save(model) on either a cpu or gpu) and set map_location=‘cpu’ when loading, I don’t need to call .to - if I’m using it on the cpu. I only need to .to it if I’m moving it onto the gpu once again?</p>

ptrblck: <p>I would recommend to save and load the <code>mode.state_dict()</code>, not the model directly.<br>
That being said, I prefer to push the model to CPU first before saving the <code>state_dict</code>.<br>
This approach makes sure that I’m able to restore the model on all systems, even when no GPU was found.</p>
<p>After loading the model, I use <code>model.to('cuda')</code> to push it to the GPU again.</p>

pinocchio: <p>this is sufficient:</p>
<pre><code class="lang-auto">model = nn.Linear(1, 1)
model_copy = copy.deepcopy(model)
</code></pre>

Ofri_Masad: <p>what about this code:</p>
<p>`<br>
model = nn.Linear(1, 1)</p>
<p>model.to(‘cuda:0’)</p>
<p>model_copy = copy.deepcopy(model)<br>
`<br>
will this create a copy in the GPU? or will I now have two models pointing to the same location in the GPU? (hopefully not)</p>
<p>thanks</p>

ynjiun_wang: <p>Hi, I have a customized model and cannot be deepcopy. What would be the major causes of a model cannot be deepcopied? could you shed some light for me to debug my model? Thanks a lot for your help.</p>
<p>The customized model is the squeezenet ssd lite model in this repo (<a href="https://github.com/qfgaohao/pytorch-ssd" class="inline-onebox" rel="noopener nofollow ugc">GitHub - qfgaohao/pytorch-ssd: MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in Pytorch 1.0 / Pytorch 0.4. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas like CoordConv.</a>)</p>
<p>The squeezenet model is defined under vision/nn<br>
The ssd model is defined under vision/ssd</p>
<p>if I do:<br>
base_net = squeezenet1_1(False).features<br>
I can deep copy base_net no problem:<br>
dbcpy = copy.deepcopy(base_net)</p>
<p>But if I do<br>
net = create_squeezenet_ssd_lite(no_classes, is_test=True)<br>
then I have trouble to deepcopy the net:<br>
dnet = copy.deepcopy(net)</p>
<p>it will report the following error:<br>
Traceback (most recent call last):<br>
File “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 3129, in <br>
main()<br>
File “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 3122, in main<br>
globals = debugger.run(setup[‘file’], None, None, is_module)<br>
File “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 2195, in run<br>
return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)<br>
File “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 2202, in _exec<br>
pydev_imports.execfile(file, globals, locals)  # execute the script<br>
File “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/_pydev_imps/_pydev_execfile.py”, line 25, in execfile<br>
exec(compile(contents+"\n", file, ‘exec’), glob, loc)<br>
File “/home/paul/pytorch/od-ssd/ssd-quantized/test_3d.py”, line 50, in <br>
dnet = copy.deepcopy(net)<br>
File “/usr/lib/python3.6/copy.py”, line 180, in deepcopy<br>
y = _reconstruct(x, memo, *rv)<br>
File “/usr/lib/python3.6/copy.py”, line 280, in _reconstruct<br>
state = deepcopy(state, memo)<br>
File “/usr/lib/python3.6/copy.py”, line 150, in deepcopy<br>
y = copier(x, memo)<br>
File “/usr/lib/python3.6/copy.py”, line 240, in _deepcopy_dict<br>
y[deepcopy(key, memo)] = deepcopy(value, memo)<br>
File “/usr/lib/python3.6/copy.py”, line 169, in deepcopy<br>
rv = reductor(4)<br>
TypeError: can’t pickle module objects</p>
<p>The reason I am asking this question is when I do the QAT (Quantization Aware Training), and try to save the quantized model, using:<br>
net.eval()<br>
net_int8 = torch.quantization.convert(net)<br>
net_int8.save(model_path)<br>
I will encounter the above deepcopy error. So what shall I do? fix the model to make it deepcopy-able? or QAT doesn’t work for ssd model? could you help to point me to the right direction? Thanks a lot for your help!</p>

ptrblck: <p>Based on the error message it seems that <code>pickle</code> is failing to copy the object, if it has a <a href="https://stackoverflow.com/a/2790884">class attribute that references a module</a>. I don’t know, if this is the case for QAT or if this is caused by your custom model, but could you check for module references inside your model?</p>

ynjiun_wang: <p>Thanks for your help. I looked into the create_sqeezenet_ssd_lite() code below, do you think that ModuleList() causing the problem? If  yes, how do I fix this type of issue and enable the model created by this code deepcopyable? This type of model creation is typically used in a SSD  network creation. Does this imply QAT doesn’t work for SSD type of network? Thanks again for your help!</p>
<blockquote>
<p>def create_squeezenet_ssd_lite(num_classes, is_test=False):<br>
base_net = squeezenet1_1(False).features  # disable dropout layer</p>
<pre><code>source_layer_indexes = [
    12
]
extras = ModuleList([
    Sequential(
        Conv2d(in_channels=512, out_channels=256, kernel_size=1),
        ReLU(),
        SeperableConv2d(in_channels=256, out_channels=512, kernel_size=3, stride=2, padding=2),
    ),
    Sequential(
        Conv2d(in_channels=512, out_channels=256, kernel_size=1),
        ReLU(),
        SeperableConv2d(in_channels=256, out_channels=512, kernel_size=3, stride=2, padding=1),
    ),
    Sequential(
        Conv2d(in_channels=512, out_channels=128, kernel_size=1),
        ReLU(),
        SeperableConv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1),
    ),
    Sequential(
        Conv2d(in_channels=256, out_channels=128, kernel_size=1),
        ReLU(),
        SeperableConv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1),
    ),
    Sequential(
        Conv2d(in_channels=256, out_channels=128, kernel_size=1),
        ReLU(),
        SeperableConv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1)
    )
])

regression_headers = ModuleList([
    SeperableConv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=256, out_channels=6 * 4, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=256, out_channels=6 * 4, kernel_size=3, padding=1),
    Conv2d(in_channels=256, out_channels=6 * 4, kernel_size=1),
])

classification_headers = ModuleList([
    SeperableConv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=3, padding=1),
    SeperableConv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=3, padding=1),
    Conv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=1),
])

return SSD(num_classes, base_net, source_layer_indexes,
           extras, classification_headers, regression_headers, is_test=is_test, config=config)
</code></pre>
</blockquote>

ptrblck: <p>The <code>nn.ModuleList</code> looks correct, but I also don’t know if QAT might be causing this issue. Are you seeing the same error without using QAT?</p>

ynjiun_wang: <p>“Are you seeing the same error without using QAT?”</p>
<p>Yes.</p>
<p>I am seeing this link indicating that <a href="https://github.com/pytorch/pytorch/issues/7545" rel="noopener nofollow ugc">Can’t save a model with <code>torch.save</code> if model has a <code>torch.Device</code> attr #7545</a>  and I tried to remove the self.device attribute in SSD class and still not working.</p>
<p>I did try to run the code you suggested in the <a href="https://stackoverflow.com/questions/2790828/python-cant-pickle-module-objects-error/2790884#2790884" rel="noopener nofollow ugc">link</a> to exam where is the error by pickle_trick() it:</p>
<blockquote>
<p>net = create_squeezenet_ssd_lite(no_classes, is_test=True)<br>
print(pf(pickle_trick(net)))</p>
</blockquote>
<p>But the code crashes indicating there is exception caught within the exception even I increase the maximum depth to 10,000.</p>

ynjiun_wang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>after line by line elimination test, finally found out the root causes that make the SSD() not deepcopyable:</p>
<blockquote>
<p>class SSD(nn.Module):<br>
def <strong>init</strong>(self, num_classes: int, base_net: nn.ModuleList, source_layer_indexes: List[int],<br>
extras: nn.ModuleList, classification_headers: nn.ModuleList,<br>
regression_headers: nn.ModuleList, is_test=False, config=None, device=None):<br>
“”“Compose a SSD model using the given components.<br>
“””<br>
super(SSD, self).<strong>init</strong>()</p>
<pre><code>    self.num_classes = num_classes
    self.base_net = base_net
    self.source_layer_indexes = source_layer_indexes
    self.extras = extras
    self.classification_headers = classification_headers
    self.regression_headers = regression_headers
    self.is_test = is_test
    self.config = config  
</code></pre>
</blockquote>
<p>If I comment out the “self.config = config” line, then SSD() is deepcopyable.</p>
<p>Where the config is a module imported from:</p>
<blockquote>
<p>from .config import squeezenet_ssd_config as config</p>
</blockquote>
<p>And the squeezenet_ssd_config is as below:</p>
<blockquote>
<p>import numpy as np</p>
<p>from vision.utils.box_utils import SSDSpec, SSDBoxSizes, generate_ssd_priors</p>
<p>image_size = 300<br>
image_mean = np.array([127, 127, 127])  # RGB layout<br>
image_std = 128.0<br>
iou_threshold = 0.45<br>
center_variance = 0.1<br>
size_variance = 0.2</p>
<p>specs = [<br>
SSDSpec(17, 16, SSDBoxSizes(60, 105), [2, 3]),<br>
SSDSpec(10, 32, SSDBoxSizes(105, 150), [2, 3]),<br>
SSDSpec(5, 64, SSDBoxSizes(150, 195), [2, 3]),<br>
SSDSpec(3, 100, SSDBoxSizes(195, 240), [2, 3]),<br>
SSDSpec(2, 150, SSDBoxSizes(240, 285), [2, 3]),<br>
SSDSpec(1, 300, SSDBoxSizes(285, 330), [2, 3])<br>
]</p>
<p>priors = generate_ssd_priors(specs, image_size)</p>
</blockquote>
<p>How should I modify the code to keep the “config” and still make the SSD() deepcopyable? Any advice will be highly appreciated.</p>
<p>Thanks a lot for your help in advanced.</p>

ptrblck: <p>Is <code>squeezenet_ssd_config</code> only containing the posted code or any class definitions?<br>
Based on the <code>import</code> statement I would assume it’s a class or any other object, but the posted code shows just executable Python code with more imports.<br>
You could try to <code>deepcopy</code> each imported class and see, if one of these classes might fail due to the previously mentioned reason.</p>

ynjiun_wang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you so much for your help.</p>
<p>Eventually I “work around” the problem by doing the following:</p>
<p>avoid “self.config=config” referencing, but directly instantiate it’s content:</p>
<p>Was:</p>
<blockquote>
<p>in init<br>
self.config = config<br>
…&lt;later in the body, self.config were referenced as&gt;<br>
self.config.center_variance<br>
self.config.size_variance</p>
</blockquote>
<p>Change to:</p>
<blockquote>
<p>in init<br>
<span class="hashtag-raw">#self</span>.config = config <span class="hashtag-raw">#comment</span> out<br>
self.config_center_variance = config.center_variance<br>
self.config_size_variance = config.size_variance<br>
…&lt;later in the code, no more reference to self.config &gt;<br>
self.config_center_variance<br>
self.config_size_variance</p>
</blockquote>
<p>Then the SSD() is deepcopyable, but QAT still having problems…</p>

ynjiun_wang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>although SSD() is deepcopyable, but still having problem to run QAT…, the problem is: cannot torch.save(net_int8) and it complaints:</p>
<blockquote>
<pre><code>torch.save(net_int8, model_path) #to save the entire model
</code></pre>
<p>File “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 370, in save<br>
_legacy_save(obj, opened_file, pickle_module, pickle_protocol)<br>
File “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 443, in _legacy_save<br>
pickler.dump(obj)<br>
AttributeError: Can’t pickle local object ‘_with_args.._PartialWrapper’</p>
</blockquote>
<p>the code with problem area is:</p>
<blockquote>
<pre><code>            net.eval()
            net_int8 = torch.quantization.convert(net)
             
            #torch.save(net_int8.state_dict(), model_path)
            torch.save(net_int8, model_path) #to save the entire model
</code></pre>
</blockquote>
<p>If I only save state_dict(), there is no problem. But when save the entire model, the problem show.</p>
<p>Further debug, finding the net was loaded and torch.save-able, but after the net.qconfig and it becomes not torch.save-able anymore:</p>
<blockquote>
<pre><code>torch.save(net,"./net_before_qconfig")
net.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
torch.save(net,"./net_after_qconfig")
</code></pre>
</blockquote>
<p>As shown above, the torch.save(net, “./net_before_qconfig”) works, but after net.qconfig, the net is not torch.save-able anymore… ;((</p>
<p>And having the same complaints:</p>
<blockquote>
<pre><code>torch.save(net,"./net_after_qconfig")
</code></pre>
<p>File “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 370, in save<br>
_legacy_save(obj, opened_file, pickle_module, pickle_protocol)<br>
File “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 443, in _legacy_save<br>
pickler.dump(obj)<br>
AttributeError: Can’t pickle local object ‘_with_args.._PartialWrapper’</p>
</blockquote>
<p>Is this QAT problem? torch.save cannot save the entire model after QAT? I need the entire model saved to port to a micro processor for speed up testing. If this is QAT limitation, what would be the work around to still save the entire quantization model (not just state_dict)? Thank you again for your help!</p>

ptrblck: <p>Based on your latest debugging, the issue might be related to QAT.<br>
I’m not deeply familiar with QAT and the support to use <code>torch.save</code> on the model directly.<br>
However, generally I would not recommend to save the model directly, as it can break in various ways.<br>
The better way would be to save the <code>state_dict</code> (and additional configs etc.). When reloading you would then recreate the model object and use <code>load_state_dict</code>.</p>

n40x1: <p>I using Pytorch’s <code>swa_utils</code> that internally calls <code>deepcopy</code>.<br>
However, I get the following error:</p>
<pre><code class="lang-auto">&gt; Only Tensors created explicitly by the user (graph leaves) 
support the deepcopy protocol at the moment
</code></pre>
<p>This seems to be caused by the weight dropout scheme I am using, where the original weight matrix is moved into _raw. The code for weight dropout is here:</p>
<pre><code class="lang-auto">import torch
from torch.nn import Parameter
import torch.nn.functional as F 

class WeightDrop(object):
    def __init__(self, name, dropout):
        self.name = name
        self.dropout = dropout

    def compute_weight(self, module):
        return F.dropout(
            getattr(module, self.name + "_raw"),
            p        = self.dropout,
            training = module.training,
            inplace  = False
        )

    @staticmethod
    def apply(module, name, dropout):
        for k, hook in module._forward_pre_hooks.items():
            if isinstance(hook, WeightDrop) and hook.name == name:
                raise RuntimeError(f"Cannot register two weight_dropout hooks with name '{name}'")
        fn = WeightDrop(name, dropout)
        weight = getattr(module, name)

        del module._parameters[name]

        ## creating _raw parameter
        module.register_parameter(name + "_raw", Parameter(weight.data))
        setattr(module, name, fn.compute_weight(module))

        module.register_forward_pre_hook(fn)
        return fn

    def remove(self, module):
        weight = module._parameters[self.name + "_raw"]
        delattr(module, self.name)
        del module._parameters[self.name + "_raw"]
        module.register_parameter(self.name, Parameter(weight.data))

    def __call__(self, module, inputs):
        if self.name in module._parameters:
            del module._parameters[self.name]
        setattr(module, self.name, self.compute_weight(module))


def weight_drop(module, name, dropout):
    WeightDrop.apply(module, name, dropout)
    return module

def apply_weight_drop(module, name, dropout):
    wdrop = weight_drop(module, name, dropout)
    fp = module.flatten_parameters

    def decorator(*args, **kwargs):
        device = getattr(module, name + "_raw").device
        setattr(module, name, getattr(module, name).to(device))
        return fp(*args, **kwargs)

    module.flatten_parameters = decorator
    return wdrop
</code></pre>
<p>Here is the code for applying deep copy on a GRU:</p>
<pre><code class="lang-auto">import copy
gru = torch.nn.GRU(10, 10)
gru_wd = apply_weight_drop(gru, "weight_hh_l0", 0.2)
gru_wd_copy = copy.deepcopy(gru_wd)
&gt; RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
</code></pre>
<p>Any ideas how to fix this error?<br>
Thank you!</p>

Nikola_Andro: <p>Any updates on this question?</p>
<p>I checked by doing the following:</p>
<pre><code class="lang-auto">model = nn.Linear(1, 1)

model.to(‘cuda:0’)

model_copy = copy.deepcopy(model)

print(next(model_copy.parameters()).is_cuda)
</code></pre>
<p>Output:</p>
<p><code>True</code></p>
<p>So the copy should be on CUDA as well.</p>