Pytorch & pickled tensors

username: mortazavi
cooked: <p>While loading a regular PyTorch (1.4.0) tensor (an encoding output, generated by another network in another training experiment), I run into the following problem . . .</p>
<p>When pickling the output of an encoder, what is the best practice?</p>
<p>For example, does the output, which is just a vector/tensor, need to be detached and copied prior to pickling?</p>
<p>Side notes:</p>
<ol>
<li>
</li>
</ol>
<p>I realize the stack trace below ends in a complaint about multi-processing process forking/spawning incompatibility . . .  but that might have been a side-effect of some earlier, serialization recovery attempt . . . The incompatibility of serialization is suspicious because torch version has not changed in my virtualenv . . .</p>
<ol start="2">
<li>
</li>
</ol>
<p>In case the multi-processing complaint is actually genuine,<br>
is it possible that inclusion of tensorboard’s summary writer as follows may also be a culprit<br>
“self.summary_writer = tf.summary.create_file_writer(self.log_dir)”</p>
<p>I can research these questions on my own but it would be great if someone more familiar with the serialization code clears up the first suspicion.</p>
<p>============</p>
<p>File ". . . . ", line . . . , in get_emb<br>
x_emb = pickle.load(x_path)<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/storage.py”, line 134, in _load_from_bytes<br>
return torch.load(io.BytesIO(b))<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/serialization.py”, line 529, in load<br>
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/serialization.py”, line 702, in _legacy_load<br>
result = unpickler.load()<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/serialization.py”, line 665, in persistent_load<br>
deserialized_objects[root_key] = restore_location(obj, location)<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/serialization.py”, line 156, in default_restore_location<br>
result = fn(storage, location)<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/serialization.py”, line 135, in _cuda_deserialize<br>
with torch.cuda.device(device):<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/cuda/<strong>init</strong>.py”, line 254, in <strong>enter</strong><br>
self.prev_idx = torch._C._cuda_getDevice()<br>
File “/home/ubuntu/ve/trn/lib/python3.6/site-packages/torch/cuda/<strong>init</strong>.py”, line 195, in _lazy_init<br>
"Cannot re-initialize CUDA in forked subprocess. " + msg)<br>
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>

username: mortazavi
cooked: <p>The problem persists even if SummaryWriter from torch.utils.tensorboard is used . . .<br>
This could have something to do with not detaching the output tensor from the compute graph prior to saving it . . .</p>

username: ptrblck
cooked: <p>Are you using <code>pickle</code> manually or are you trying to load the tensor via <code>torch.load</code>?<br>
Also, in the latter case, would <code>map_location='cpu'</code> help?<br>
Based on the error message it seems you are working with multiprocessing and the tensor loading is trying to re-initialize CUDA. Have you tried to use the suggested <code>spawn</code> method?</p>

username: mortazavi
cooked: <p>I resolved this problem . . . but unfortunately, I don’t recall how I did it. I will have to review the code later and write back.</p>