Copy_() and memory format

seliad: <p>I know that in pytorch 1.5 <code>to()</code> and <code>clone()</code> can preserve formats and therefore we can send non-contiguous tensors between devices.</p>
<p>I wonder, what is the case for <code>copy_()</code>?  can we send non-contagious tensors with it?</p>
<p>If not, is there any suggested workaround for avoiding copy?<br>
for example</p>
<pre><code class="lang-auto">a = torch.randn(10,1, device="cuda:1").share_memory_()
b = torch.randn(10,2, device="cuda:0")
b = torch.transpose(b, 0,1)
a.copy_(b)  # is it OK?
</code></pre>
<p>In the example above we want to avoid using <code>to()</code>/<code>clone()</code> to avoid creating a new tensor and moving it to shared memory.</p>

albanD: <p>Hi,</p>
<p><code>.copy_()</code> will not change the contiguity of any Tensor.<br>
It will just read the content from b and write it to a. Not changing the size/strides.</p>

VitalyFedyunin: <p><code>a.copy_</code> is inplace operation and it never changes strides (and memory format) of <code>a</code>. So the result of <code>a.copy_(b)</code> going to be <code>a</code> with data of <code>b</code> and strides of <code>a</code>.</p>

seliad: <p>Thanks! Is there a way to copy the stride?</p>

VitalyFedyunin: <p>If <code>a</code> is the same size as <code>b</code>, you can aggressively restride <code>a</code> with <code>a.as_strided_(b.shape, b.stride())</code> and do <code>a.copy_(b)</code> as next step. But this will surely break autograd.</p>

albanD: <p>For (good) reasons, <code>.as_strided()</code> is actually supported by the autograd, so that will work <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>But even beyond that, here, since you override all the content of <code>a</code> anyway with the copy, all the gradients will flow towards <code>b</code> and the original value of <code>a</code> will just get 0s.<br>
So autograd will work just fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

seliad: <p>I somewhat still get data corruption, doing just the forward pass and calling <code>as_strided_</code> when sizes are equal.<br>
If I replace <code>copy_()</code> with <code>to()</code>, its totally OK.</p>
<p>I wonder if its related to cuda streams or something?<br>
<code>a.copy_(b)</code> when a,b are on different devices?</p>

albanD: <p>Well, it’s not because they have the same size that the as_strided will be valid.<br>
Is <code>a</code> contiguous ? If <code>a</code> has some overlapping memory already, then it’s backing memory won’t be big enough <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

seliad: <p>I made sure its contiguous too<br>
Here is what I did.</p>
<pre><code class="lang-auto">device = ...
ranks = [....]
saved = [None for rank in ranks]
a = saved[rank]
if a is not None and b.size() == a.size() and b.storage_offset() == a.storage_offset() and b.stride() == a.stride() and b.is_contiguous() and a.is_contiguous():
    # no need to call as_strided_
    a.copy_(b)
else:
    a = b.to(device)
    saved[rank] = a
</code></pre>
<p>when we replace that ugly if with <code>if False:</code>  everything works.<br>
(what happens next is sending a through queue and cloning the the receiver, and then normal neural net)</p>

albanD: <p>One if the risks of <code>as_strided</code> is that you can do fairly bad stuff. In particular, these are no checks and you can end up reading out of bounds of the original Tensor’s values (or even out of the Storage backing it).</p>
<p>In the ocd above, if you just do <code>.to()</code> the first time and <code>.copy_()</code> afterwards. The layout of a will just be the layout of the first b. Is that not ok?<br>
Why is it so important to keep b’s layout at each iteration?</p>

seliad: <p>I thought it would be OK (that’s exactly what I did at first, just the <code>is not None</code> check)<br>
but it didn’t work.<br>
Then I gradually increased the checks.</p>

albanD: <p>So you mean this does not work?</p>
<pre><code class="lang-auto">device = ...
ranks = [....]
saved = [None for rank in ranks]
a = saved[rank]
if a is not None:
    a.copy_(b)
else:
    a = b.to(device)
    saved[rank] = a
</code></pre>
<p>What is the issue you see with this?</p>

seliad: <p>exactly.</p>
<p>I tried 2 networks.</p>
<ol>
<li>WideResnet: it works fine with <code>copy_()</code>. all <code>b</code> tensors are contiguous there.</li>
<li>GPT2 (from huggingface): does not work. I know that some <code>b</code> tensors are non-contiguous.</li>
</ol>
<p>For the GPT2 the task is zero-shot on wikitext2.<br>
With <code>to(device)</code> I restore the perplexity from the paper (~29)<br>
With <code>copy_()</code> the perplexity explodes.(high crazy numbers, like 8400, 121931 and so on)</p>
<p>I tired to look at the tensors with the debugger, they look fine.</p>

albanD: <p>Ho ok… that looks more like a multiprocessing issue then if the Tensors looks fine?<br>
Layout won’t change anything about the values computed (unless bugs). So you should not see any difference here !</p>
<p>Can you make a small code sample that repro this?</p>

VitalyFedyunin: <p>Can you please try to <code>cuda.synchronize()</code> after <code>copy_</code> calls? When you are looking at tensors in debugger you are actually synchronizing to make them (data) observable.</p>

seliad: <p>Hi,<br>
I found the bug. It was indeed was some multiprocessing/threading/multi-cuda-stream/ issue + contiguous we discussed.<br>
Everything solved and <code>copy_()</code> works. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

seliad: <p>In  case it will help someone:<br>
I actually do notice some deadlocks when combining <code>to()</code> and <code>copy_()</code> like mentioned above.<br>
Deadlock happens inside (i.e compiled cuda code) of <code>to()</code> at the second call, that is<br>
<code>to()</code>, <code>copy_()</code>, <code>copy_()</code>,…,<code>copy_()</code>, <code>to()</code>, deadlock.<br>
Anyway, I got frustrated and now I’m using only <code>to()</code>, as its quite minor optimization I wasted too much time on.</p>
<p>I think its related to RTX2080ti not supporting p2p (I looked at the cuda code that does the <code>to()</code>  few weeks ago, and I think it assumes something about p2p, but I did not bother to check it thoroughly.)</p>
<p>I can’t share my code (yet), but as soon as I will I’ll share the full example</p>

albanD: <p>Hi,</p>
<p>If you have a repro that deadlock on a given hardware (and is reproducible on other similar card to be sure it’s not a faulty card), please open an issue on github!  Thanks.</p>