What can cause "[WARNING] DeviceCopy in input program"?

mpagli: <p>I have the following warnings:</p>
<pre><code class="lang-auto">[2023-03-22 12:32:20,041] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,163] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:23,366] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,174] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,333] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,465] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,598] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,730] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-03-22 12:32:27,863] torch._inductor.utils: [WARNING] DeviceCopy in input program
</code></pre>
<p>I failed to find a good description of what this warning means. Any idea?</p>

ptrblck: <p>The warning seems to be raised from <a href="https://github.com/pytorch/pytorch/blob/a7856e18a7e83f894461ed3e236aa4d4eb7727b3/torch/_inductor/ir.py#L2881">here</a> and based on the definition of <a href="https://github.com/pytorch/pytorch/blob/a7856e18a7e83f894461ed3e236aa4d4eb7727b3/torch/_inductor/utils.py#L629-L638"><code>developer_warning</code></a> I assume it should not be enabled in stable releases. Are you using the latest <code>2.0.0</code> release or a nightly build?</p>

sjyhne: <p>I’m getting the same warning after installing it through this: <a href="https://pytorch.org/get-started/pytorch-2.0/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2.0 | PyTorch</a></p>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue, as I’m not seeing this warning using e.g. this simple model:</p>
<pre><code class="lang-python">model = models.resnet50().cuda()
x = torch.randn(1, 3, 224, 224, device="cuda")

model = torch.compile(model)
out = model(x)
</code></pre>

sjyhne: <p>It seems to be related to the use of Lightning in my case. I tried a minimal snippet, though, with my model, and only got this warning:</p>
<p>[2023-03-23 10:58:49,540] torch._inductor.utils: [WARNING] using triton random, expect difference from eager</p>

mpagli: <p>I’m using a nightly build 2.1.0.dev20230314+cu118. I’ll try with the latest stable release asap.</p>
<p>I only have this warning with one of my models and I couldn’t find which part of the code triggers it. If you have some insight into what could cause this I might be able to provide a minimal executable code snippet.</p>

marksaroufim: <p>The trigger is likely to be a Device copy which is copying a tensor from one device to another - I would try removing to <code>x_cuda1 = x.to("cuda:1")</code> calls like this in your code and see if it solves the problem</p>

ptrblck: <p>That’s right and indeed a simple tensor copy triggers the warning:</p>
<pre><code class="lang-python">def fun(x):
    x = x.to("cuda:0")
    
fun_compiled = torch.compile(fun)

x = torch.randn(1)
out = fun_compiled(x)
# [2023-03-23 10:08:02,025] torch._inductor.utils: [WARNING] DeviceCopy in input program
</code></pre>
<p>Did we miss to disable these warning for the stable release?</p>

marksaroufim: <p>Yeah this was a miss - just made the PR <a href="https://github.com/pytorch/pytorch/pull/97451" class="inline-onebox" rel="noopener nofollow ugc">Disable inductor developer warnings in official releases by msaroufim · Pull Request #97451 · pytorch/pytorch · GitHub</a></p>

aifartist: <p>What does “input program” mean in this context?<br>
For stable diffusion of course there is a transfer of the image off of the GPU after the VAE/decode_first_stage step.  Also, I suspect the initial noise is transferred to the GPU at the beginning.  It’d be nice to know where in the complex set of operations that Stable Diffusion is, where this is coming from.</p>
<p>I have torch 2.0.1+cu118 and trying the default ‘inductor’.</p>