How to make torch.nn.functional.interpolate deterministic?

sbelharbi: <p>Hi,<br>
Is there a way to make <code>torch.nn.functional.interpolate</code> deterministic? (its backward is <a href="https://pytorch.org/docs/1.0.0/_modules/torch/nn/functional.html#interpolate" rel="noopener nofollow ugc">not deterministic</a>. This seems the case when upscaling. Downscaling seems to be deterministic).</p>
<blockquote>
<p>When using the CUDA backend, this operation may induce <strong>nondeterministic behaviour in be backward</strong> that is not easily switched off. Please see the notes on <a href="https://pytorch.org/docs/1.0.0/notes/randomness.html" rel="noopener nofollow ugc">Reproducibility</a> for background.</p>
</blockquote>
<p><a href="https://discuss.pytorch.org/t/non-deterministic-behavior-of-pytorch-upsample-interpolate/42842">Related</a> (with more details) topic.</p>
<p>Thanks!</p>

sbelharbi: <p>Any news on this?<br>
<code>torch.nn.ConvTranspose2d</code> does not work as well as <code>torch.nn.functional.interpolate</code>. I want to upscale some feature maps right away (in a non-parametric way) without doing transposed convolution many times!! this last one seems to lose information!!!</p>
<ol>
<li>
<code>torch.nn.functional.interpolate</code> seems to work but it is not deterministic (unstable results)</li>
<li>
<code>torch.nn.ConvTranspose2d</code> does not seem to work, but it is deterministic (stable results).</li>
</ol>

MariosOreo: <p>Hi,</p>
<p>If you want to get stable results, I am afraid that you should use <code>nn.ConvTranspose2d</code> instead of <code>interpolate</code>. I agree the view you mentioned that the transposed convolution makes the information lose.</p>
<p>I evaluated that the transposed convolution loses the more information as the bigger upsampling rate used (caused by its implemented mechanism). For counterworking lose information, you could add some addtion block to refine the output from convtransposed convlution.</p>
<p>But it is more expensive compared to the parameter-free operation <code>interpolate</code>.</p>

sbelharbi: <p>Hi <a class="mention" href="/u/mariosoreo">@MariosOreo</a> ,<br>
Thanks for your comment!<br>
Unfortunately, in my case, the upscaling factor is huge! (<code>x32</code>) from a single map of <code>15x15</code>. I assume that’s why <code>F.interpolate</code> is extremely unstable, and what makes transposed convolution (as you mentioned) lose information (it requires <code>4</code> transposed convolution). The learning in my case, for some considerations, is achieved without output target which makes the spatial loss of information worse (I think that most architectures that work using upscaling, in segmentation for instance, it is due to the supervised target provided at the output that allows the transposed convolution to learn something spatial). In my experiments, the transposed convolution loses entirely the spatial information that was present in the map of <code>15x15</code>  (compared to <code>F.interpolate</code> which preserves in a nice way the learned abstract spatial activations).</p>
<p>I wonder if people working on GANs and super-resolution come across such issue of instability of the interpolation? (by the way, it makes learning unstable and difficult since the backward is not deterministic which makes the gradient non-deterministic as well).</p>
<p>I checked some super-resolution (SR) papers, and it seems that interpolation, and transposed convolution is the tool to go. However, often, the interpolation is done over the input image then it is fed to the network (I do not think they include it in the training graph. So, it does not have an impact on the stability).</p>
<p>Actually, I am exploring another tool that mimics interpolation and which was developed within SR domain: <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.PixelShuffle" rel="nofollow noopener">pixel shuffle</a> (<a href="https://arxiv.org/abs/1609.05158" rel="nofollow noopener">paper</a>).  If one wants to upscale a map of ‘(h, w)’ by a factor of <code>xR</code>, one needs to have ‘R^2’ feature maps to obtain an upscaled map of <code>(R*h, R*w)</code>. I am doing this in two ways:</p>
<ol>
<li>The way it is supposed to be done: learn a feature map of <code>(#batch, R*R, h, w)</code>, then call <code>torch.nn.functional.pixel_shuffle</code>. It behaves the same way as transposed convolution: total loss of spatial information.</li>
<li>Duplicate the map to be upscaled, of size <code>15x15</code>, <code>RxR</code> times using <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand" rel="nofollow noopener"><code>torch.Tensor.expand()</code></a>: <code>my_map.expand(-1, RxR, -1, -1)</code>. Pixel shuffle seems to upscale right away the map as it is. Two downsides (<code>R=32</code>):
<ol>
<li>Since I duplicate each pixel in the small map <code>32x23</code> times the upscaled map is not smooth compared to <code>F.interpolate</code>. It is pixelized, and it does not look like it was upscaled (as we know upscaling. It looks like when you extremely zoom into an image and you start seeing pixels!!!). At this point, the main advantage of the interpolation is that it does interpolation!!! (i.e., it adds new values to cover the new large scale) while the way I did it using duplication does not add anything new.</li>
<li>Probably due to this unusual way to upscale, after the first update during training, the layer before this operation starts to output ONLY large positive values (up to <code>10^3</code>). So, it is a problem. The network does not learn (compared to when using interpolation).</li>
</ol>
</li>
</ol>
<p>Note: <code>torch.nn.functional.pixel_shuffle</code> is deterministic.</p>
<p>So, for now, I did not find something that does the job as <code>F.interpolate</code>.</p>

dashesy: <p>Just as a suggestion, have you tried to use depth_to_space?</p>
<ol>
<li>Use convolution to increase the depth</li>
<li>Use depth_to_space (combination of view, permute to shuffle) to increase the size</li>
</ol>
<p>Would that be more stable than interpolate?</p>