Nn.utils.parametrizations not working correctly?

musl: <p>Hi,<br>
I’m getting very different results switching from the old spectral_norm to the new <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.spectral_norm.html" rel="noopener nofollow ugc">parametrization</a>.<br>
Here is the output features of <a href="https://github.com/muslll/neosr/blob/master/neosr/archs/unet_arch.py" rel="noopener nofollow ugc">unet</a> using the old <code>torch.nn.utils.spectral_norm</code>:</p>
<pre><code class="lang-auto">tensor([[[[-0.0099, -0.0102, -0.0013,  ..., -0.0101, -0.0100, -0.0076],
          [-0.0214, -0.0407, -0.0349,  ..., -0.0185, -0.0293, -0.0054],
          [-0.0211, -0.0214, -0.0356,  ..., -0.0229, -0.0362, -0.0165],
          ...,
          [-0.0133, -0.0229, -0.0189,  ..., -0.0240, -0.0370, -0.0003],
          [-0.0028, -0.0027, -0.0099,  ..., -0.0031, -0.0209, -0.0099],
          [-0.0068, -0.0027, -0.0033,  ..., -0.0076,  0.0003,  0.0011]]],
</code></pre>
<p>And here is the output features using the new <code>torch.nn.utils.parametrizations.spectral_norm</code>:</p>
<pre><code class="lang-auto">tensor([[[[-6.4015e-03, -5.9812e-03, -5.5848e-04,  ..., -6.6797e-03,
           -5.4990e-03, -4.3101e-03],
          [-1.1050e-02, -2.3454e-02, -2.0890e-02,  ..., -1.0111e-02,
           -1.7187e-02, -2.5395e-03],
          [-1.0570e-02, -9.7215e-03, -1.9345e-02,  ..., -1.3104e-02,
           -2.1078e-02, -8.5044e-03],
          ...,
          [-7.6973e-03, -1.3273e-02, -9.9956e-03,  ..., -1.3413e-02,
           -2.1045e-02,  1.4347e-03],
          [-2.9391e-04,  5.1208e-05, -4.4562e-03,  ..., -7.5174e-04,
           -1.0988e-02, -5.6328e-03],
          [-3.7524e-03, -8.9492e-04, -8.4569e-04,  ..., -3.2808e-03,
            1.0416e-03,  1.3095e-03]]],
</code></pre>
<p>This is a bit-wise deterministic run, by the way.<br>
Here’s a naive way of showing that the old function produces less extremal values. Min/Max of tensor outputs from the last 10 iters of 100 iters:</p>
<p>Old:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>min</th>
<th>max</th>
</tr>
</thead>
<tbody>
<tr>
<td>-4.7022</td>
<td>-0.1812</td>
</tr>
<tr>
<td>-4.4896</td>
<td>-0.1821</td>
</tr>
<tr>
<td>-3.4386</td>
<td>23.3624</td>
</tr>
<tr>
<td>-4.4896</td>
<td>-0.1821</td>
</tr>
<tr>
<td>-4.5798</td>
<td>-0.1844</td>
</tr>
<tr>
<td>-1.1253</td>
<td>19.4105</td>
</tr>
<tr>
<td>-4.5797</td>
<td>-0.1844</td>
</tr>
<tr>
<td>-4.6552</td>
<td>-0.1855</td>
</tr>
<tr>
<td>0.2628</td>
<td>22.2968</td>
</tr>
<tr>
<td>-4.6551</td>
<td>-0.1855</td>
</tr>
</tbody>
</table>
</div><p>Parametrization:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>min</th>
<th>max</th>
</tr>
</thead>
<tbody>
<tr>
<td>-4.8454</td>
<td>-0.1950</td>
</tr>
<tr>
<td>-4.5237</td>
<td>-0.1968</td>
</tr>
<tr>
<td>-2.6446</td>
<td>32.7689</td>
</tr>
<tr>
<td>-4.5234</td>
<td>-0.1968</td>
</tr>
<tr>
<td>-4.7137</td>
<td>-0.2026</td>
</tr>
<tr>
<td>0.1529</td>
<td>27.5732</td>
</tr>
<tr>
<td>-4.7134</td>
<td>-0.2026</td>
</tr>
<tr>
<td>-4.8490</td>
<td>-0.2062</td>
</tr>
<tr>
<td>0.4218</td>
<td>31.8953</td>
</tr>
<tr>
<td>-4.8486</td>
<td>-0.2062</td>
</tr>
</tbody>
</table>
</div><p>So it looks like on the new function the max values are about 30% higher than on the old one.<br>
Is there a reason for that, or am I using it wrong? Thanks in advance.</p>