Trying to understand AffineChannel operation

username: JoshVarty
cooked: <p>I’m trying to reproduce RetinaNet in PyTorch by directly porting the original <a href="https://github.com/facebookresearch/Detectron/tree/master/detectron" rel="nofollow noopener">Caffe implementation</a>. One issue I’ve run into is that Detectron makes use of a normalization operation called <code>AffineChannel</code> instead of batch normalization. This is due to the small batch sizes one encounters when training object detection models.</p>
<p>AffineChannelOp: <a href="https://github.com/pytorch/pytorch/blob/54c4b5a4db6e939fd441a00c42d185547e77aaa2/caffe2/operators/affine_channel_op.h#L32-L57" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/54c4b5a4db6e939fd441a00c42d185547e77aaa2/caffe2/operators/affine_channel_op.h#L32-L57</a></p>
<p>Cuda Kernel for AffineChanne: <a href="https://github.com/pytorch/pytorch/blob/2db847b3a7edc48652e144e7c9d7aa0bbed66aaa/caffe2/utils/math/broadcast.cu#L22-L41" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/2db847b3a7edc48652e144e7c9d7aa0bbed66aaa/caffe2/utils/math/broadcast.cu#L22-L41</a></p>
<p>Am I correct in my understanding that <code>AffineChannel</code> simply multiplies each channel but it’s own learnable scale parameter and adds a learnable bias parameter?</p>
<p>Is there any evidence that this helps learning? I haven’t encountered this approach before.</p>

username: ptrblck
cooked: <p>Based on your description and skimming through the code, it seems you are right in your assumption.<br>
Wouldn’t the operation thus correspond to a <code>nn.BatchNorm2d</code> layer with <code>track_running_stats=False</code>? This might make sense, if the batch size is small, as the running estimates will most likely be quite noisy.</p>

username: JoshVarty
cooked: <p>Yes I think you might be right. I’ve never used <code>track_running_stats</code> before and I see it’s described as:</p>
<pre><code class="lang-auto">track_running_stats: a boolean value that when set to `True`, this
            module tracks the running mean and variance, and when set to `False`,
            this module does not track such statistics and always uses batch
            statistics in both training and eval modes. Default: `True`
</code></pre>
<p>It sounds like while it doesn’t track a running mean/variance, it is still using the batch mean and variance? In my case I think I would want a fixed mean of <code>0</code> and a variance of <code>1</code>, but still allow for <code>gamma</code> and <code>beta</code> to be learnable parameters, correct?</p>

username: ptrblck
cooked: <p>Yeah, you are right, sorry for the mistake.<br>
It would rather correspond to <code>track_running_stats=True</code> and set to evaluation mode (<code>bn.eval()</code>), which would then use the initial running stats.</p>

username: JoshVarty
cooked: <blockquote>
<p>It would rather correspond to  <code>track_running_stats=True</code>  and set to evaluation mode ( <code>bn.eval()</code> ), which would then use the initial running stats.</p>
</blockquote>
<p>That sounds like it will freeze the statistics (eg. mean and variance) but it also sounds like it will freeze <code>gamma</code> and <code>beta</code> which appear to be learnable in <code>AffineChannel</code>.</p>
<p>I think I might be able to mimic the<code>AffineChannel</code> operation by creating a vector of <code>torch.ones([num_channels])</code> and <code>torch.zeros([num_channels])</code> and using these for <code>gamma</code> and <code>beta</code> respectively. I’ll give it a shot and see how it goes.</p>

username: ptrblck
cooked: <p><code>bn.eval()</code> will not freeze the trainable parameters, just use the running statistics, which would have their initial values.</p>