How to implement conditional instance normalization

nima_rafiee: <p>Hi</p>
<p>I’m going to implement conditional instance normalization. what I planned to do is to pass the weight and bias for each sample as a second argument to the forward function as follows.</p>
<pre><code class="lang-auto">class ConditionalInstanceNorm(Module):
    
    __constants__ = ['num_groups', 'num_channels', 'eps', 'affine']
    num_groups: int
    num_channels: int
    eps: float
    affine: bool

    def __init__(self, num_channels: int, eps: float = 1e-5, affine: bool = True) -&gt; None:
        super(ConditionalInstanceNorm, self).__init__()
        self.num_groups = num_channels
        self.num_channels = num_channels
        self.eps = eps
        self.affine = affine
        self.register_parameter('weight', None)
        self.register_parameter('bias', None)        

    def forward(self, input: Tensor, condition) -&gt; Tensor:
        weight = condition[:, :, 0]
        bias = condition[:, :, 1]
        return F.group_norm(
            input, self.num_groups, weight, bias, self.eps)

    def extra_repr(self) -&gt; str:
        return '{num_groups}, {num_channels}, eps={eps}, ' \
            'affine={affine}'.format(**self.__dict__)
</code></pre>
<p>here is my question:<br>
whether weight. and bias calculated for each sample in a batch independently when as a condition I pass a tensor of <code>[bs, num_channels]</code> instead of<code> [num_channels]</code> or not. I tried to follow the source code until <a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/torch/onnx/symbolic_opset9.py#L2837" rel="noopener nofollow ugc">here</a> but can not understand it completely.</p>

ptrblck: <p>The parameters usually do not depend on the batch size, so you should make sure they have the shape <code>[num_channels]</code>:</p>
<pre><code class="lang-python">norm = nn.GroupNorm(num_groups=3, num_channels=9)
print(norm.weight.shape)
&gt; torch.Size([9])
print(norm.bias.shape)
&gt; torch.Size([9])
</code></pre>
<p>Also, note that you are using <code>self.weight</code> and <code>self.bias</code> in your code, which are both set to <code>None</code>, so you might want to change it.</p>

nima_rafiee: <p>Thanks for your reply.  yes, they should be independent but as I’m implementing conditional instance norm, the <code>weight</code> and <code>bias</code> for each sample should be calculated from another sample as the condition.  that is why I want to know if the <code>F.group_norm()</code>  can handle weight and bias with size <code>[bs, num_channels].</code>  I think  I can update the result of  <code>F.group_norm(bias=None, weight=None)</code> with  condition instead of passing as an argument to <code>F.group_norm()</code></p>
<pre><code class="lang-auto">class ConditionalInstanceNorm(Module):
    
    __constants__ = ['num_groups', 'num_channels', 'eps', 'affine']
    num_groups: int
    num_channels: int
    eps: float
    affine: bool

    def __init__(self, num_channels: int, eps: float = 1e-5, affine: bool = True) -&gt; None:
        super(ConditionalInstanceNorm, self).__init__()
        self.num_groups = num_channels
        self.num_channels = num_channels
        self.eps = eps
        self.affine = affine
        self.register_parameter('weight', None)
        self.register_parameter('bias', None)        

    def forward(self, input: Tensor, condition) -&gt; Tensor:
        weight = condition[:, :, 0]
        bias = condition[:, :, 1]
        unnormalized =  F.group_norm(
            input, self.num_groups,  self.weight,  self.bias,  self.eps)
        weight = weight.unsqueeze(-1).unsqueeze(-1)
        bias = bias.unsqueeze(-1).unsqueeze(-1)
        normalized = (unnormalized*weight)  +  bias 
        return normalized 

    def extra_repr(self) -&gt; str:
        return '{num_groups}, {num_channels}, eps={eps}, ' \
            'affine={affine}'.format(**self.__dict__)
</code></pre>