Doing a very basic (simple) moving average?

username: pinocchio
cooked: <p>I was trying to do a moving average but was worried that it would negatively interfere with my backprop or something weird (sorry new to pytorch. If I do:</p>
<pre><code>W = Variable(w_init, requires_grad=True)
W_avg = Variable(torch.FloatTensor(W).type(dtype), requires_grad=False)
for i in range(nb_iterations):
    #some GD stuff...
    W_avg = (1/nb_iter)*W + W_avg
</code></pre>
<p>would that be ok? Would it compute my average parameters and not interfere with other stuff?</p>

username: pinocchio
cooked: <p>weird got this error:</p>
<pre><code>W_avg = Variable(torch.FloatTensor(W).type(dtype), requires_grad=False)
RuntimeError: already counted a million dimensions in a given sequence. Most likely your items are also sequences and there's no way to infer how many dimension should the tensor have</code></pre>

username: colesbury
cooked: <p>The error is because you’re trying to create a <code>FloatTensor</code> from a <code>Variable</code>:</p>
<pre><code class="lang-auto">torch.FloatTensor(W)  # W is of type 'Variable'
</code></pre>
<p>Here’s a way to do a average (assuming you don’t care about back-propagating through the average):</p>
<pre><code class="lang-auto">W = Variable(w_init, requires_grad=True)
W_avg = torch.zeros(W.size()).type(dtype)  # Tensor, not Variable since you don't care about gradients
for i in range(nb_iterations):
    #some GD stuff...
    W_avg += (1/nb_iter)*W.data
</code></pre>
<p>Note the <code>W.data</code>. This extracts the Tensor from the Variable wrapper, since you don’t care about back-propagating through the average.</p>

username: PiotrSokol
cooked: <p><a class="mention" href="/u/colesbury">@colesbury</a></p>
<p>How would I go about using this to write a mean-only batch normalization module?</p>

username: smth
cooked: <p><code>out = x - torch.mean(x, dim=0)</code></p>

username: PiotrSokol
cooked: <p><a class="mention" href="/u/smth">@smth</a></p>
<p>Admittedly, still not getting it. I  tried creating a mean only bn module</p>
<pre><code>def __init__(self, num_features,momentum=0.1):
    super(_MeanOnlyBatchNorm, self).__init__()
    self.num_features = num_features
    self.momentum = momentum
    self.register_buffer('running_mean', torch.zeros(num_features))
    self.reset_parameters()

def forward(self, input):
    mu = torch.mean(input,dim=0).data
    self.running_mean = self.momentum*mu + (1-self.momentum) * self.running_mean
    return torch.add(input, -self.running_mean)
</code></pre>
<p>but get errors. I’m lost.</p>
<p>EDIT:</p>
<p>I’ve settled on this, but still am not sure if it’s correct.</p>
<pre><code>def forward(self, input):

    # mu = Variable(torch.mean(input,dim=0, keepdim=True).data, requires_grad=False)
    if self.training is True:
        mu = input.mean(dim=0, keepdim=True)
        mu = self.momentum * mu + (1 - self.momentum) * Variable(self.running_mean)

        self.running_mean = mu.data
        return input.add_(-mu)
    else:
        return  input.add_(-Variable(self.running_mean))</code></pre>

username: J_Johnson
cooked: <p>The fastest and most efficient way is to make use of the AvgPool1d module. Then you get the advantage of parallelization and can also run the calc on GPU.</p>
<p>You can specify the period of the simple moving average via the kernel.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.AvgPool1d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.AvgPool1d.html</a></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

list = torch.arange(0,10,1,dtype=torch.float).view(1,1,-1)
kernel = 3
sma = nn.AvgPool1d(kernel_size=kernel, stride = 1)
out = sma(list)
</code></pre>
<p>Of course, you’ll have to figure out how you want the ends handled. You could just repeat the first and final value and torch.cat them so that your output is the same size as the input.</p>