Batch Whitening (BW) instead of Batch Normalization,

username: zaki
cooked: <p>Hi<br>
I’m new to PyTorch and I want to test batch whitening (BW) as defined in  <a href="https://github.com/roysubhankar/dwt-domain-adaptation" rel="nofollow noopener">https://github.com/roysubhankar/dwt-domain-adaptation</a>, to train a model from scratch to do some domain adaptation and  I don’t know how to define my BW layer, how I get the running_mean and running_variance?</p>
<pre><code class="lang-auto">class whitening_scale_shift(nn.Module):
	def __init__(self, planes, group_size, running_mean, running_variance, track_running_stats=True, affine=True):
		super(whitening_scale_shift, self).__init__()
		self.planes = planes
		self.group_size = group_size
		self.track_running_stats = track_running_stats
		self.affine = affine
		self.running_mean = running_mean
		self.running_variance = running_variance

		self.wh = whitening.WTransform2d(self.planes, self.group_size, 
								 running_m=self.running_mean, 
							 running_var=self.running_variance, 
							track_running_stats=self.track_running_stats)
		if self.affine:
			self.gamma = nn.Parameter(torch.ones(self.planes, 1, 1))
			self.beta = nn.Parameter(torch.zeros(self.planes, 1, 1))

	def forward(self, x):
		out = self.wh(x)
		if self.affine:
			out = out * self.gamma + self.beta
return out
</code></pre>
<p>Could someone help me</p>

username: tom
cooked: <p>If you pass in None, it will instantiate the right thing:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/roysubhankar/dwt-domain-adaptation/blob/master/utils/whitening.py#L19" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/roysubhankar/dwt-domain-adaptation/blob/master/utils/whitening.py#L19" target="_blank" rel="nofollow noopener">roysubhankar/dwt-domain-adaptation/blob/master/utils/whitening.py#L19</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="9" style="counter-reset: li-counter 8 ;">
<li>		self.num_features = num_features</li>
<li>		self.momentum = momentum</li>
<li>		self.track_running_stats = track_running_stats</li>
<li>		self.eps = eps</li>
<li>		self.alpha = alpha</li>
<li>		self.group_size = min(self.num_features, group_size)</li>
<li>		self.num_groups = self.num_features // self.group_size</li>
<li>		self.running_m = running_m</li>
<li>		self.running_var = running_var</li>
<li>
</li>
<li class="selected">		if self.track_running_stats and self.running_m is not None:</li>
<li>			self.register_buffer('running_mean', self.running_m)</li>
<li>			self.register_buffer('running_variance', self.running_var)</li>
<li>		else:</li>
<li>			self.register_buffer('running_mean', torch.zeros([1, self.num_features, 1, 1], out=torch.cuda.FloatTensor() if torch.cuda.is_available() else torch.FloatTensor()))</li>
<li>			self.register_buffer('running_variance', torch.ones([self.num_groups, self.group_size, self.group_size], out=torch.cuda.FloatTensor() if torch.cuda.is_available() else torch.FloatTensor()))</li>
<li>		"""</li>
<li>		if self.track_running_stats:</li>
<li>			self.register_buffer('running_mean', torch.zeros([1, self.num_features, 1, 1], out=torch.cuda.FloatTensor() if torch.cuda.is_available() else torch.FloatTensor()))</li>
<li>			self.register_buffer('running_variance', torch.zeros([self.num_groups, self.group_size, self.group_size], out=torch.cuda.FloatTensor() if torch.cuda.is_available() else torch.FloatTensor()))</li>
<li>		"""</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I’m not quite sure whether the authors of the code want to achieve something specific or whether the interface of optionally passing in the tensors is only accidental (the initialization seems very unidiomatic, too) – in theory that should just go through regular state dict loading and saving.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: Use triple backticks  ``` before and after your code to have better formatting.</p>

username: zaki
cooked: <p>thank you <a class="mention" href="/u/tom">@tom</a> I did like you said and it works</p>