How to dropout with non zero value?

hadaev8: <p>I want to use feature dropout like dropout2d and fill it with mean value (or Gaussian noise for example) instead of zeros.<br>
How to do so?<br>
Easiest thing to do is runing dropout2d and fill zeros, but i have zeros in data.</p>

ptrblck: <p>You could sample a binary mask using the drop probability <code>p</code> and fill the tensor with your desired value:</p>
<pre><code class="lang-python">x = torch.ones(10, 20)

p = 0.5
mask = torch.distributions.Bernoulli(probs=(1-p)).sample(x.size())
x[~mask.bool()] = x.mean()
</code></pre>
<p>Note that dropout scales also the activation to make sure the expected activation ranges are equal during training and validation via:</p>
<pre><code class="lang-python"># manual dropout
out = x * mask * 1/(1-p)
</code></pre>
<p>So you would have to take care of this scaling for your approach.</p>

hadaev8: <p>Can i sample values from tensor for mask values?<br>
Also do i really need to scale, if dropping out with non zero values?</p>

ptrblck: <p>I’m not sure I understand the first question.<br>
In my code I sample the mask manually. Would that work of what would you like to sample additionally?</p>
<p>If you enable dropout during training and disable it during evaluation, the expected activation values will have a different range and thus your model will most likely perform poorly.<br>
The scaling is described in the <a href="http://jmlr.org/papers/v15/srivastava14a.html">original Dropout paper</a>.</p>

hadaev8: <p>I want to calculate mean frame with taking into account zero padding and replace some random frames with it.<br>
This code raises an error.<br>
What is the right way?</p>
<pre><code>def mean_frame_dropout(x, lens, p=0.2):
    x = x.clone()
    batch_size, length, features = x.size()
    mask = torch.distributions.Bernoulli(
        probs=(1 - p)).sample((batch_size, length)).to(x.device)
    mask = ~mask.bool()
    mask = mask.unsqueeze(-1).expand(-1, -1, features)
    mean = x.detach().sum(1) / lens
    x[mask] = mean
    return x</code></pre>

ptrblck: <p>What kind of error are you getting? Could you post the stack trace as well as the shapes of the input tensors?</p>

hadaev8: <p>Mean and x sizes<br>
torch.Size([1, 128])<br>
torch.Size([1, 167, 128])</p>
<p>RuntimeError: shape mismatch: value tensor of shape [128] cannot be broadcast to indexing result of shape [5120]</p>

ptrblck: <p>The <code>mean</code> tensor has the shape <code>[1, 128]</code> as it’s calculated by <code>x.sum(1)</code>.<br>
However, <code>x[mask]</code> has a variable number of elements so how should the assignment work?<br>
E.g. lets assume <code>mask</code> samples 5120 <code>True</code> values, thus <code>x[mask]</code> would have the shape <code>[5120]</code>.<br>
How should the 128 values of <code>mean</code> be assigned to which value?</p>

hadaev8: <p>What is the right way to mask it?</p>

ptrblck: <p>It depends, what you want to achieve.<br>
The original <code>mask</code> before the <code>unsqueeze</code> and <code>expand</code> operations can be used to index <code>x</code> directly, which would yield:</p>
<pre><code class="lang-python">x = torch.ones([1, 167, 128])
batch_size, length, features = x.size()
p = 0.5
mask = torch.distributions.Bernoulli(
    probs=(1 - p)).sample((batch_size, length)).to(x.device)
mask = ~mask.bool()
print(mask.shape)
&gt; torch.Size([1, 167])
print(x[mask].shape)
&gt; torch.Size([81, 128])
print(mask.sum())
&gt; tensor(81)
</code></pre>
<p>So would you like to replace all these <code>81</code> values in <code>dim1</code> with the sum of them?</p>

hadaev8: <p>Should you advice, how to replace some block of channels with mean values?<br>
And perform it independly for each sample in batch.<br>
Im not sure how to select indexes over batch and len axis at the same time.</p>
<p>For x = torch.rand(32, 1024, 128)</p>
<pre><code>def mean_freq_mask(x, p=0.2):
	if torch.rand(1) &lt; p:
		x = x.clone()
		batch_size, length, features = x.size()
		F = features // 3
		mean = x.detach().mean(2)
		mask_len = (torch.rand(batch_size)*F).long()
		mask_start = (torch.rand(batch_size)*(features-F)).long()
		x[mask_start:mask_start+mask_len] = mean.unsqueeze(-1).expand(-1, -1, features)[mask_start:mask_start+mask_len]
	return x
</code></pre>
<p>This gives error only integer tensors of a single element can be converted to an index</p>

Torcione: <p>What is the correct way to use this block inside a model class?<br>
Let’s say I have the following CNN:</p>
<pre><code class="lang-auto">cfg = {

    'VGG16': [64, 'Dp', 64, 'M', 128, 'Dp', 128, 'M', 256, 'Dp', 256, 'Dp', 256, 'M', 512,'Dp', 512,'Dp', 512, 'M', 512,'Dp', 512,'Dp', 512, 'A', 'Dp'], #dropouts dependent from a single parameter (useful for hyper-par optim.) 
    
}


class VGG(nn.Module, NetVariables, OrthoInit):
    def __init__(self, params):
        

        self.params = params.copy()
        

        nn.Module.__init__(self)
        NetVariables.__init__(self, self.params)
        OrthoInit.__init__(self)
        
     
        self.features = self._make_layers(cfg['VGG16'])
        self.classifier = nn.Linear(512, self.num_classes)



        self.weights_init() #call the orthogonal initial condition
        


    def forward(self, x):
        outs = {} 
        L2 = self.features(x)
        outs['l2'] = L2
        Out = L2.view(L2.size(0), -1)
        Out = self.classifier(Out)
        outs['out'] = Out
        return outs

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
                
            elif x=='A':
                layers += [nn.AvgPool2d(kernel_size=2, stride=2)]
                
            elif x == 'D3':
                layers += [nn.Dropout(0.3)]

            elif x == 'D4':
                layers += [nn.Dropout(0.4)]            

            elif x == 'D5':
                layers += [nn.Dropout(0.5)]   
                
            elif x == 'Dp':
                layers += [nn.Dropout(self.params['dropout_p'])] 
                
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),

                           nn.Tanh()
                           ,nn.GroupNorm(int(x/self.params['group_factor']), x)
                           ]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers) 

</code></pre>
<p>I’d like to substitute the dropout layer with the block you proposed, giving each time I forward the mask as input; how should I do it?</p>

ptrblck: <p>If you want to pass the <code>mask</code> as an additional argument to the <code>forward</code> method of your custom <code>Dropout</code> layer, you should write this logic into the <code>forward</code> method of <code>VGG</code> explicitly and remove the usage of the <code>nn.Sequential</code> block as it expects a single input/output in the default implementation.</p>

Torcione: <p>Is there some other way around to keep the code more flexible and clean; in the above solution with <code>nn.Sequential</code> I can easly modify the architecture ‘chain’ of modules directly from <code>cfg</code> dict.<br>
Can I do something similar without <code>nn.Sequential</code>?</p>

ptrblck: <p>You could try to use an <code>nn.ModuleList</code> or <code>nn.ModuleDict</code>, add the layers using your <code>cfg</code>, and iterate it in the <code>forward</code> method.</p>