Reproduce same network architecture (with dropout active) in multiple steps: zeroes multiple times the same elements with dropout

Torcione: <p>Hi everyone,<br>
I have a network architecture with <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html" rel="noopener nofollow ugc">Dropout</a> layers.<br>
From the above official reference:</p>
<blockquote>
<p>During training, randomly zeroes some of the elements of the input tensor with probability <code>p</code> using samples from a Bernoulli distribution.</p>
</blockquote>
<p>I need to fix the same configuration for multiples consecutive N steps; when I say <em>fix the same configuration</em> I mean:</p>
<ul>
<li>at step 0, randomly zeroes some of the elements of the input tensor with probability <code>p</code> using samples from a Bernoulli distribution.</li>
<li>In the following N steps, zeroes the same input’s elements chosen in step 0 (so the network seen by the system for each of these steps is exactly the same)</li>
<li>At step N+1 we restart from the beginning (randomly zeroes some of the elements of the input tensor with probability <code>p</code> using samples from a Bernoulli distribution.)</li>
<li>repeat</li>
</ul>
<p>How can I do this?</p>
<p><strong>[EDIT]</strong></p>
<p>There are some example of problem similar to the above (<a href="https://discuss.pytorch.org/t/custom-mask-for-dropout/66985">example_1</a>, <a href="https://discuss.pytorch.org/t/how-to-fix-the-dropout-mask-for-different-batch/7119">Example_2</a>, <a href="https://discuss.pytorch.org/t/how-to-dropout-with-non-zero-value/79561">Example_3</a>).<br>
Anyhow I still have some doubt about how to proceed:</p>
<ul>
<li>In my network I have something like this:</li>
</ul>
<pre><code class="lang-auto">cfg = {

    'VGG16': [64, 'Dp', 64, 'M', 128, 'Dp', 128, 'M', 256, 'Dp', 256, 'Dp', 256, 'M', 512,'Dp', 512,'Dp', 512, 'M', 512,'Dp', 512,'Dp', 512, 'A', 'Dp'], #dropouts dependent from a single parameter (useful for hyper-par optim.) 
    
}


class VGG(nn.Module, NetVariables, OrthoInit):
    def __init__(self, params):
        

        self.params = params.copy()
        

        nn.Module.__init__(self)
        NetVariables.__init__(self, self.params)
        OrthoInit.__init__(self)
        
     
        self.features = self._make_layers(cfg['VGG16'])
        self.classifier = nn.Linear(512, self.num_classes)



        self.weights_init() #call the orthogonal initial condition
        


    def forward(self, x):
        outs = {} 
        L2 = self.features(x)
        outs['l2'] = L2
        Out = L2.view(L2.size(0), -1)
        Out = self.classifier(Out)
        outs['out'] = Out
        return outs

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
                
            elif x=='A':
                layers += [nn.AvgPool2d(kernel_size=2, stride=2)]
                
            elif x == 'D3':
                layers += [nn.Dropout(0.3)]

            elif x == 'D4':
                layers += [nn.Dropout(0.4)]            

            elif x == 'D5':
                layers += [nn.Dropout(0.5)]   
                
            elif x == 'Dp':
                layers += [nn.Dropout(self.params['dropout_p'])] 
                
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),

                           nn.Tanh()
                           ,nn.GroupNorm(int(x/self.params['group_factor']), x)
                           ]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers) 

</code></pre>
<p>How can I adapt the code in such a way:</p>
<ol>
<li>
<p>Substitute Dropout in <code>nn.Sequential</code> by a layer that is <code>nn.Identity</code> when <code>model.training()==False</code> (evaluation mode) and is a mask layer when <code>model.training()==False</code> (training mode)</p>
</li>
<li>
<p>the mask mentioned in the point 1. keeps constant and changes only when triggered by a given external input flag</p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>Either use the proposed approach from <a href="https://discuss.pytorch.org/t/how-to-dropout-with-non-zero-value/79561/13">your cross post</a> or initialize the layers first and use a switch in the custom <code>forward</code> method.</p>
</li>
<li>
<p>I would write a custom dropout layer, which accepts an additional flag to resample the mask.</p>
</li>
</ol>

Torcione: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> ! your advices were very helpful.<br>
I report here the explicit code to make it available for users who will need it.<br>
If anyone notice something that seems wrong please don’t esitate to report it.</p>
<pre><code class="lang-auto">class VGG_Custom_Dropout(nn.Module, NetVariables, OrthoInit):
    def __init__(self, params):
      
        self.params = params.copy()
        
        nn.Module.__init__(self)
        NetVariables.__init__(self, self.params)
        OrthoInit.__init__(self)
        
        
        self.ModuleDict = self._make_layers(cfg['VGG16']) #CREATE A DICT OF MODULES WITH IDENTITY ON DROPOUT PLACE
        self.classifier = nn.Linear(512, self.num_classes)
        self.mask_dict = {}


        self.weights_init() #call the orthogonal initial condition
        


    def forward(self, x, Mask_Flag, TrainMode_Flag):
        outs = {} 
        #WE ITERATE OVER THE MODULE DICT SUBSTITUTING IDENTITY WITH THE DROPOUT MASK WHEN THE TrainingMode_Flag TELL US WE ARE IN TRAINING MODE
        
        for key in self.ModuleDict:
            if key.startswith('!'):
                
                if not TrainMode_Flag: #if we are in eval mode the dropout is substitute by a identity layer
                    x = self.ModuleDict[key](x)
                else:
                
                
                    if Mask_Flag==1: #this flag trigger the update of masks 
                        self.mask_dict[key] = torch.distributions.Bernoulli(probs=(1-self.params['dropout_p'])).sample(x.size())

                    x = x * self.mask_dict[key] * 1/(1-self.params['dropout_p']) #dropout layer

                    
            else: #for modules different from dropout regular forward
                x = self.ModuleDict[key](x)
        
        L2 = x
        outs['l2'] = L2
        Out = L2.view(L2.size(0), -1)
        Out = self.classifier(Out)
        outs['out'] = Out
        return outs

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        ModuleDict = nn.ModuleDict()
        NumberKey=0
        
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
                
            elif x=='A':
                layers += [nn.AvgPool2d(kernel_size=2, stride=2)]
                 
                
            elif x == 'Dp':
                
                ModuleDict[str(NumberKey)] = nn.Sequential(*copy.deepcopy(layers)) 
                NumberKey+=1
                ModuleDict['!'+str(NumberKey)] = nn.Identity()
                NumberKey+=1
                layers = []
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           #nn.ReLU(inplace=True),
                           #nn.LeakyReLU(negative_slope=0.1, inplace=False),
                           nn.Tanh()
                           #,nn.BatchNorm2d(x)   #For now Batch Norm is excluded because it is incompatible with PCNGD, GD, PCNSGD where I forward sample by sample
                           ,nn.GroupNorm(int(x/self.params['group_factor']), x)
                           #,nn.GroupNorm(int(1), x)
                           ]
                in_channels = x
        NumberKey+=1
        ModuleDict[str(NumberKey)] = nn.AvgPool2d(kernel_size=1, stride=1)
        return ModuleDict
</code></pre>
<p>In this way I have control on where the mask has to change and when it has to be substitued by an identity layer (when we are in the evaluation mode). Do I miss something? For example should I detach the mask multiplication?<br>
I think it should not make difference because also if the gradient record that operation I’m turning off the nodes, so no signal leave from them. Is it right?</p>

ptrblck: <p>You could replace <code>TrainMode_Flag</code> with <code>self.training</code>, as the latter flag is changed by calling <code>model.train()</code> and <code>model.eval()</code>. Besides that I don’t see any obvious issues.</p>