Dynamically update input mask

Marco_Repetto: <p>I’m implementing a binary mask at the input level of a logistic regression. This is how I defined it:</p>
<pre><code class="lang-auto">new_mask = torch.ones(1, input_dim)

class DropFeatures(nn.Module):
    def __init__(self, input_dim, *kargs, **kwargs):
        super(DropFeatures, self).__init__()
        self.weights = new_mask
        
    def forward(self, x):
        return (x * self.weights)
    
class drop_logit(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(drop_logit, self).__init__()
        self.drop = DropFeatures(input_dim)
        self.fc1 = nn.Linear(input_dim, 1)
        self.fc2 = nn.Sigmoid()
          
    def forward(self, x):
        x = self.drop(x)
        x = self.fc1(x)
        x = self.fc2(x)
        return x
    
model = drop_logit(input_dim = 69, output_dim = 1)
</code></pre>
<p>Now I would like to be able to modify the <code>new_mask</code> dynamically inside the training loop to make the model more parsimonious overtime.</p>
<p>But it is not clear to me how I could achieve that.</p>

ptrblck: <p>Based on your code snippet it seems <code>self.weights</code> is an <code>int</code> and will not be trained, so I’m unsure how the masking is performed as it seems you are scaling the activation instead.</p>
<aside class="quote no-group" data-username="Marco_Repetto" data-post="1" data-topic="132404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marco_repetto/48/42346_2.png" class="avatar"> Marco_Repetto:</div>
<blockquote>
<p>Now I would like to be able to modify the <code>new_mask</code> dynamically</p>
</blockquote>
</aside>
<p>If you would like to train the <code>self.weights</code> parameter, create it as an <code>nn.Parameter</code> and pass it to the optimizer.</p>

Marco_Repetto: <p>Ok, so in this configuration, the mask is applied only on evaluation? Because when I do model(X_test) the mask seems to be applied.</p>
<p>I tried what you suggested, namely:</p>
<pre><code class="lang-auto">class DropFeatures(nn.Module):
    def __init__(self, input_dim, *kargs, **kwargs):
        super(DropFeatures, self).__init__()
        self.weights = nn.Parameter(new_mask)
        
    def forward(self, x):
        return (x * self.weights)
</code></pre>
<p>The problem is that if I train the model setting <code>new_mask = torch.zeros(1, 69)</code> the loss is actually decreasing. Instead should not as all the features are zeroed, correct me if I am wrong.</p>

ptrblck: <p>The new code snippet isn’t consistent with the previous one, as you were previously using an <code>int</code> and not a mask.</p>
<pre data-code-wrap="python"><code class="lang-python">        self.drop = DropFeatures(input_dim)
        self.fc1 = nn.Linear(input_dim, 1)
</code></pre>
<p>This indicates that <code>input_dim</code> is an <code>int</code>, not a tensor.</p>
<aside class="quote no-group" data-username="Marco_Repetto" data-post="3" data-topic="132404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marco_repetto/48/42346_2.png" class="avatar"> Marco_Repetto:</div>
<blockquote>
<p>so in this configuration, the mask is applied only on evaluation?</p>
</blockquote>
</aside>
<p>No, since you are not checking the <code>self.training</code> attribute to switch between training and evaluation mode.</p>
<aside class="quote no-group" data-username="Marco_Repetto" data-post="3" data-topic="132404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marco_repetto/48/42346_2.png" class="avatar"> Marco_Repetto:</div>
<blockquote>
<p>The problem is that if I train the model setting <code>new_mask = torch.zeros(1, 69)</code> the loss is actually decreasing.</p>
</blockquote>
</aside>
<p>I don’t know where you are setting this, but yes, if you are using a mask with all-zeros, the output will also be all zero.</p>

Marco_Repetto: <p>I just took <code>new_mask</code> outside to change it at convenience the original code was:</p>
<pre><code class="lang-auto">class DropFeatures(nn.Module):
    def __init__(self, input_dim, *kargs, **kwargs):
        super(DropFeatures, self).__init__()
        self.weights = torch.zeros(1, input_dim)
        
    def forward(self, x):
        return (x * self.weights)
    
class drop_logit(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(drop_logit, self).__init__()
        self.drop = DropFeatures(input_dim)
        self.fc1 = nn.Linear(input_dim, 1)
        self.fc2 = nn.Sigmoid()
          
    def forward(self, x):
        x = self.drop(x)
        x = self.fc1(x)
        x = self.fc2(x)
        return x
    
model = drop_logit(input_dim = 69, output_dim = 1)
</code></pre>
<p>I do not see why you are telling me that <code>self.weights</code> is an int when if I print it is a tensor. Of course in this example all features are zeroed so I expect the loss to not decrease.</p>