How to add customized l1/l2 penalty on parameter slice?

username: fly2mars
cooked: <p>I’ve defined a submodule which contains parameters I want to penalize. To be specific, I want to penalize the 1st dim of this parameter only!</p>
<p>Here is a demo of my implementation. it works on cpu, while throws RuntimeError on ‘cuda’.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import numpy as np

class mySubModule(torch.nn.Module):
    def __init__(self, n):
        super().__init__()
        self.my_param = torch.nn.Parameter(
            torch.empty(n, 2).uniform_(0.0, 1.0), requires_grad=True)
        self.register_buffer('mask_choice', torch.tensor([[1.], [0.]]))

    def forward(self, x):
        out = torch.matmul(
            torch.matmul(x, self.my_param),
            self.mask_choice)
        return out

class myModule(torch.nn.Module):
    def __init__(self, n):
        super().__init__()
        self.subModule = mySubModule(n)
        self.linear = torch.nn.Linear(n, 1)

        self.param_need_l1_penalty_case_1 = [self.subModule.my_param]
        #### Here's where Error happens
        self.param_need_l1_penalty_case_2 = [self.subModule.my_param[:,0]]
        ####

    def forward(self, x):
        return self.linear(x) + self.subModule(x)
</code></pre>
<p>a tipical useage during training looks like</p>
<pre data-code-wrap="python"><code class="lang-python"># demo of trainer
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
np.random.seed(999)

model = myModule(n=2)
model.to(device)

my_data = torch.from_numpy(np.random.random((4,2))).to(torch.float).to(device)
label = torch.from_numpy(np.ones((4,))).to(torch.float).to(device)

pred = model(my_data).squeeze(1)
criteria = torch.nn.MSELoss()
loss = criteria(label, pred)
# add penalty
for param in model.param_need_l1_penalty_case_2:
    loss += 0.1 * torch.norm(param, 1)
print(loss)
loss.backward()
</code></pre>
<p>When cuda is available, it raises error saying</p>
<blockquote>
<p>RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.cuda.FloatTensor</p>
</blockquote>
<p>Any suggestion on how to add l1/l2 penalization in this scenario?</p>
<p>Appreciated~</p>

username: fly2mars
cooked: <p>I’ve found a solution to avoid it, making <code>param_need_l1_penalty_case_2</code> a property of my class.</p>
<pre><code class="lang-python">    @property
    def param_need_l1_penalty_case_2(self):
        return [self.subModule.my_param[:,0]]
</code></pre>
<p>Then this is called after <code>mode.to(device)</code>.<br>
and the trainer code do not need modification.</p>

username: KFrank
cooked: <p>Hi Fly!</p>
<p>First, I’m glad you found a solution.</p>
<p>Second, I’m very confused by all of this, and have some comments,<br>
below.</p>
<aside class="quote no-group quote-modified" data-username="fly2mars" data-post="1" data-topic="62504" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fly2mars/48/17160_2.png" class="avatar"> fly2mars:</div>
<blockquote>
<p>Here is a demo of my implementation. it works on cpu, while throws RuntimeError on ‘cuda’.</p>
<pre data-code-wrap="python"><code class="lang-python">...
class myModule(torch.nn.Module):
    def __init__(self, n):
        ...
        self.param_need_l1_penalty_case_1 = [self.subModule.my_param]
        #### Here's where Error happens
        self.param_need_l1_penalty_case_2 = [self.subModule.my_param[:,0]]
        ####
...
</code></pre>
<p>a tipical useage during training looks like</p>
<pre data-code-wrap="python"><code class="lang-python"># demo of trainer
...
loss = criteria(label, pred)
# add penalty
for param in model.param_need_l1_penalty_case_2:
    loss += 0.1 * torch.norm(param, 1)
print(loss)
loss.backward()
</code></pre>
<p>When cuda is available, it raises error saying</p>
<blockquote>
<p>RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.cuda.FloatTensor</p>
</blockquote>
</blockquote>
</aside>
<p>I have been able to reproduce and “fix” your issue, but I don’t really<br>
understand it.</p>
<p>Also, a  disclaimer:  My particular pytorch-gpu setup requires the<br>
old pytorch version 0.3.0.  So part or all of my analysis might be<br>
a red herring, and specific to 0.3.0.</p>
<p>First, for reasons I don’t understand, I had to explicitly move the<br>
parameters in <code>param_need_l1_penalty_case_2</code> to the gpu<br>
(but not those in <code>param_need_l1_penalty_case_1</code>).</p>
<p>Second, when running on the gpu, I had to convert the “penalty loss”<br>
to a python scalar before adding it to <code>loss</code> in order to get rid of your<br>
specific error:</p>
<pre data-code-wrap="python"><code class="lang-python">    if  scalarPenalty:
        penalty = 0.1 * torch.norm (param, 1).data[0]
        loss += penalty
    else:            
        loss += 0.1 * torch.norm (param, 1)
</code></pre>
<p>Here is a complete, runnable pytorch-version-0.3.0 test program,<br>
modelled after yours:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

torch.manual_seed (0)

gpu = True
print  ('gpu = ' + str (gpu))

scalarPenalty = True
print  ('scalarPenalty = ' + str (scalarPenalty))

class mySubModule(torch.nn.Module):
    def __init__(self, n):
        super().__init__()
        self.my_param = torch.nn.Parameter (torch.rand (n, 2), requires_grad=True)
        self.register_buffer('mask_choice', torch.autograd.Variable (torch.Tensor([[1.], [0.]])))

    def forward(self, x):
        out = torch.matmul(
            torch.matmul(x, self.my_param),
            self.mask_choice)
        return out

class myModule (torch.nn.Module):
    def __init__(self, n):
        super().__init__()
        self.subModule = mySubModule(n)
        self.linear = torch.nn.Linear(n, 1)

        self.param_need_l1_penalty_case_1 = [self.subModule.my_param]
        #### Here's where Error happens
        self.param_need_l1_penalty_case_2 = [self.subModule.my_param[:,0]]
        ####

    def forward(self, x):
        return self.linear(x) + self.subModule(x)

model = myModule(n=2)

my_data = torch.autograd.Variable (torch.ones (4,2))
label = torch.autograd.Variable (torch.ones (4), requires_grad = False)

if  gpu:
    model.cuda()
    my_data = my_data.cuda()
    label = label.cuda()
    for  i in range (len (model.param_need_l1_penalty_case_2)):
        model.param_need_l1_penalty_case_2[i] = model.param_need_l1_penalty_case_2[i].cuda()


pred = model (my_data).squeeze(1)
criteria = torch.nn.MSELoss()
# loss = criteria (label, pred)
loss = criteria (pred, label)  # probably an 0.3.0 requirement

print ('loss (pre-penalty) = ...\n', loss)
for param in model.param_need_l1_penalty_case_2:
    if  scalarPenalty:
        penalty = 0.1 * torch.norm (param, 1).data[0]
        loss += penalty
    else:            
        loss += 0.1 * torch.norm (param, 1)
print ('loss (post-penalty) = ...\n', loss)

print ('calling loss.backward()...')
loss.backward()
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">0.3.0b0+591e73e
gpu = True
scalarPenalty = True
loss (pre-penalty) = ...
 Variable containing:
 0.2621
[torch.cuda.FloatTensor of size 1 (GPU 0)]

loss (post-penalty) = ...
 Variable containing:
 0.3206
[torch.cuda.FloatTensor of size 1 (GPU 0)]

calling loss.backward()...
</code></pre>
<p>When I turn off the scalar-penalty modification:</p>
<pre data-code-wrap="python"><code class="lang-python">scalarPenalty = False
</code></pre>
<p>I get what appears to be your error:</p>
<pre><code class="lang-plaintext">0.3.0b0+591e73e
gpu = True
scalarPenalty = False
loss (pre-penalty) = ...
 Variable containing:
 0.2621
[torch.cuda.FloatTensor of size 1 (GPU 0)]

loss (post-penalty = ...
 Variable containing:
 0.3206
[torch.cuda.FloatTensor of size 1 (GPU 0)]

calling loss.backward()...
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;string&gt;", line 66, in &lt;module&gt;
  File "C:\Users\LisaBrown\Documents\admin\programs\Miniconda3\lib\site-packages\torch\autograd\variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "C:\Users\LisaBrown\Documents\admin\programs\Miniconda3\lib\site-packages\torch\autograd\__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #3 'other'
</code></pre>
<p>(<code>line 66, in &lt;module&gt;</code> is <code>loss.backward()</code>.)</p>
<p>So, something fishy is definitely going on here.  (Whether pytorch 0.3.0<br>
and pytorch 1.x.x share the same fishiness, I don’t know.)</p>
<p>As practical matter, it sounds like you have your program working.<br>
But, for the greater good, it would be nice to see if my results are<br>
reproducible on an up-to-date version of pytorch.</p>
<p>And if any experts have an idea of what is going on under the hood,<br>
please chime in.</p>
<p>Best.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="62504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>First, for reasons I don’t understand, I had to explicitly move the<br>
parameters in <code>param_need_l1_penalty_case_2</code> to the gpu<br>
(but not those in <code>param_need_l1_penalty_case_1</code> ).</p>
</blockquote>
</aside>
<p>Take this “debugging” with a grain of salt, but that would be my best guess:<br>
<code>param_need_l1_penalty_case_1</code> was defined as an <code>nn.Parameter</code> and just wrapped in a list.<br>
Iterating this list will yield these parameters, which were properly pushed to the device by calling <code>model.to('cuda')</code>, since they were also properly registered inside the module.<br>
However, an operation on <code>param_need_l1_penalty_case_2</code> was executed before storing them in the list (the slicing op). The result of this operation will be a <code>tensor</code> with a <code>grad_fn</code> (<code>SelectBackward</code> in this case).<br>
Since this operation was performed inside the <code>__init__</code> function of the module, and thus before pushing the parameters to the device, the slice will stay on CPU and you would need to push it manually to the device afterwards.</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="62504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Second, when running on the gpu, I had to convert the “penalty loss”<br>
to a python scalar before adding it to <code>loss</code> in order to get rid of your<br>
specific error</p>
</blockquote>
</aside>
<p>I doubt this will yield the desired behavior, since you are detaching the tensor with the <code>.data</code> call.<br>
I would consider the usage of <code>.data</code> dangerous, as you <em>might</em> get rid of error messages, but in fact create unwanted behavior.<br>
Is the code working without the <code>.data</code> call and by manually using <code>cuda()</code> on the slice?<br>
If so, I would stick to this solution. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: KFrank
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="62504" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Take this “debugging” with a grain of salt, but that would be my best guess:<br>
<code>param_need_l1_penalty_case_1</code> was defined as an <code>nn.Parameter</code> and just wrapped in a list.<br>
Iterating this list will yield these parameters, which were properly pushed to the device by calling <code>model.to('cuda')</code>, since they were also properly registered inside the module.<br>
However, an operation on <code>param_need_l1_penalty_case_2</code> was executed before storing them in the list (the slicing op). The result of this operation will be a <code>tensor</code> with a <code>grad_fn</code> (<code>SelectBackward</code> in this case).<br>
Since this operation was performed inside the <code>__init__</code> function of the module, and thus before pushing the parameters to the device, the slice will stay on CPU and you would need to push it manually to the device afterwards.</p>
</blockquote>
</aside>
<p>That was my thinking as well.  Anyway, explicitly moving the list<br>
elements did work (for me).</p>
<p>A couple of odd things:  According to Fly’s original post, he apparently<br>
<em>didn’t</em> see this error.  For me, this error occurred before calling<br>
<code>loss.backward()</code>, and was, understandably, of the form<br>
<code>expected type torch.cuda.FloatTensor</code> (instead of the<br>
other way around).</p>
<p>Also I <em>think</em> (I don’t remember everything I tried) that it involved<br>
the list in an essential way.  I <em>think</em> I tried getting rid of the list:</p>
<pre><code class="lang-auto">self.param_need_l1_penalty_case_2 = self.subModule.my_param[:,0]
</code></pre>
<p>and the (pre-<code>loss.backward()</code>) error went away, even though I was<br>
slicing before moving the model (and hence <code>my_param</code>  to the gpu.<br>
(Maybe <code>model.cuda()</code> knows to move <code>Tensor</code> variables of the<br>
model, but not lists.  And since Fly didn’t see it, maybe it’s an 0.3.0<br>
thing.)</p>
<aside class="quote no-group">
<blockquote>
<p>I doubt this will yield the desired behavior, since you are detaching the tensor with the <code>.data</code> call.<br>
I would consider the usage of <code>.data</code> dangerous, as you <em>might</em> get rid of error messages, but in fact create unwanted behavior.</p>
</blockquote>
</aside>
<p>I’m sure you’re right about this – I should have realized.  The forward<br>
calculation works, but (presumably) the backward calculation will fail<br>
to include the penalty-loss gradient in its overall gradient (defeating<br>
the purpose).</p>
<aside class="quote no-group">
<blockquote>
<p>Is the code working without the <code>.data</code> call and by manually using <code>cuda()</code> on the slice?<br>
If so, I would stick to this solution. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>No, two different errors:</p>
<p>Explicitly moving the slice gets rid of the reasonably-understandable<br>
<code>expected type torch.cuda.FloatTensor</code> error.</p>
<p>The error I tried to address with my incorrect modification was the<br>
<code>expected type torch.FloatTensor</code> that Fly originally reported,<br>
and that I was able to reproduce.</p>
<p>To me it’s baffling.  I probed all of the tensors in sight to make sure<br>
that they were <code>torch.cuda.FloatTensor</code>s, so I couldn’t figure out<br>
where (in the bowels of <code>loss.backward()</code>) somebody was expecting<br>
a <code>torch.FloatTensor</code>.  And it seems like this isn’t just 0.3.0 weirdness<br>
because Fly sees it with whatever his version is, as well.</p>
<p>Remaining baffled,</p>
<p>K. Frank</p>