Partially reset a Variable, in-place vs. new Variable

Atcold: <SCODE>def selective_zero(s, new):
    for b, reset in enumerate(new):
        if reset:
            for state_layer in s:
                state_layer.data[b].zero_()

selective_zero(state, y[t + 1] != y[t])
<ECODE> In order to complete this, I was thinking to register a hook in order to zero the gradient correspondingly as well. Something about this: <SCODE>for state_layer in s:
    state_layer.data[b].zero_()
    state_layer.register_hook(lambda grad: grad[b].zero_())
<ECODE>

albanD: If what you want is to reset the content of the tensor to use it again independently of how you were using it before, you should repack it in a new Variable otherwise it will still have the history of the previous usage.

Atcold: <SCODE>state_layer.data[b].zero_()
state_layer.register_hook(lambda grad: grad[b].zero_())
<ECODE> is equivalent to your <SCODE>state_layer[b].zero_()
<ECODE>

albanD: 

Atcold: 

albanD: 

Atcold: OK, making progress: <SCODE>In [22]: c
Out[22]: 
Variable containing:
 0.8491
 0.1877
 0.1560
 0.5188
 0.7464
[torch.FloatTensor of size 5]

In [23]: set(dir(c)) - set(dir(c.data))
Out[23]: 
{'__getattr__',
 '__rpow__',
 '__setstate__',
 '_add',
 '_addcop',
 '_backward_hooks',
 '_blas',
 '_creator',
 '_do_backward',
 '_execution_engine',
 '_fallthrough_methods',
 '_get_type',
 '_grad',
 '_static_blas',
 '_sub',
 '_version',
 'backward',
 'creator',
 'data',
 'detach',
 'detach_',
 'grad',
 'index_add',
 'index_copy',
 'index_fill',
 'masked_copy',
 'masked_fill',
 'output_nr',
 'register_hook',
 'reinforce',
 'requires_grad',
 'resize',
 'resize_as',
 'scatter',
 'volatile'}
<ECODE>

albanD: 

Atcold: <SCODE>>>> a = V(torch.rand(5), requires_grad=True)
Variable containing:
 0.8491
 0.1877
 0.1560
 0.5188
 0.7464
[torch.FloatTensor of size 5]

>>> c = a[2] * 0
Variable containing:
 0
[torch.FloatTensor of size 1]
<ECODE> <SCODE>>>> c = a.index_fill(0, 3, 0)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-25-ad8f493be136> in <module>()
----> 1 c = a.index_fill(0, 3, 0)

/home/atcold/anaconda3/lib/python3.5/site-packages/torch/autograd/variable.py in index_fill(self, dim, index, value)
    627 
    628     def index_fill(self, dim, index, value):
--> 629         return IndexFill(dim, value)(self, index)
    630 
    631     def index_fill_(self, dim, index, value):

RuntimeError: expected a Variable argument, but got int
<ECODE> The only think that worked is masking, but it’s not acceptable… <SCODE>In [12]: mask = V(torch.Tensor([1, 1, 1, 0, 1]))

In [13]: b = a * mask

In [14]: b
Out[14]: 
Variable containing:
 0.8491
 0.1877
 0.1560
 0.0000
 0.7464
[torch.FloatTensor of size 5]

In [15]: m = b.mean()

In [16]: m
Out[16]: 
Variable containing:
 0.3879
[torch.FloatTensor of size 1]

In [17]: m.backward()

In [18]: a.grad
Out[18]: 
Variable containing:
 0.2000
 0.2000
 0.2000
 0.0000
 0.2000
[torch.FloatTensor of size 5]
<ECODE>

albanD: <SCODE>import torch
from torch.autograd import Variable

a = Variable(torch.rand(5, 3), requires_grad=True)
a = a.clone() # Otherwise we change inplace a leaf Variable
print(a)

ind = Variable(torch.LongTensor([3]))
a.index_fill_(0, ind, 0)

print(a)

a[1, :] = 0

print(a)
<ECODE>

Atcold: