How to clear the modules output history when using register_forward_hook?

sakimarquis: <p>First I use a pytorch pretrained Resnet, then I use these codes to get the hidden feature.</p>
<pre><code class="lang-auto">feat_out = []

def hook_fn_forward(module, input, output):
    feat_out.append(output)
    print(output)

modules = model.named_children()
for name, module in modules:
    module.register_forward_hook(hook_fn_forward)

pred = model(x)
</code></pre>
<p>But when I run these codes the first time, <code>len(feat_out)</code> gives me 10, and <code>print</code> in hook function prints 10 lines. If I run these codes again, <code>len(feat_out)</code> gives me 20, and <code>print</code> in hook function prints 20 lines. Every time I run, the length of <code>output</code> in hook function increase by 1. The output is output in this time plus all past output. Only if I reinitialize the model and run these codes, the past output history will be removed.</p>
<p>How can I clear the output every time I run the model?</p>
<p>I use these codes in <a href="https://colab.research.google.com/drive/1pzVPBkZiHCzu7MiIcEm_gLcMqXmbUpLY?usp=sharing" rel="nofollow noopener">colab</a> to reproduce this problem in minimum length (5 lines to load data, 2 lines to initialize model, 8 lines for this problem).</p>

ptrblck: <p>Since you are initializing the <code>feat_out</code> <code>list</code> as a global object, this behavior is expected.<br>
You could either reinitialize it after the first forward pass or use a <code>dict</code> instead, which can be easily used to replace the outputs.<br>
Also note, if you are not running the forward pass in a <code>with torch.no_grad()</code> block, the <code>output</code> tensors will stay attached to the computation graph, which will prevent PyTorch from deleting this graph after the <code>backward</code> call. If you don’t need these <code>output</code> tensors for a gradient calculation you could alternatively store the detached version via <code>feat_out.append(output.detach())</code>.</p>

sakimarquis: <p>These are my full code, in case you don’t want to go to Colab.</p>
<pre><code class="lang-auto">!wget https://download.pytorch.org/tutorial/hymenoptera_data.zip
!unzip -q hymenoptera_data.zip

data_transforms = transforms.Compose([
                             transforms.RandomResizedCrop(224),
                             transforms.ToTensor()])
data_dir = 'hymenoptera_data//train'
image_datasets = datasets.ImageFolder(os.path.join(data_dir), transform=data_transforms)
dataloaders = torch.utils.data.DataLoader(image_datasets, batch_size=4, num_workers=4)

model_ft = models.resnet18(pretrained=True)
model_ft.fc = nn.Linear(model_ft.fc.in_features, 2) #  size of each output is set to 2

dat, lab = next(iter(dataloaders))


total_feat_out = []

def hook_fn_forward(module, input, output):
    total_feat_out.append(output) 
    print(output.shape)

modules = model_ft.named_children()
for name, module in modules:
    module.register_forward_hook(hook_fn_forward)

model_ft.eval() 
with torch.no_grad():
    pred = model_ft(dat)
</code></pre>

sakimarquis: <p>Thank you for your attention. This problem also make output in hook function increase, every time it runs, it increases by 1. So the  <code>print</code>  in hook function prints 10 lines more.</p>

ptrblck: <p>You are still appending to the global list, so the list will grow in each forward pass.<br>
To store only the activation of the last forward pass you would need to reinitialize the list or clear it in any other way:</p>
<pre><code class="lang-python">with torch.no_grad():
    out = model_ft(torch.randn(1, 3, 224, 224))
    print(len(total_feat_out)) # prints 10
    total_feat_out = [] # reinitialize list to delete old activations
</code></pre>

sakimarquis: <p>Thank you very much. I finally get what I want!!</p>
<p>I avoid this problem by using  <code>dict</code>  as you told me to replace the outputs.</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)

feature_out = {}
layers_name = list(model_ft._modules.keys())
layers = list(model_ft._modules.values())


def hook_fn_forward(module, input, output):
    layer = layers_name[np.argwhere([module == m for m in layers])[0, 0]]
    total_feat_out[layer] = output
    

modules = model_ft.named_children()
for name, module in modules:
    module.register_forward_hook(hook_fn_forward)

model_ft.eval() 
with torch.no_grad():
    pred = model_ft(dat)
</code></pre>

sakimarquis: <p>Also, from one of your past answer, I found that I didn’t remove the hooks, which essencially caused the problems.</p>
<pre><code class="lang-auto">total_feat_out = []

def hook_fn_forward(module, input, output):
    total_feat_out.append(output) 
    print(output.shape)

modules = model_ft.named_children()
handles = {}
for name, module in modules:
    handles[name] = module.register_forward_hook(hook_fn_forward)

model_ft.eval() 
with torch.no_grad():
    pred = model_ft(dat)
    for k, v in handles.items():
        handles[k].remove()
``</code></pre>