How to access to a layer by module name?

Cyril_Tang: <p>I have a ResNet34 model and I want to find all the ReLU layer. I used named_modules() method to get the layers.</p>
<pre><code class="lang-auto">for name, layer in model.named_modules():
    if isinstance(layer, nn.ReLU):
        print(name, layer)
</code></pre>
<p>And got out put like</p>
<pre><code class="lang-auto">0.2 ReLU(inplace=True)
0.4.0.relu ReLU(inplace=True)
0.4.1.relu ReLU(inplace=True)
0.4.2.relu ReLU(inplace=True)
</code></pre>
<p>Is there any way that I can use the name (for example 0.4.0.relu) directly to access to a relu layer? Or I have to index the number in [] like model[0][4][0].relu?</p>

surya00060: <p>Hi</p>
<pre><code class="lang-auto">model = torchvision.models.resnet34()
&gt;&gt;&gt; model
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
</code></pre>
<p>You can access the relu followed by conv1</p>
<pre><code class="lang-auto">model.relu
</code></pre>
<p>Also, If you want to access the ReLU layer in layer1, you can use the following code to access ReLU in basic block 0 and 1.</p>
<pre><code class="lang-auto">model.layer1[0].relu
model.layer1[1].relu
</code></pre>
<p>You can index the numbers in the name obtained from named_modules using <strong>model[]</strong>.<br>
If you have a string layer1, you have to use it as <strong>model.layer1</strong>.<br>
nn.Sequential objects are indexable whereas nn.Module objects are not.</p>
<p>I hope this solves your question.</p>

edyuan: <p>I have the same question as OP which I don’t think was answered by <a class="mention" href="/u/surya00060">@surya00060</a></p>
<p>If model.name_modules() contains names for all the different layers, there must be a way to access the layers by using the name string, No?</p>
<p>The name is much more convenient than something like:</p>
<pre><code>model.layer1[0].relu</code></pre>

surya00060: <p>Hi <a class="mention" href="/u/edyuan">@edyuan</a></p>
<p>I think it is not possible to access all layers of PyTorch by their names. If you see the names, it has indices when the layer was created inside nn.Sequential and otherwise has a module name.</p>
<pre><code class="lang-auto">for name, layer in model.named_modules():
...     if isinstance(layer, torch.nn.Conv2d):
...             print(name, layer)
</code></pre>
<p>The output for this snippet is</p>
<pre><code class="lang-auto">conv1 Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
layer1.0.conv1 Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer1.0.conv2 Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer1.1.conv1 Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer1.1.conv2 Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer2.0.conv1 Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
layer2.0.conv2 Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer2.0.downsample.0 Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
layer2.1.conv1 Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer2.1.conv2 Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer3.0.conv1 Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
layer3.0.conv2 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer3.0.downsample.0 Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
layer3.1.conv1 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer3.1.conv2 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer4.0.conv1 Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
layer4.0.conv2 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer4.0.downsample.0 Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
layer4.1.conv1 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
layer4.1.conv2 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
</code></pre>
<p>If you see the name of all layers, it’s of the format <code>string.number</code>. If there’s a number in your name, you need to access it by <code>model.string.[number]</code>.</p>
<p>I hope this clarifies your doubt.</p>

tree: <p>Hi there, I had a somewhat related problem, with the use case of applying some function to specific modules based on their name (as <code>state_dict</code> and <code>named_modules</code> do return some kind of names) instead of based on their type (as I’ve seen everywhere by now, e.g. <a href="https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.apply" rel="nofollow noopener">here</a>).</p>
<p>And I ended up with this hack - calling it hack as I’m pretty new to PyTorch and not sure yet it is 100% safe - just create the names by yourself:</p>
<pre><code class="lang-auto">for n, m in model.named_modules():
     m.auto_name = n
</code></pre>
<p>You can then filter when iterating over e.g. <code>model.modules()</code>, using <code>.auto_name</code>.</p>
<p>Hope this helps!</p>

klory: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b97f4a0c7606df7c6680a827ec92476f82d13612.png" data-download-href="https://discuss.pytorch.org/uploads/default/b97f4a0c7606df7c6680a827ec92476f82d13612" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b97f4a0c7606df7c6680a827ec92476f82d13612_2_690x359.png" alt="image" data-base62-sha1="qsZ2YvwTo43kJLQnTdEOtlR3Vyq" width="690" height="359" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b97f4a0c7606df7c6680a827ec92476f82d13612_2_690x359.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b97f4a0c7606df7c6680a827ec92476f82d13612_2_1035x538.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b97f4a0c7606df7c6680a827ec92476f82d13612_2_1380x718.png 2x" data-dominant-color="EEEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1746×910 155 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Inder: <p>If the layers are named you can access them as you described:</p>
<pre><code class="lang-auto">for name, layer in model.named_modules():
    if isinstance(layer, nn.ReLU):
        print(name, layer)

    pytorch_layer_obj = getattr(model, name)
</code></pre>

remisphere: <h1><a name="tldr-1" class="anchor" href="#tldr-1"></a>TL;DR</h1>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; from functools import reduce
&gt;&gt;&gt; from torchvision.models import resnet34
&gt;&gt;&gt;
&gt;&gt;&gt; def get_module_by_name(module, access_string):
...     names = access_string.split(sep='.')
...     return reduce(getattr, names, module)
&gt;&gt;&gt;
&gt;&gt;&gt; model = resnet34()
&gt;&gt;&gt; get_module_by_name(model, 'layer1.0.relu')
ReLU(inplace=True)
</code></pre>
<hr>
<p>The most confusing point is <code>nn.Sequential</code> defaulting to naming its modules with numbers, that are not valid python identifiers, thus not being reachable by a dot notation like <code>layer1.0.relu</code>, if I take <code>torchvision</code>’s <code>resnet</code>.</p>
<p>But you can work around it.</p>
<hr>
<h1><a name="accessing-direct-child-by-name-2" class="anchor" href="#accessing-direct-child-by-name-2"></a>Accessing direct child by name</h1>
<p>If you have your hands on the network implementation, you can choose valid names for the <code>Sequential</code> by using an <code>OrderedDict</code> like shown in the <a href="https://pytorch.org/docs/1.10.1/generated/torch.nn.Sequential.html?highlight=sequential#torch.nn.Sequential" rel="noopener nofollow ugc">documentation</a>, and access the layers like you would for anything other than <code>Sequential</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; model = nn.Sequential(OrderedDict([
...     ('conv1', nn.Conv2d(1,20,5)),
...     ('relu1', nn.ReLU()),
...     ('conv2', nn.Conv2d(20,64,5)),
...     ('relu2', nn.ReLU())]))
&gt;&gt;&gt; model.relu1
ReLU()
</code></pre>
<p>As <a class="mention" href="/u/inder">@Inder</a>  pointed out, an alternative to the dot notation in order to access attributes is the built-in <code>getattr()</code> function.<br>
But unlike the dot notation, it accepts <strong>any string</strong>, and it happens to work on <code>Sequential</code>’s default naming when using numerical strings (<code>Sequential</code> is actually built not on a list but on a <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/container.py#L74" rel="noopener nofollow ugc"><code>dict</code> with <code>string</code> keys</a>).</p>
<blockquote>
<p>To get back to my first example, doing <code>getattr(model.layer1, '0').relu</code> works.</p>
</blockquote>
<hr>
<h1><a name="accessing-nested-child-by-name-3" class="anchor" href="#accessing-nested-child-by-name-3"></a>Accessing nested child by name</h1>
<p>Something nice of <code>getattr()</code> is that you can go multiple levels at a time, for example <code>getattr(foo, 'bar.baz')</code> to get to <code>foo.bar.baz</code>.<br>
Unfortunately, it only works if the whole dot chain is made of valid identifiers.<br>
The solution is to use <code>getattr()</code> recursively yourself, and that’s easy enough with <code>reduce()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">from functools import reduce
from typing import Union

import torch
from torch import nn


def get_module_by_name(module: Union[torch.Tensor, nn.Module],
                       access_string: str):
    """Retrieve a module nested in another by its access string.

    Works even when there is a Sequential in the module.
    """
    names = access_string.split(sep='.')
    return reduce(getattr, names, module)


if __name__ == '__main__':
    from torchvision.models import resnet34
    
    model = resnet34()
    get_module_by_name(model, 'layer1.0.relu')
</code></pre>
<p>Output:</p>
<pre data-code-wrap="txt"><code class="lang-txt">ReLU(inplace=True)
</code></pre>

Sushmanth_reddy_Mere: <pre><code class="lang-auto">import torch.nn as nn
from typing import Union

def find_layer(model: nn.Module, identifier: Union[str, int]) -&gt; nn.Module:
    """
    Find a layer in a PyTorch model either by its name using dot notation for nested layers or by its index.

    Parameters
    ----------
    model : nn.Module
        Model from which to search for the layer.
    identifier : str or int
        Layer name using dot notation for nested layers or layer index to find in the model.

    Returns
    -------
    nn.Module
        The layer found, or None if no such layer exists.

    Raises
    ------
    ValueError
        If the identifier is neither a string nor an integer.
    """
    # Flatten the model into a list of layers if index is provided
    if isinstance(identifier, int):
        layers = []
        def flatten_model(module):
            for child in module.children():
                if len(list(child.children())) == 0:
                    layers.append(child)
                else:
                    flatten_model(child)
        flatten_model(model)
        if 0 &lt;= identifier &lt; len(layers):
            return layers[identifier]
        return None

    elif isinstance(identifier, str):
        # Access by dot-notated name
        parts = identifier.split('.')
        current_module = model
        try:
            for part in parts:
                current_module = getattr(current_module, part)
            return current_module
        except AttributeError:
            return None
    else:
        raise ValueError(f"Identifier must be either an integer or a string, got {type(identifier)}.")

# Example usage:
import torchvision.models as models

model = models.resnet18(pretrained=True)

# Retrieve a nested layer by dot notation name
layer_by_name = find_layer(model, 'layer1.0.conv1')
if layer_by_name:
    print(f"Retrieved by name: {layer_by_name}")
else:
    print("No such layer found by name.")

# Retrieve layer by index
layer_by_index = find_layer(model, 10)
if layer_by_index:
    print(f"Retrieved by index: {layer_by_index}")
else:
    print("No such layer found by index.")

</code></pre>
<p>By this code u can access layers by string  and int also…</p>