How to use forward on modules in a ModuleList?

ramsamy: <p>Hi,<br>
I have a net composed of layers in a nn::ModuleList. I didn’t succeed to apply the forward method on the modules in the list. How can I use them?</p>
<p>Here is the code to reproduce the issue:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

using namespace torch;

struct LinearImpl : nn::Module
{
    LinearImpl(int in_features, int out_features, bool bias):
        linear_layer(nn::LinearOptions(in_features, out_features).bias(bias))
    {
        register_module("linear_layer", linear_layer);
    }

    torch::Tensor forward(torch::Tensor x)
    {
        x = linear_layer(x);
        return x;
    }

    nn::Linear linear_layer;
};
TORCH_MODULE(Linear);

struct NetImpl : nn::Module
{
    NetImpl(int in_features, int out_features, bool bias):
        layers(Linear(in_features, out_features, bias),
               Linear(out_features, out_features, bias))
    {
        register_module("layers", layers);
    }

    torch::Tensor forward(torch::Tensor x)
    {
        for (const auto &amp;module : *layers)
        {
            x = module-&gt;forward(x);
        }
        return x;
    }

    nn::ModuleList layers;
};
TORCH_MODULE(Net);

int main()
{
    Net net(32, 64, false);

    torch::Tensor inputs = torch::randn(32);
    torch::Tensor outputs = net(inputs);

    std::cout &lt;&lt; outputs &lt;&lt; std::endl;

    return 0;
}
</code></pre>
<p>And the error I have during compilation:</p>
<pre><code class="lang-auto">main.cpp: In member function ‘at::Tensor NetImpl::forward(at::Tensor)’:
main.cpp:36:25: error: ‘using element_type = class torch::nn::Module {aka class torch::nn::Module}’ has no member named ‘forward’
             x = module-&gt;forward(x);
                         ^~~~~~~
</code></pre>
<p>I am using libtorch version 1.6.0.dev20200415+cpu on linux.</p>

albanD: <p>Hi,</p>
<p>I think you should be calling the module directly like you do in python (not call the forward function directly).</p>

ramsamy: <p>No, I tried this also but I have this error:</p>
<pre><code class="lang-auto">main.cpp: In member function ‘at::Tensor NetImpl::forward(at::Tensor)’:
main.cpp:36:25: error: no match for call to ‘(const std::shared_ptr&lt;torch::nn::Module&gt;) (at::Tensor&amp;)’
             x = module(x);
</code></pre>
<p>It works only when I defined a Linear layer outside the ModuleList. However I need the layers to be in a ModuleList for the rest of my program.</p>

albanD: <p>Not sure what is happening here<br>
<a class="mention" href="/u/yf225">@yf225</a> will know <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

yf225: <p><code>ModuleList</code> doesn’t store the modules’ type information, and we need to convert the modules to the concrete types for <code>forward</code> to work. So instead of doing <code>module-&gt;forward(x)</code>, we should do <code>module-&gt;as&lt;Linear&gt;()(x)</code>.</p>

ramsamy: <p><a class="mention" href="/u/yf225">@yf225</a> That solves the problem, thank you!</p>

yf225: <p><a class="mention" href="/u/ramsamy">@ramsamy</a> Sorry I made a mistake in my original answer, please make sure you are using  <code>module-&gt;as&lt;Linear&gt;()(x)</code> instead of <code>module-&gt;as&lt;Linear&gt;()-&gt;forward(x)</code>, otherwise module hooks (to be implemented) won’t work <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Alexey_Stolpovskiy: <p>This method seems not to work with a ModuleList of nn::Sequential. Is it possible to use nn::Sequential in the ModuleList, and if it is, then how to code the forward?</p>
<p>=============<br>
EDIT:<br>
Just found out, that<br>
module-&gt;as&lt;nn::Sequential&gt;()-&gt;forward(x)<br>
is working fine. However module-&gt;as&lt;nn::Sequential&gt;()(x) throws a compile-time error</p>

Ziyuan: <p>What if we create the ModuleList dynamically, and have no further knowledge about the actual sub-types?</p>

Ziyuan: <p>At the end I had to create an abstract class with a pure virtual function <code>forward</code> and let my modules derive from it.</p>

wongkenchiang97: <p>I tried the forward style as <a class="mention" href="/u/yf225">@yf225</a> mentioned above, it’s not working.</p>
<p>“encoder_” is a <strong>torch::nn::ModuleList</strong> with some <strong>DoubleConv</strong> in it.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c77a2d1d0e3c73ecb8bde6cfb78eca3b43be2fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c77a2d1d0e3c73ecb8bde6cfb78eca3b43be2fc" title="Screenshot from 2023-08-02 19-41-42"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c77a2d1d0e3c73ecb8bde6cfb78eca3b43be2fc_2_690x388.png" alt="Screenshot from 2023-08-02 19-41-42" data-base62-sha1="1Mi30UiIFSFW6AIIReiFAx8xtYE" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c77a2d1d0e3c73ecb8bde6cfb78eca3b43be2fc_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c77a2d1d0e3c73ecb8bde6cfb78eca3b43be2fc_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c77a2d1d0e3c73ecb8bde6cfb78eca3b43be2fc_2_1380x776.png 2x" data-dominant-color="262A2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-08-02 19-41-42</span><span class="informations">1920×1080 317 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>forward style in line 79 passed compilation but not sure is it implemented correctly?</p>

wongkenchiang97: <p>Then I used the <strong>Unet2D</strong> as a model for a training phase.<br>
I’ve encounter “aten::resize is not implemented” runtime error during the backward of the loss at line 134.</p><aside class="quote quote-modified" data-post="1" data-topic="184582">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wongkenchiang97/48/62403_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/encounter-aten-resize-is-not-implemented-runtime-error-during-backward-of-loss/184582">Encounter "aten::resize is not implemented" runtime error during backward of loss</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" style="--category-badge-color: #9EB83B; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    I’m using C++ frontend API to implement the training loop. 

“pred” are the the predictions of a 2D Unet model from a batch of training data.
“target” are the label of the training data.
“loss_f” is the torch::nn::BCEWithLogitsLoss().
“aten::resize is not implemented” runtime error appear at line 134.

 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec88eb8c63acf9eec7d328d6f36e91094b09c141.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec88eb8c63acf9eec7d328d6f36e91094b09c141" title="image">[image]</a> 
Any idea to solve this issue?
  </blockquote>
</aside>

<p>Does this runtime error relevent to the forward style?</p>

jayandra_l: <p><a class="mention" href="/u/yf225">@yf225</a> This solution solves when all the Objects are of same dtype. Could you point out what should be done when there are more than one dtype in ModuleList?</p>