Error when i create new model and split model

vophuctinh: <p>I need to process and refine the model for many different purposes, one automatically (split, concat, …) both in terms of model and weight. I can use some models (VGG, EffciennetB3, Xception, ViT), however in some variants of ViT (CrossViT, CrossFormer) I get error messages, still can’t find a fix.</p>
<p>import torch<br>
from vit_pytorch.cross_vit import CrossViT<br>
import torch.nn as nn<br>
v = CrossViT(<br>
image_size = 256,<br>
num_classes = 1000,<br>
depth = 4,               # number of multi-scale encoding blocks<br>
sm_dim = 192,            # high res dimension<br>
sm_patch_size = 16,      # high res patch size (should be smaller than lg_patch_size)<br>
sm_enc_depth = 2,        # high res depth<br>
sm_enc_heads = 8,        # high res heads<br>
sm_enc_mlp_dim = 2048,   # high res feedforward dimension<br>
lg_dim = 384,            # low res dimension<br>
lg_patch_size = 64,      # low res patch size<br>
lg_enc_depth = 3,        # low res depth<br>
lg_enc_heads = 8,        # low res heads<br>
lg_enc_mlp_dim = 2048,   # low res feedforward dimensions<br>
cross_attn_depth = 2,    # cross attention rounds<br>
cross_attn_heads = 8,    # cross attention heads<br>
dropout = 0.1,<br>
emb_dropout = 0.1<br>
)</p>
<p>v = nn.Sequential(*list(v.children()))<br>
v1 = nn.Sequential(*list(v.children()[:5]))</p>
<p>img = torch.randn(1, 3, 256, 256)</p>
<p>pred = v(img) # (1, 1000)<br>
print(pred)</p>
<p>“einops.EinopsError: Expected 4 dimensions, got 3”</p>
<p>from vit_pytorch.crossformer import CrossFormer</p>
<p>model = CrossFormer(<br>
num_classes = 1000,                # number of output classes<br>
dim = (64, 128, 256, 512),         # dimension at each stage<br>
depth = (2, 2, 8, 2),              # depth of transformer at each stage<br>
global_window_size = (8, 4, 2, 1), # global window sizes at each stage<br>
local_window_size = 7,             # local window size (can be customized for each stage, but in paper, held constant at 7 for all stages)<br>
)</p>
<p>img = torch.randn(1, 3, 224, 224)<br>
model = nn.Sequential(*list(model.children()))<br>
pred = model(img) # (1, 1000)</p>
<p>“NotImplementedError: Module [ModuleList] is missing the required “forward” function”</p>

ptrblck: <p>I don’t know where this error is raised from:</p>
<aside class="quote no-group" data-username="vophuctinh" data-post="1" data-topic="181528">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vophuctinh/48/61066_2.png" class="avatar"> vophuctinh:</div>
<blockquote>
<p>“einops.EinopsError: Expected 4 dimensions, got 3”</p>
</blockquote>
</aside>
<p>but the error message is pointing towards a dimension mismatch so you would need to check which tensors are used and that they have the expected number of dimensions.</p>
<aside class="quote no-group" data-username="vophuctinh" data-post="1" data-topic="181528">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vophuctinh/48/61066_2.png" class="avatar"> vophuctinh:</div>
<blockquote>
<p>“NotImplementedError: Module [ModuleList] is missing the required “forward” function”</p>
</blockquote>
</aside>
<p>This error is raise if you try to “call” an <code>nn.ModuleList</code> object like a module, which won’t work as you would need to iterate it.<br>
Here is a small example:</p>
<pre data-code-wrap="python"><code class="lang-python">modules = nn.ModuleList([
    nn.Linear(10, 10),
    nn.Linear(10, 10),
    nn.Linear(10, 10),
    nn.Linear(10, 10),
    nn.Linear(10, 10),
])

x = torch.randn(1, 10)

# breaks
out = modules(x)
# NotImplementedError: Module [ModuleList] is missing the required "forward" function

# works
for module in modules:
    x = module(x)
</code></pre>

walkii22: <p>Please I have a similar issue with my script here:<br>
from IPython.core.completerlib import module_list<br>
from numpy.core.fromnumeric import shape<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
from torch.nn.modules.container import ModuleList, Sequential<br>
from methods.utils.model_utilities_orth import   ( CustomCNN,PositionalEncoding, init_layer) #( DoubleConv,PositionalEncoding,  init_layer)<br>
from methods.utils.cbam import  CBAM<br>
from performer_stuffs.slim_performer_model import SLiMPerformerLayer</p>
<p><span class="hashtag-raw">#import</span> performer.models.slim_performer.pytorch.numerator_and_denominator as num_and_den<br>
from performer_stuffs import numerator_and_denominator as num_and_den</p>
<p>class EINV2(nn.Module):<br>
def <strong>init</strong>(self, cfg, dataset):<br>
super().<strong>init</strong>()<br>
self.pe_enable = False  # Ture | False</p>
<pre><code>    if cfg['data']['audio_feature'] == 'logmel&amp;intensity':
        self.f_bins = cfg['data']['n_mels']
        self.in_channels = 7
        n_layers = 2
        feature_type = 'logmel&amp;intensity'
    self.downsample_ratio = 2 ** 2
    self.sed_conv_block1 = nn.Sequential(
    CustomCNN(in_channels=4, out_channels=64, kernel_size = 3),
       
    )
    # self.cbam_sed1 = CBAM(gate_channels =64) 
    nn.AvgPool2d(kernel_size=(2, 2))
    
    self.sed_conv_block2 = nn.Sequential(
       CustomCNN(in_channels=64, out_channels=128, kernel_size = 3),
        nn.AvgPool2d(kernel_size=(2, 2)),
    )
    # self.cbam_sed2 = CBAM(gate_channels =128) 
    # nn.AvgPool2d(kernel_size=(2, 2)),
    
    self.sed_conv_block3 = nn.Sequential(
    CustomCNN(in_channels=128, out_channels=256, kernel_size = 3),
        nn.AvgPool2d(kernel_size=(1, 2)),
    )
    # self.cbam_sed3 = CBAM(gate_channels =256) 
    # nn.AvgPool2d(kernel_size=(2, 2))
    self.sed_conv_block4 = nn.Sequential(
      CustomCNN(in_channels=256, out_channels=512, kernel_size = 3),
        nn.AvgPool2d(kernel_size=(1, 2)),
    )
    # self.cbam_sed4 = CBAM(gate_channels =512) 
    # nn.AvgPool2d(kernel_size=(2, 2))
    self.doa_conv_block1 = nn.Sequential(
        CustomCNN(in_channels=self.in_channels, out_channels=64, kernel_size = 3),
        nn.AvgPool2d(kernel_size=(2, 2)),
    )
    # self.cbam_doa1 = CBAM(gate_channels =64) 
    # nn.AvgPool2d(kernel_size=(2, 2))
    self.doa_conv_block2 = nn.Sequential(
     CustomCNN(in_channels=64, out_channels=128, kernel_size = 3),
        nn.AvgPool2d(kernel_size=(2, 2)),
    )
    # self.cbam_doa2 = CBAM(gate_channels =128) 
    nn.AvgPool2d(kernel_size=(2, 2))
    self.doa_conv_block3 = nn.Sequential(
      CustomCNN(in_channels=128, out_channels=256, kernel_size = 3),
        nn.AvgPool2d(kernel_size=(1, 2)),
    )
   
    self.doa_conv_block4 = nn.Sequential(
       CustomCNN(in_channels=256, out_channels=512, kernel_size = 3),
        nn.AvgPool2d(kernel_size=(1, 2)),
    )
   
    self.stitch = nn.ParameterList([
        nn.Parameter(torch.FloatTensor(64, 2, 2).uniform_(0.1, 0.9)),
        nn.Parameter(torch.FloatTensor(128, 2, 2).uniform_(0.1, 0.9)),
        nn.Parameter(torch.FloatTensor(256, 2, 2).uniform_(0.1, 0.9)),
    ])
    if self.pe_enable:
        self.pe = PositionalEncoding(pos_len=100, d_model=512, pe_type='t', dropout=0.0)
    self.sed_trans_track1 = torch.nn.ModuleList([
    SLiMPerformerLayer(hidden_dim = 512, ffn_dim= 2048, n_heads=8, feature_type = feature_type,
                       compute_type =iter, on_gptln =True) for _ in range(n_layers)]

)  
    self.sed_trans_track2 = torch.nn.ModuleList([
    SLiMPerformerLayer(hidden_dim = 512, ffn_dim= 2048, n_heads=8, feature_type = feature_type,
                       compute_type =iter, on_gptln =True) for _ in range(n_layers)]
    )
    self.doa_trans_track1 = torch.nn.ModuleList([
    SLiMPerformerLayer(hidden_dim = 512, ffn_dim= 2048, n_heads=8, feature_type = feature_type,
                       compute_type =iter, on_gptln =True) for _ in range(n_layers)]
) 
    
    self.doa_trans_track2 = torch.nn.ModuleList([
    SLiMPerformerLayer(hidden_dim = 512, ffn_dim= 2048, n_heads=8, feature_type = feature_type,
                       compute_type =iter, on_gptln =True) for _ in range(n_layers)]

)

    self.fc_sed_track1 = nn.Linear(512, 14, bias=True)
    self.fc_sed_track2 = nn.Linear(512, 14, bias=True)
    self.fc_doa_track1 = nn.Linear(512, 3, bias=True)
    self.fc_doa_track2 = nn.Linear(512, 3, bias=True)
    self.final_act_sed = nn.Sequential() # nn.Sigmoid()
    self.final_act_doa = nn.Tanh()

    self.init_weight()

def init_weight(self):

    init_layer(self.fc_sed_track1)
    init_layer(self.fc_sed_track2)
    init_layer(self.fc_doa_track1)
    init_layer(self.fc_doa_track2)

def forward(self, x):
    """
    x: waveform, (batch_size, num_channels, data_length)
    """
    x_sed = x[:, :4]
    x_doa = x

    # cnn
    x_sed = self.sed_conv_block1(x_sed)
    x_doa = self.doa_conv_block1(x_doa)
    

    x_sed = self.sed_conv_block4(x_sed)
    x_doa = self.doa_conv_block4(x_doa)
    x_sed = x_sed.mean(dim=3) # (N, C, T)
    x_doa = x_doa.mean(dim=3) # (N, C, T)

    #transformer
    # if self.pe_enable:
    #     x_sed = self.pe(x_sed)
    # if self.pe_enable:
    #     x_doa = self.pe(x_doa)
    x_sed = self.sed_trans_track1(x_sed)
    x_sed = self.sed_trans_track2(x_sed)
    
    x_sed = self.doa_trans_track1(x_doa)
    x_sed = self.doa_trans_track1(x_doa)
    x_sed = x_sed.permute(2, 0, 1) # (T, N, C)
    x_doa = x_doa.permute(2, 0, 1) # (T, N, C)

    x_sed_1 = self.sed_trans_track1(x_sed).transpose(0, 1) # (N, T, C)
    x_sed_2 = self.sed_trans_track2(x_sed).transpose(0, 1) # (N, T, C)   
    x_doa_1 = self.doa_trans_track1(x_doa).transpose(0, 1) # (N, T, C)
    x_doa_2 = self.doa_trans_track2(x_doa).transpose(0, 1) # (N, T, C)

    
    x_sed_1 = self.final_act_sed(self.fc_sed_track1(x_sed_1))
    x_sed_2 = self.final_act_sed(self.fc_sed_track2(x_sed_2))
    x_sed = torch.stack((x_sed_1, x_sed_2), 2)
    x_doa_1 = self.final_act_doa(self.fc_doa_track1(x_doa_1))
    x_doa_2 = self.final_act_doa(self.fc_doa_track2(x_doa_2))
    x_doa = torch.stack((x_doa_1, x_doa_2), 2)
    output = {
        'sed': x_sed,
        'doa': x_doa,
    }

    return output
</code></pre>
<p>raise NotImplementedError(f"Module [{type(self).<strong>name</strong>}] is missing the required "forward" function")<br>
NotImplementedError: Module [ModuleList] is missing the required “forward” function</p>

ptrblck: <pre data-code-wrap="python"><code class="lang-python">NotImplementedError: Module [ModuleList] is missing the required “forward” function
</code></pre>
<p>You cannot call the <code>nn.ModuleList</code> directly as it’s not implementing a <code>forward</code> method and you thus need to iterate each module separately in a loop. If you want to execute the internal modules sequentially, use <code>nn.Sequential</code> instead.</p>

walkii22: <p>Could you provide me with an example of how to implement this using my script? Thanks</p>

ptrblck: <p>In the <code>forward</code> method you would need to replace the usage of each <code>nn.ModuleList</code> from:</p>
<pre data-code-wrap="python"><code class="lang-python">x_sed = self.sed_trans_track1(x_sed)
</code></pre>
<p>to</p>
<pre data-code-wrap="python"><code class="lang-python">for module in self.sed_trans_track1:
    x_sed = module(x_sed)
</code></pre>

walkii22: <p>Thanks , but the error  still persists after I have replaced the implementation of the nn. ModuleList  in the forward method like this;<br>
for module in self.sed_trans_track1:<br>
x_sed = module(x_sed)<br>
for module in self.sed_trans_track2:<br>
x_doa = module(x_sed)<br>
for module in self.doa_trans_track1:<br>
x_sed = module(x_sed)<br>
for module in self.doa_trans_track2:<br>
x_doa = module(x_sed)<br>
x_sed = x_sed.permute(2, 0, 1) # (T, N, C)<br>
x_doa = x_doa.permute(2, 0, 1) # (T, N, C)</p>
<pre><code>    x_sed_1 = self.sed_trans_track1(x_sed).transpose(0, 1) # (N, T, C)
    x_sed_2 = self.sed_trans_track2(x_sed).transpose(0, 1) # (N, T, C)   
    x_doa_1 = self.doa_trans_track1(x_doa).transpose(0, 1) # (N, T, C)
    x_doa_2 = self.doa_trans_track2(x_doa).transpose(0, 1) # (N, T, C)
    x_sed_1 = self.final_act_sed(self.fc_sed_track1(x_sed_1))
    x_sed_2 = self.final_act_sed(self.fc_sed_track2(x_sed_2))
    x_sed = torch.stack((x_sed_1, x_sed_2), 2)
    x_doa_1 = self.final_act_doa(self.fc_doa_track1(x_doa_1))
    x_doa_2 = self.final_act_doa(self.fc_doa_track2(x_doa_2))
    x_doa = torch.stack((x_doa_1, x_doa_2), 2)
    output = {
        'sed': x_sed,
        'doa': x_doa,
    }

    return output
</code></pre>
<p>I have four tracks in all (2 tracks for sed and doa each) and each track has 2 SLiMperformer layers</p>

ptrblck: <p>This would mean that you are still calling an <code>nn.ModuleList</code> directly:</p>
<pre data-code-wrap="python"><code class="lang-python">modules = nn.ModuleList([
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1)
])
x = torch.randn(1, 1)

out = modules(x)
# NotImplementedError: Module [ModuleList] is missing the required "forward" function
</code></pre>
<p>instead of iterating it:</p>
<pre data-code-wrap="python"><code class="lang-python">for module in modules:
    x = module(x)
</code></pre>
<p>and the stacktrace should show which line of code fails.</p>

walkii22: <p>the code fails@<br>
x = module(x)</p>
<pre><code class="lang-auto">I am using the SLiMPerformerLayer from 'How to Make Performers SLiM' paper</code></pre>

ptrblck: <p>Could you post the stacktrace including the error message?</p>

walkii22: <p>0%|                                                                                                                                                                             | 0/11800 [00:13&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File “seld/main.py”, line 59, in <br>
sys.exit(main(args, cfg))<br>
File “seld/main.py”, line 43, in main<br>
train.train(cfg, **train_initializer)<br>
File “/mnt/raid/ni/WALE_SEdl/EIN-SELD/seld/learning/train.py”, line 47, in train<br>
epoch_it=epoch_it<br>
File “/mnt/raid/ni/WALE_SEdl/EIN-SELD/seld/methods/ein_seld/training.py”, line 127, in validate_step<br>
pred = self.model(batch_x)<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 168, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 178, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py”, line 86, in parallel_apply<br>
output.reraise()<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/_utils.py”, line 461, in reraise<br>
raise exception<br>
NotImplementedError: Caught NotImplementedError in replica 0 on device 0.<br>
Original Traceback (most recent call last):<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker<br>
output = module(*input, **kwargs)<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/mnt/raid/ni/WALE_SEdl/EIN-SELD/seld/methods/ein_seld/models/seld.py”, line 173, in forward<br>
x_sed = module(x_sed)<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/mnt/antares_raid/home/wale/anaconda3/envs/ein/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 201, in _forward_unimplemented<br>
raise NotImplementedError(f"Module [{type(self).<strong>name</strong>}] is missing the required "forward" function")<br>
NotImplementedError: Module [SLiMPerformerLayer] is missing the required “forward” function<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7da78633c2625fbe213cd11eb58dde9f3d687881.png" data-download-href="https://discuss.pytorch.org/uploads/default/7da78633c2625fbe213cd11eb58dde9f3d687881" title="Screenshot from 2024-03-03 23-43-59"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da78633c2625fbe213cd11eb58dde9f3d687881_2_690x392.png" alt="Screenshot from 2024-03-03 23-43-59" data-base62-sha1="hVABnbYbbQ6nOFGMShUtSzB8HO9" width="690" height="392" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da78633c2625fbe213cd11eb58dde9f3d687881_2_690x392.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da78633c2625fbe213cd11eb58dde9f3d687881_2_1035x588.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da78633c2625fbe213cd11eb58dde9f3d687881_2_1380x784.png 2x" data-dominant-color="AB9CA6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-03-03 23-43-59</span><span class="informations">1850×1053 265 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thanks</p>

ptrblck: <p>I don’t know which module you are using but <a href="https://github.com/yinkalario/EIN-SELD/blob/9f42da23f4ef4620ca495af37e079dc7f8cde06c/seld/methods/ein_seld/models/seld.py"><code>EIN-SELD:/seld/methods/ein_seld/models/seld.py</code></a> does not contain any code beyond line 141, so no idea what exactly is being called there.</p>