Multi model DataParallel

sujeongEOM: <p>Hi.<br>
I’m trying to train Deep Ensembles using PyTorch.</p>
<p>My model is has this structure.</p>
<pre><code class="lang-auto">class ResNetDeepEnsembles1d(nn.Module):
    def __init__(self, input_dim, blocks_dim, resblock_repeat, num_models, n_classes, kernel_size=17, dropout_rate=0.8):
        super(ResNetDeepEnsembles1d, self).__init__()
        self.num_models = num_models
        # Ensembles
        for i in range(self.num_models):
            model = ResNet1d(input_dim=input_dim,
                        blocks_dim=blocks_dim, 
                        resblock_repeat=resblock_repeat, 
                        n_classes=n_classes, 
                        kernel_size=kernel_size, 
                        dropout_rate=dropout_rate)
            setattr(self, f'model_{str(i)}',  model)

    def forward(self, x):
        mus, sigmas = [], []
        for i in range(self.num_models):
            model = getattr(self, f'model_{str(i)}')
            mu, sigma = model(x)
            mus.append(mu)
            sigmas.append(sigma)

        mus = torch.stack(mus)
        mu = mus.mean(dim=0)
        sigmas = torch.stack(sigmas)
        sigma = (sigmas + mus.pow(2)).mean(dim=0) - mu.pow(2)

        return mu, sigma
</code></pre>
<p>And here is my train script.</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

for ep in range(start_epoch, config.epochs):
        train_total_loss = np.zeros(setup['num_models'])
        train_n_entries = np.zeros(setup['num_models'])
        train_desc = "Epoch {:2d}: train - Model {:2d} Loss: {:.6f}"
        train_bar = tqdm(initial=0, leave=True, total=len(train_loader), 
                        desc=train_desc.format(ep, 0, 0), position=0)

        for traces, ages, weights in train_loader:
            traces = traces.transpose(1, 2).float()
            for i in range(setup['num_models']):
                model = getattr(resnet, f'model_{str(i)}')
                model.train()
        
                model = nn.DataParallel(model)
                model.to(device)
                traces, ages, weights = traces.to(device), ages.to(device), weights.to(device)
                
                model.zero_grad()

                pred_ages, pred_sigma = model(traces)
                loss = compute_loss(ages, pred_ages, pred_sigma, weights)

                loss.backward()

                optimizers[i].step()

                bs = len(traces)
                train_total_loss[i] += loss.detach().cpu().numpy()
                train_n_entries[i] += bs

                train_bar.desc = train_desc.format(ep, i+1, train_total_loss[i] / train_n_entries[i])
                train_bar.update(1)
        train_loss = train_total_loss / train_n_entries

</code></pre>
<p>I want to make sure that each model is trained with same data batch input.<br>
But this keeps giving this error.</p>
<pre><code class="lang-auto">RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper__cudnn_convolution)
</code></pre>
<p>I’m on this error for weeks, please help <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Thanks.</p>

ptrblck: <p>I guess something is wrong in these lines:</p>
<pre><code class="lang-python">                model = getattr(resnet, f'model_{str(i)}')
                model.train()
        
                model = nn.DataParallel(models)
                model.to(device)
</code></pre>
<p>as <code>models</code> is undefined or a global variable while I assume that you want to wrap <code>model</code> into <code>nn.DataParallel</code>.<br>
Are you also calling <code>to(device)</code> somewhere inside the model’s <code>__init__</code> or <code>forward</code> method? If so, remove it and let <code>nn.DataParallel</code> move the modules to the right device.</p>

sujeongEOM: <p>Sorry, it was a mistypo.<br>
I don’t call to(device) inside the model’s <strong>init</strong>.</p>

ptrblck: <p>Could you post the missing model definitions as well as the input shapes so that I could execute and debug the code?</p>

sujeongEOM: <p>This is the whole model codes.</p>
<pre><code class="lang-auto">import os
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import torch

"""
https://towardsdev.com/implement-resnet-with-pytorch-a9fb40a77448
making various layer ResNet -&gt; 18, 34, 50, 101, 152
"""


def _padding(downsample, kernel_size):
    """Compute required padding"""
    padding = max(0, int(np.floor((kernel_size - downsample + 1) / 2)))
    return padding


def _downsample(n_samples_in, n_samples_out):
    """Compute downsample rate"""
    downsample = int(n_samples_in // n_samples_out)
    if downsample &lt; 1:
        raise ValueError("Number of samples should always decrease")
    if n_samples_in % n_samples_out != 0:
        raise ValueError("Number of samples for two consecutive blocks "
                         "should always decrease by an integer factor.")
    return downsample

class ResBlock1d(nn.Module):
    """Residual network unit for unidimensional signals."""

    def __init__(self, n_filters_in, n_filters_out, downsample, kernel_size, dropout_rate):
        if kernel_size % 2 == 0:
            raise ValueError("The current implementation only support odd values for `kernel_size`.")
        super(ResBlock1d, self).__init__()
        # Forward path
        padding = _padding(1, kernel_size)
        self.conv1 = nn.Conv1d(n_filters_in, n_filters_out, kernel_size, padding=padding, bias=False)
        self.bn1 = nn.BatchNorm1d(n_filters_out)
        self.relu = nn.ReLU()
        self.dropout1 = nn.Dropout(dropout_rate)
        padding = _padding(downsample, kernel_size)
        self.conv2 = nn.Conv1d(n_filters_out, n_filters_out, kernel_size,
                               stride=downsample, padding=padding, bias=False)
        self.bn2 = nn.BatchNorm1d(n_filters_out)
        self.dropout2 = nn.Dropout(dropout_rate)

        # Skip connection
        skip_connection_layers = []
        # Deal with downsampling
        if downsample &gt; 1:
            maxpool = nn.MaxPool1d(downsample, stride=downsample)
            skip_connection_layers += [maxpool]
        # Deal with n_filters dimension increase
        if n_filters_in != n_filters_out:
            conv1x1 = nn.Conv1d(n_filters_in, n_filters_out, 1, bias=False)
            skip_connection_layers += [conv1x1]
        # Build skip conection layer
        if skip_connection_layers:
            self.skip_connection = nn.Sequential(*skip_connection_layers)
        else:
            self.skip_connection = None

    def forward(self, x, y):
        """Residual unit."""
        if self.skip_connection is not None:
            y = self.skip_connection(y)
        else:
            y = y
        # 1st layer
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.dropout1(x)

        # 2nd layer
        x = self.conv2(x)
        x += y  # Sum skip connection and main connection
        y = x
        x = self.bn2(x)
        x = self.relu(x)
        x = self.dropout2(x)
        return x, y

        
class ResBottleneckBlock1d(nn.Module):
    """Residual network unit for unidimensional signals."""

    def __init__(self, n_filters_in, n_filters_out, downsample, kernel_size, dropout_rate):
        if kernel_size % 2 == 0:
            raise ValueError("The current implementation only support odd values for `kernel_size`.")
        super(ResBottleneckBlock1d, self).__init__()
        # Forward path
        padding = _padding(1, kernel_size)
        self.conv1 = nn.Conv1d(n_filters_in, n_filters_out//4, kernel_size, padding=padding, bias=False)
        self.bn1 = nn.BatchNorm1d(n_filters_out//4)
        self.relu = nn.ReLU()
        self.dropout1 = nn.Dropout(dropout_rate)

        padding = _padding(downsample, kernel_size)
        self.conv2 = nn.Conv1d(n_filters_out//4, n_filters_out//4, kernel_size,
                               stride=downsample, padding=padding, bias=False)
        self.bn2 = nn.BatchNorm1d(n_filters_out//4)
        self.dropout2 = nn.Dropout(dropout_rate)

        padding = _padding(1, kernel_size)
        self.conv3 = nn.Conv1d(n_filters_out//4, n_filters_out, kernel_size, padding=padding, bias=False)
        self.bn3 = nn.BatchNorm1d(n_filters_out)
        self.dropout3 = nn.Dropout(dropout_rate)

        # Skip connection
        skip_connection_layers = []
        # Deal with downsampling
        if downsample &gt; 1:
            maxpool = nn.MaxPool1d(downsample, stride=downsample)
            skip_connection_layers += [maxpool]
        # Deal with n_filters dimension increase
        if n_filters_in != n_filters_out:
            conv1x1 = nn.Conv1d(n_filters_in, n_filters_out, 1, bias=False)
            skip_connection_layers += [conv1x1]
        # Build skip conection layer
        if skip_connection_layers:
            self.skip_connection = nn.Sequential(*skip_connection_layers)
        else:
            self.skip_connection = None

    def forward(self, x, y):
        """Residual unit."""
        if self.skip_connection is not None:
            y = self.skip_connection(y)
        else:
            y = y
        # 1st layer
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.dropout1(x)

        # 2nd layer
        x = self.conv2(x)
        x = self.bn2(x)
        x = self.relu(x)
        x = self.dropout2(x)

        # 3rd layer
        x = self.conv3(x)
        x += y  # Sum skip connection and main connection
        y = x
        x = self.bn3(x)
        x = self.relu(x)
        x = self.dropout3(x)
        return x, y


class ResNet1d(nn.Module):
    """Residual network for unidimensional signals.
    Parameters
    ----------
    input_dim : tuple
        Input dimensions. Tuple containing dimensions for the neural network
        input tensor. Should be like: ``(n_filters, n_samples)``.
    blocks_dim : list of tuples
        Dimensions of residual blocks.  The i-th tuple should contain the dimensions
        of the output (i-1)-th residual block and the input to the i-th residual
        block. Each tuple shoud be like: ``(n_filters, n_samples)``. `n_samples`
        for two consecutive samples should always decrease by an integer factor.
    dropout_rate: float [0, 1), optional
        Dropout rate used in all Dropout layers. Default is 0.8
    kernel_size: int, optional
        Kernel size for convolutional layers. The current implementation
        only supports odd kernel sizes. Default is 17.
    References
    ----------
    .. [1] K. He, X. Zhang, S. Ren, and J. Sun, "Identity Mappings in Deep Residual Networks,"
           arXiv:1603.05027, Mar. 2016. https://arxiv.org/pdf/1603.05027.pdf.
    .. [2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in 2016 IEEE Conference
           on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778. https://arxiv.org/pdf/1512.03385.pdf
    """

    def __init__(self, input_dim, blocks_dim, resblock_repeat, n_classes, kernel_size=17, dropout_rate=0.8):
        super(ResNet1d, self).__init__()
        # First layers
        n_filters_in, n_filters_out = input_dim[0], blocks_dim[0][0] #8, 64
        n_samples_in, n_samples_out = input_dim[1], blocks_dim[0][1]
        downsample = _downsample(n_samples_in, n_samples_out)
        padding = _padding(downsample, kernel_size)
        self.conv1 = nn.Conv1d(n_filters_in, n_filters_out, kernel_size, bias=False,
                               stride=downsample, padding=padding)
        self.bn1 = nn.BatchNorm1d(n_filters_out)
        self.relu = nn.ReLU()

        # Residual block layers
        self.res_blocks = []
        for i, (n_filters, n_samples) in enumerate(blocks_dim):
            n_filters_in, n_filters_out = n_filters_out, n_filters
            n_samples_in, n_samples_out = n_samples_out, n_samples
            downsample = _downsample(n_samples_in, n_samples_out)
            resblk1d = ResBottleneckBlock1d(n_filters_in, n_filters_out, downsample, kernel_size, dropout_rate)
            self.add_module(f'resblock1d_{i}_1', resblk1d)
            for n in range(1, resblock_repeat[i]):
                self.add_module(f'resblock1d_{i}_{n+1}', ResBottleneckBlock1d(n_filters_out, n_filters_out, downsample, kernel_size, dropout_rate))
            self.res_blocks += [resblk1d]

        n_filters_last, n_samples_last = blocks_dim[-1]
        last_layer_dim = n_filters_last * n_samples_last #2048*20
        self.lin = nn.Linear(last_layer_dim, n_classes+1)
        self.n_blk = len(blocks_dim)

    def forward(self, x):
        """Implement ResNet1d forward propagation"""
        # First layers
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)

        # Residual blocks
        y = x
        for blk in self.res_blocks:
            x, y = blk(x, y)

        # Flatten array
        x = x.view(x.size(0), -1) #shape == [batch_size, 6400]
        
        # Fully conected layer
        x = self.lin(x)
        
        mu, sigma = torch.split(x, 1, dim=-1)
        sigma = F.softplus(sigma) + 1e-6 #log(1+exp(sigma))
        
        return mu, sigma


class ResNetDeepEnsembles1d(nn.Module):
    """Residual network for unidimensional signals.
    Parameters
    ----------
    input_dim : tuple
        Input dimensions. Tuple containing dimensions for the neural network
        input tensor. Should be like: ``(n_filters, n_samples)``.
    blocks_dim : list of tuples
        Dimensions of residual blocks.  The i-th tuple should contain the dimensions
        of the output (i-1)-th residual block and the input to the i-th residual
        block. Each tuple shoud be like: ``(n_filters, n_samples)``. `n_samples`
        for two consecutive samples should always decrease by an integer factor.
    block_repeat : list
        Number of repeats in each block. 
    num_models : int
        Number of ensembles. Default: 5
    dropout_rate: float [0, 1), optional
        Dropout rate used in all Dropout layers. Default is 0.8
    kernel_size: int, optional
        Kernel size for convolutional layers. The current implementation
        only supports odd kernel sizes. Default is 17.
    References
    ----------
    .. [1] K. He, X. Zhang, S. Ren, and J. Sun, "Identity Mappings in Deep Residual Networks,"
           arXiv:1603.05027, Mar. 2016. https://arxiv.org/pdf/1603.05027.pdf.
    .. [2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in 2016 IEEE Conference
           on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778. https://arxiv.org/pdf/1512.03385.pdf
    """

    def __init__(self, input_dim, blocks_dim, resblock_repeat, num_models, n_classes, kernel_size=17, dropout_rate=0.8):
        super(ResNetDeepEnsembles1d, self).__init__()
        self.num_models = num_models
        # Ensembles
        for i in range(self.num_models):
            model = ResNet1d(input_dim=input_dim,
                        blocks_dim=blocks_dim, 
                        resblock_repeat=resblock_repeat, 
                        n_classes=n_classes, 
                        kernel_size=kernel_size, 
                        dropout_rate=dropout_rate)
            setattr(self, f'model_{str(i)}',  model)

    def forward(self, x):
        mus, sigmas = [], []
        for i in range(self.num_models):
            model = getattr(self, f'model_{str(i)}')
            mu, sigma = model(x)
            mus.append(mu)
            sigmas.append(sigma)

        mus = torch.stack(mus)
        mu = mus.mean(dim=0)
        sigmas = torch.stack(sigmas)
        sigma = (sigmas + mus.pow(2)).mean(dim=0) - mu.pow(2)

        return mu, sigma
</code></pre>
<p>Most parts of script</p>
<pre><code class="lang-auto">import json
import torch
import os
from tqdm import tqdm
from models.resnet import ResNet1d, ResNetDeepEnsembles1d
from data.CustomDataset import CustomDataset
import torch.optim as optim
import numpy as np

# Deep Ensembles a.k.a ensembles NLL
def compute_loss(ages, pred_ages, pred_sigma, weights): 
  loss = (0.5*torch.log(pred_sigma.flatten()) + 0.5*((ages.flatten() - pred_ages.flatten()).pow(2))/pred_sigma.flatten()) + 1e-6
  weight_loss = torch.sum(weights.flatten() * loss)
  return weight_loss


if __name__ == "__main__":
    import pandas as pd
    import argparse
    import yaml
    from torch.utils.data import DataLoader
    from warnings import warn
    import wandb
    import torch.nn as nn

    torch.manual_seed(42)
    
    # Set device
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    
    tqdm.write("Building data loaders...")
    # Get csv data
    # train
    train_df = pd.read_csv(data['path_to_train_csv'])
    train_ages = np.array(train_df[data['age_col']])
    # valid
    valid_df = pd.read_csv(data['path_to_valid_csv'])
    valid_ages = np.array(valid_df[data['age_col']])

    # weights; must be done all together
    whole_ages = np.append(train_ages, valid_ages)
    weights = compute_weights(whole_ages)

    # Dataset and Dataloader
    train_dataset = CustomDataset(train_ages, weights[:len(train_ages)], data['path_to_train_traces'])
    train_loader = DataLoader(dataset=train_dataset, num_workers=1, batch_size=128, shuffle=True, drop_last=False)
    valid_dataset = CustomDataset(valid_ages, weights[len(train_ages):], data['path_to_valid_traces'])
    valid_loader = DataLoader(dataset=valid_dataset, num_workers=1, batch_size=128, shuffle=False, drop_last=False)
    tqdm.write("Done!")

    tqdm.write("Define model...")
    N_LEADS = 8  # the 8 leads
    N_CLASSES = 1  # just the age
    net_filter_size = [256, 512, 1024, 2048] 
    net_seq_length = [5120, 1280, 320, 80]
    resblock_repeat = [3, 4, 6, 3]
    num_models = 5

    resnet = ResNetDeepEnsembles1d(input_dim=(N_LEADS, 5120),
                blocks_dim=list(zip(net_filter_size, net_seq_length)),
                resblock_repeat=resblock_repeat], 
                num_models=num_models, 
                n_classes=N_CLASSES,
                kernel_size=17,
                dropout_rate=0.8)     
    tqdm.write("Done!")

    tqdm.write("Define optimizer...")
    optimizers = []
    for i in range(num_models ):
        model = getattr(resnet, f'model_{str(i)}')
        optimizers.append(optim.Adam(model.parameters(), 0.001))
    tqdm.write("Done!")

    tqdm.write("Define scheduler...")

    schedulers = []
    for i in range(num_models ):
        model = getattr(resnet, f'model_{str(i)}')
        schedulers.append(optim.lr_scheduler.ReduceLROnPlateau(optimizers[i], patience=7,
                                                     min_lr=0.1 * 0.0000001,
                                                     factor=0.1))
    tqdm.write("Done!")
    
    tqdm.write("Training...")
    start_epoch = 0
    best_loss = np.full(num_models, np.Inf)
    history = pd.DataFrame(columns=['epoch', 'train_loss', 'valid_loss', 'lr',
                                    'weighted_rmse', 'weighted_mae', 'rmse', 'mse'])

    for ep in range(start_epoch, 50):
        train_total_loss = np.zeros(num_models)
        train_n_entries = np.zeros(num_models)
        train_desc = "Epoch {:2d}: train - Model {:2d} Loss: {:.6f}"
        train_bar = tqdm(initial=0, leave=True, total=len(train_loader), 
                        desc=train_desc.format(ep, 0, 0), position=0)

        for traces, ages, weights in train_loader:
            traces = traces.transpose(1, 2).float()
            for i in range(num_models):
                model = getattr(resnet, f'model_{str(i)}')
                #model.train()
        
                model = nn.DataParallel(model)
                model.to(device)
                traces, ages, weights = traces.to(device), ages.to(device), weights.to(device)
                
                model.zero_grad()

                pred_ages, pred_sigma = model(traces)
                loss = compute_loss(ages, pred_ages, pred_sigma, weights)

                loss.backward()

                optimizers[i].step()

                bs = len(traces)
                train_total_loss[i] += loss.detach().cpu().numpy()
                train_n_entries[i] += bs

                train_bar.desc = train_desc.format(ep, i+1, train_total_loss[i] / train_n_entries[i])
                train_bar.update(1)
        print(train_total_loss, train_n_entries)
        train_loss = train_total_loss / train_n_entries
</code></pre>
<p>The input shape is (N_batch, 5120, 8).</p>
<p>Thank you.</p>

ptrblck: <p>Could you make the script executable by defining the input arguments etc., please?</p>

sujeongEOM: <p>Sorry, I updated the arguments. Think it’s executable. Is it okay?</p>

ptrblck: <p>Yes, after removing some undefined/unneeded parts, the code is executable, thanks.<br>
The error is raised in <code>self.res_blocks</code> as you are defining it as a plain Python <code>list</code> which is not properly registering these modules in the parent.</p>
<pre><code class="lang-python">            resblk1d = ResBottleneckBlock1d(n_filters_in, n_filters_out, downsample, kernel_size, dropout_rate)
            self.add_module(f'resblock1d_{i}_1', resblk1d)
            for n in range(1, resblock_repeat[i]):
                self.add_module(f'resblock1d_{i}_{n+1}', ResBottleneckBlock1d(n_filters_out, n_filters_out, downsample, kernel_size, dropout_rate))
            self.res_blocks += [resblk1d]
</code></pre>
<p>You would have to use an <code>nn.ModuleList</code> instead to register each submodule:</p>
<pre><code class="lang-python">    ...
    self.res_blocks += [resblk1d]

self.res_blocks = nn.ModuleList(self.res_blocks)
</code></pre>
<p>which would fix the error.</p>
<p>However, check the logic of this code snippet again as it seems you are registering new modules via:</p>
<pre><code class="lang-python">self.add_module(f'resblock1d_{i}_{n+1}', ResBottleneckBlock1d(n_filters_out, n_filters_out, downsample, kernel_size, dropout_rate))
</code></pre>
<p>but add the same <code>resblk1d</code> to the <code>list</code>, which sounds wrong.</p>

sujeongEOM: <p>Hi, thanks to your advice, I fixed the plain Python <code>list</code> error!</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://blog.d-and-j.net/deep-learning/2021/04/23/pytorch-add_module.html">
  <header class="source">
      <img src="https://blog.d-and-j.net/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://blog.d-and-j.net/deep-learning/2021/04/23/pytorch-add_module.html" target="_blank" rel="noopener nofollow ugc">blog.d-and-j.net</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://blog.d-and-j.net/deep-learning/2021/04/23/pytorch-add_module.html" target="_blank" rel="noopener nofollow ugc">The PyTorch add_module() function › Julian's musings</a></h3>

  <p>This is a space for random thoughts about mathematics, deep learning, teaching, mathematical pedagogy, computing, and other things that may be topical.
</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Following this blog. I found out the proper usage of add_module.</p>
<pre><code class="lang-auto">        # Residual block layers
        self.res_blocks = []
        for i, (n_filters, n_samples) in enumerate(blocks_dim):
            n_filters_in, n_filters_out = n_filters_out, n_filters
            n_samples_in, n_samples_out = n_samples_out, n_samples
            downsample = _downsample(n_samples_in, n_samples_out)
            resblk1d = ResBottleneckBlock1d(n_filters_in, n_filters_out, downsample, kernel_size, dropout_rate)
            self.add_module(f'resblock1d_{i}_1', resblk1d)
            self.res_blocks += [resblk1d]
            for n in range(1, resblock_repeat[i]):
                downsample = 1
                resblk1d = ResBottleneckBlock1d(n_filters_out, n_filters_out, downsample, kernel_size, dropout_rate)
                self.add_module(f'resblock1d_{i}_{n+1}', resblk1d)
                self.res_blocks += [resblk1d]

        n_filters_last, n_samples_last = blocks_dim[-1]
        last_layer_dim = n_filters_last * n_samples_last #2048*20
        self.lin = nn.Linear(last_layer_dim, n_classes+1)
        self.n_blk = len(blocks_dim)
</code></pre>
<p>I fixed it this way! but still getting same error <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You didn’t fix the <code>self.res_blocks</code> usage as it’s still a <code>list</code>.<br>
Check my previous post and use the <code>nn.ModuleList</code> as indicated in the posted code snippet.</p>

sujeongEOM: <p>Hi, related this issue, i’ve made this architecture.</p>
<pre><code class="lang-auto">def _padding(downsample, kernel_size):
    """Compute required padding"""
    padding = max(0, int(np.floor((kernel_size - downsample + 1) / 2)))
    return padding


def _downsample(n_samples_in, n_samples_out):
    """Compute downsample rate"""
    downsample = int(n_samples_in // n_samples_out)
    if downsample &lt; 1:
        raise ValueError("Number of samples should always decrease")
    if n_samples_in % n_samples_out != 0:
        raise ValueError("Number of samples for two consecutive blocks "
                         "should always decrease by an integer factor.")
    return downsample

class ResBlock1d(nn.Module):
    """Residual network unit for unidimensional signals."""

    def __init__(self, n_filters_in, n_filters_out, downsample, kernel_size, dropout_rate):
        if kernel_size % 2 == 0:
            raise ValueError("The current implementation only support odd values for `kernel_size`.")
        super(ResBlock1d, self).__init__()
        # Forward path
        padding = _padding(1, kernel_size)
        self.conv1 = nn.Conv1d(n_filters_in, n_filters_out, kernel_size, padding=padding, bias=False)
        self.bn1 = nn.BatchNorm1d(n_filters_out)
        self.relu = nn.ReLU()
        self.dropout1 = nn.Dropout(dropout_rate)
        padding = _padding(downsample, kernel_size)
        self.conv2 = nn.Conv1d(n_filters_out, n_filters_out, kernel_size,
                               stride=downsample, padding=padding, bias=False)
        self.bn2 = nn.BatchNorm1d(n_filters_out)
        self.dropout2 = nn.Dropout(dropout_rate)

        # Skip connection
        skip_connection_layers = []
        # Deal with downsampling
        if downsample &gt; 1:
            maxpool = nn.MaxPool1d(downsample, stride=downsample)
            skip_connection_layers += [maxpool]
        # Deal with n_filters dimension increase
        if n_filters_in != n_filters_out:
            conv1x1 = nn.Conv1d(n_filters_in, n_filters_out, 1, bias=False)
            skip_connection_layers += [conv1x1]
        # Build skip conection layer
        if skip_connection_layers:
            self.skip_connection = nn.Sequential(*skip_connection_layers)
        else:
            self.skip_connection = None

    def forward(self, x, y):
        """Residual unit."""
        if self.skip_connection is not None:
            y = self.skip_connection(y)
        else:
            y = y
        # 1st layer
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.dropout1(x)

        # 2nd layer
        x = self.conv2(x)
        x += y  # Sum skip connection and main connection
        y = x
        x = self.bn2(x)
        x = self.relu(x)
        x = self.dropout2(x)
        return x, y


class ResNet1d_mse(nn.Module):
    """Residual network for unidimensional signals.
    Parameters
    ----------
    input_dim : tuple
        Input dimensions. Tuple containing dimensions for the neural network
        input tensor. Should be like: ``(n_filters, n_samples)``.
    blocks_dim : list of tuples
        Dimensions of residual blocks.  The i-th tuple should contain the dimensions
        of the output (i-1)-th residual block and the input to the i-th residual
        block. Each tuple shoud be like: ``(n_filters, n_samples)``. `n_samples`
        for two consecutive samples should always decrease by an integer factor.
    dropout_rate: float [0, 1), optional
        Dropout rate used in all Dropout layers. Default is 0.8
    kernel_size: int, optional
        Kernel size for convolutional layers. The current implementation
        only supports odd kernel sizes. Default is 17.
    References
    ----------
    .. [1] K. He, X. Zhang, S. Ren, and J. Sun, "Identity Mappings in Deep Residual Networks,"
           arXiv:1603.05027, Mar. 2016. https://arxiv.org/pdf/1603.05027.pdf.
    .. [2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in 2016 IEEE Conference
           on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778. https://arxiv.org/pdf/1512.03385.pdf
    """

    def __init__(self, input_dim, blocks_dim, n_classes, kernel_size=17, dropout_rate=0.8):
        super(ResNet1d_mse, self).__init__()
        # First layers
        n_filters_in, n_filters_out = input_dim[0], blocks_dim[0][0]
        n_samples_in, n_samples_out = input_dim[1], blocks_dim[0][1]
        downsample = _downsample(n_samples_in, n_samples_out)
        padding = _padding(downsample, kernel_size)
        self.conv1 = nn.Conv1d(n_filters_in, n_filters_out, kernel_size, bias=False,
                               stride=downsample, padding=padding)
        self.bn1 = nn.BatchNorm1d(n_filters_out)
        self.relu = nn.ReLU()


        # Residual block layers
        **self.res_blocks = nn.ModuleList()**
        for i, (n_filters, n_samples) in enumerate(blocks_dim):
            n_filters_in, n_filters_out = n_filters_out, n_filters
            n_samples_in, n_samples_out = n_samples_out, n_samples
            downsample = _downsample(n_samples_in, n_samples_out)
            resblk1d = ResBlock1d(n_filters_in, n_filters_out, downsample, kernel_size, dropout_rate)
            **self.add_module('resblock1d_{0}'.format(i), resblk1d)**
            self.res_blocks += [resblk1d]

        # Linear layer
        n_filters_last, n_samples_last = blocks_dim[-1]
        last_layer_dim = n_filters_last * n_samples_last
        self.lin = nn.Linear(last_layer_dim, n_classes)
        self.n_blk = len(blocks_dim)

    def forward(self, x):
        """Implement ResNet1d forward propagation"""
        # First layers
        x = self.conv1(x)
        x = self.bn1(x)

        # Residual blocks
        y = x
        for blk in self.res_blocks:
            x, y = blk(x, y)

        # Flatten array
        x = x.view(x.size(0), -1)

        # Fully conected layer
        x = self.lin(x)
        return x

</code></pre>
<p>But as you can see in ResNet1d_mse, when layering residual blocks, i’ve used both nn.ModuleList() and add_module. Therefore, my model architecture become this.</p>
<p>ResNet1d_mse(<br>
(conv1): Conv1d(8, 64, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(res_blocks): ModuleList(<br>
(0): ResBlock1d(<br>
(conv1): Conv1d(64, 64, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(64, 64, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn2): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
)<br>
(1): ResBlock1d(<br>
(conv1): Conv1d(64, 128, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(128, 128, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(64, 128, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
(2): ResBlock1d(<br>
(conv1): Conv1d(128, 196, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(196, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(196, 196, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(196, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(128, 196, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
(3): ResBlock1d(<br>
(conv1): Conv1d(196, 256, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(256, 256, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(196, 256, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
(4): ResBlock1d(<br>
(conv1): Conv1d(256, 320, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(320, 320, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(256, 320, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
)<br>
(resblock1d_0): ResBlock1d(<br>
(conv1): Conv1d(64, 64, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(64, 64, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn2): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
)<br>
(resblock1d_1): ResBlock1d(<br>
(conv1): Conv1d(64, 128, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(128, 128, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(64, 128, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
(resblock1d_2): ResBlock1d(<br>
(conv1): Conv1d(128, 196, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(196, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(196, 196, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(196, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(128, 196, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
(resblock1d_3): ResBlock1d(<br>
(conv1): Conv1d(196, 256, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(256, 256, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(196, 256, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
(resblock1d_4): ResBlock1d(<br>
(conv1): Conv1d(256, 320, kernel_size=(17,), stride=(1,), padding=(8,), bias=False)<br>
(bn1): BatchNorm1d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(relu): ReLU()<br>
(dropout1): Dropout(p=0.5, inplace=False)<br>
(conv2): Conv1d(320, 320, kernel_size=(17,), stride=(5,), padding=(6,), bias=False)<br>
(bn2): BatchNorm1d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(dropout2): Dropout(p=0.5, inplace=False)<br>
(skip_connection): Sequential(<br>
(0): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)<br>
(1): Conv1d(256, 320, kernel_size=(1,), stride=(1,), bias=False)<br>
)<br>
)<br>
(lin): Linear(in_features=2560, out_features=1, bias=True)<br>
)</p>
<p>With module list at the front and individual blocks at the back again. I’ve already trained huge dataset to this model, but i just found out. I wonder, how does this work, is this same with only individual blocks architecture?</p>
<h2>
<a name="this-is-summary-of-the-architecutre-from-torchsummary-1" class="anchor" href="#this-is-summary-of-the-architecutre-from-torchsummary-1"></a>This is summary of the architecutre from torchsummary.</h2>
<pre><code>    Layer (type)               Output Shape         Param #
</code></pre>
<h1>
<a name="conv1d-1-1-64-5000-8704-batchnorm1d-2-1-64-5000-128-conv1d-3-1-64-5000-69632-conv1d-4-1-64-5000-69632-batchnorm1d-5-1-64-5000-128-batchnorm1d-6-1-64-5000-128-relu-7-1-64-5000-0-relu-8-1-64-5000-0-dropout-9-1-64-5000-0-dropout-10-1-64-5000-0-conv1d-11-1-64-5000-69632-conv1d-12-1-64-5000-69632-batchnorm1d-13-1-64-5000-128-batchnorm1d-14-1-64-5000-128-relu-15-1-64-5000-0-relu-16-1-64-5000-0-dropout-17-1-64-5000-0-dropout-18-1-64-5000-0-resblock1d-19-1-64-5000-1-64-5000-0-resblock1d-20-1-64-5000-1-64-5000-0-maxpool1d-21-1-64-1000-0-maxpool1d-22-1-64-1000-0-conv1d-23-1-128-1000-8192-conv1d-24-1-128-1000-8192-conv1d-25-1-128-5000-139264-conv1d-26-1-128-5000-139264-batchnorm1d-27-1-128-5000-256-batchnorm1d-28-1-128-5000-256-relu-29-1-128-5000-0-relu-30-1-128-5000-0-dropout-31-1-128-5000-0-dropout-32-1-128-5000-0-conv1d-33-1-128-1000-278528-conv1d-34-1-128-1000-278528-batchnorm1d-35-1-128-1000-256-batchnorm1d-36-1-128-1000-256-relu-37-1-128-1000-0-relu-38-1-128-1000-0-dropout-39-1-128-1000-0-dropout-40-1-128-1000-0-resblock1d-41-1-128-1000-1-128-1000-0-resblock1d-42-1-128-1000-1-128-1000-0-maxpool1d-43-1-128-200-0-maxpool1d-44-1-128-200-0-conv1d-45-1-196-200-25088-conv1d-46-1-196-200-25088-conv1d-47-1-196-1000-426496-conv1d-48-1-196-1000-426496-batchnorm1d-49-1-196-1000-392-batchnorm1d-50-1-196-1000-392-relu-51-1-196-1000-0-relu-52-1-196-1000-0-dropout-53-1-196-1000-0-dropout-54-1-196-1000-0-conv1d-55-1-196-200-653072-conv1d-56-1-196-200-653072-batchnorm1d-57-1-196-200-392-batchnorm1d-58-1-196-200-392-relu-59-1-196-200-0-relu-60-1-196-200-0-dropout-61-1-196-200-0-dropout-62-1-196-200-0-resblock1d-63-1-196-200-1-196-200-0-resblock1d-64-1-196-200-1-196-200-0-maxpool1d-65-1-196-40-0-maxpool1d-66-1-196-40-0-conv1d-67-1-256-40-50176-conv1d-68-1-256-40-50176-conv1d-69-1-256-200-852992-conv1d-70-1-256-200-852992-batchnorm1d-71-1-256-200-512-batchnorm1d-72-1-256-200-512-relu-73-1-256-200-0-relu-74-1-256-200-0-dropout-75-1-256-200-0-dropout-76-1-256-200-0-conv1d-77-1-256-40-1114112-conv1d-78-1-256-40-1114112-batchnorm1d-79-1-256-40-512-batchnorm1d-80-1-256-40-512-relu-81-1-256-40-0-relu-82-1-256-40-0-dropout-83-1-256-40-0-dropout-84-1-256-40-0-resblock1d-85-1-256-40-1-256-40-0-resblock1d-86-1-256-40-1-256-40-0-maxpool1d-87-1-256-8-0-maxpool1d-88-1-256-8-0-conv1d-89-1-320-8-81920-conv1d-90-1-320-8-81920-conv1d-91-1-320-40-1392640-conv1d-92-1-320-40-1392640-batchnorm1d-93-1-320-40-640-batchnorm1d-94-1-320-40-640-relu-95-1-320-40-0-relu-96-1-320-40-0-dropout-97-1-320-40-0-dropout-98-1-320-40-0-conv1d-99-1-320-8-1740800-conv1d-100-1-320-8-1740800-batchnorm1d-101-1-320-8-640-batchnorm1d-102-1-320-8-640-relu-103-1-320-8-0-relu-104-1-320-8-0-dropout-105-1-320-8-0-dropout-106-1-320-8-0-resblock1d-107-1-320-8-1-320-8-0-resblock1d-108-1-320-8-1-320-8-0-linear-109-1-1-2561-2" class="anchor" href="#conv1d-1-1-64-5000-8704-batchnorm1d-2-1-64-5000-128-conv1d-3-1-64-5000-69632-conv1d-4-1-64-5000-69632-batchnorm1d-5-1-64-5000-128-batchnorm1d-6-1-64-5000-128-relu-7-1-64-5000-0-relu-8-1-64-5000-0-dropout-9-1-64-5000-0-dropout-10-1-64-5000-0-conv1d-11-1-64-5000-69632-conv1d-12-1-64-5000-69632-batchnorm1d-13-1-64-5000-128-batchnorm1d-14-1-64-5000-128-relu-15-1-64-5000-0-relu-16-1-64-5000-0-dropout-17-1-64-5000-0-dropout-18-1-64-5000-0-resblock1d-19-1-64-5000-1-64-5000-0-resblock1d-20-1-64-5000-1-64-5000-0-maxpool1d-21-1-64-1000-0-maxpool1d-22-1-64-1000-0-conv1d-23-1-128-1000-8192-conv1d-24-1-128-1000-8192-conv1d-25-1-128-5000-139264-conv1d-26-1-128-5000-139264-batchnorm1d-27-1-128-5000-256-batchnorm1d-28-1-128-5000-256-relu-29-1-128-5000-0-relu-30-1-128-5000-0-dropout-31-1-128-5000-0-dropout-32-1-128-5000-0-conv1d-33-1-128-1000-278528-conv1d-34-1-128-1000-278528-batchnorm1d-35-1-128-1000-256-batchnorm1d-36-1-128-1000-256-relu-37-1-128-1000-0-relu-38-1-128-1000-0-dropout-39-1-128-1000-0-dropout-40-1-128-1000-0-resblock1d-41-1-128-1000-1-128-1000-0-resblock1d-42-1-128-1000-1-128-1000-0-maxpool1d-43-1-128-200-0-maxpool1d-44-1-128-200-0-conv1d-45-1-196-200-25088-conv1d-46-1-196-200-25088-conv1d-47-1-196-1000-426496-conv1d-48-1-196-1000-426496-batchnorm1d-49-1-196-1000-392-batchnorm1d-50-1-196-1000-392-relu-51-1-196-1000-0-relu-52-1-196-1000-0-dropout-53-1-196-1000-0-dropout-54-1-196-1000-0-conv1d-55-1-196-200-653072-conv1d-56-1-196-200-653072-batchnorm1d-57-1-196-200-392-batchnorm1d-58-1-196-200-392-relu-59-1-196-200-0-relu-60-1-196-200-0-dropout-61-1-196-200-0-dropout-62-1-196-200-0-resblock1d-63-1-196-200-1-196-200-0-resblock1d-64-1-196-200-1-196-200-0-maxpool1d-65-1-196-40-0-maxpool1d-66-1-196-40-0-conv1d-67-1-256-40-50176-conv1d-68-1-256-40-50176-conv1d-69-1-256-200-852992-conv1d-70-1-256-200-852992-batchnorm1d-71-1-256-200-512-batchnorm1d-72-1-256-200-512-relu-73-1-256-200-0-relu-74-1-256-200-0-dropout-75-1-256-200-0-dropout-76-1-256-200-0-conv1d-77-1-256-40-1114112-conv1d-78-1-256-40-1114112-batchnorm1d-79-1-256-40-512-batchnorm1d-80-1-256-40-512-relu-81-1-256-40-0-relu-82-1-256-40-0-dropout-83-1-256-40-0-dropout-84-1-256-40-0-resblock1d-85-1-256-40-1-256-40-0-resblock1d-86-1-256-40-1-256-40-0-maxpool1d-87-1-256-8-0-maxpool1d-88-1-256-8-0-conv1d-89-1-320-8-81920-conv1d-90-1-320-8-81920-conv1d-91-1-320-40-1392640-conv1d-92-1-320-40-1392640-batchnorm1d-93-1-320-40-640-batchnorm1d-94-1-320-40-640-relu-95-1-320-40-0-relu-96-1-320-40-0-dropout-97-1-320-40-0-dropout-98-1-320-40-0-conv1d-99-1-320-8-1740800-conv1d-100-1-320-8-1740800-batchnorm1d-101-1-320-8-640-batchnorm1d-102-1-320-8-640-relu-103-1-320-8-0-relu-104-1-320-8-0-dropout-105-1-320-8-0-dropout-106-1-320-8-0-resblock1d-107-1-320-8-1-320-8-0-resblock1d-108-1-320-8-1-320-8-0-linear-109-1-1-2561-2"></a>================================================================<br>
Conv1d-1             [-1, 64, 5000]           8,704<br>
BatchNorm1d-2             [-1, 64, 5000]             128<br>
Conv1d-3             [-1, 64, 5000]          69,632<br>
Conv1d-4             [-1, 64, 5000]          69,632<br>
BatchNorm1d-5             [-1, 64, 5000]             128<br>
BatchNorm1d-6             [-1, 64, 5000]             128<br>
ReLU-7             [-1, 64, 5000]               0<br>
ReLU-8             [-1, 64, 5000]               0<br>
Dropout-9             [-1, 64, 5000]               0<br>
Dropout-10             [-1, 64, 5000]               0<br>
Conv1d-11             [-1, 64, 5000]          69,632<br>
Conv1d-12             [-1, 64, 5000]          69,632<br>
BatchNorm1d-13             [-1, 64, 5000]             128<br>
BatchNorm1d-14             [-1, 64, 5000]             128<br>
ReLU-15             [-1, 64, 5000]               0<br>
ReLU-16             [-1, 64, 5000]               0<br>
Dropout-17             [-1, 64, 5000]               0<br>
Dropout-18             [-1, 64, 5000]               0<br>
ResBlock1d-19  [[-1, 64, 5000], [-1, 64, 5000]]               0<br>
ResBlock1d-20  [[-1, 64, 5000], [-1, 64, 5000]]               0<br>
MaxPool1d-21             [-1, 64, 1000]               0<br>
MaxPool1d-22             [-1, 64, 1000]               0<br>
Conv1d-23            [-1, 128, 1000]           8,192<br>
Conv1d-24            [-1, 128, 1000]           8,192<br>
Conv1d-25            [-1, 128, 5000]         139,264<br>
Conv1d-26            [-1, 128, 5000]         139,264<br>
BatchNorm1d-27            [-1, 128, 5000]             256<br>
BatchNorm1d-28            [-1, 128, 5000]             256<br>
ReLU-29            [-1, 128, 5000]               0<br>
ReLU-30            [-1, 128, 5000]               0<br>
Dropout-31            [-1, 128, 5000]               0<br>
Dropout-32            [-1, 128, 5000]               0<br>
Conv1d-33            [-1, 128, 1000]         278,528<br>
Conv1d-34            [-1, 128, 1000]         278,528<br>
BatchNorm1d-35            [-1, 128, 1000]             256<br>
BatchNorm1d-36            [-1, 128, 1000]             256<br>
ReLU-37            [-1, 128, 1000]               0<br>
ReLU-38            [-1, 128, 1000]               0<br>
Dropout-39            [-1, 128, 1000]               0<br>
Dropout-40            [-1, 128, 1000]               0<br>
ResBlock1d-41  [[-1, 128, 1000], [-1, 128, 1000]]               0<br>
ResBlock1d-42  [[-1, 128, 1000], [-1, 128, 1000]]               0<br>
MaxPool1d-43             [-1, 128, 200]               0<br>
MaxPool1d-44             [-1, 128, 200]               0<br>
Conv1d-45             [-1, 196, 200]          25,088<br>
Conv1d-46             [-1, 196, 200]          25,088<br>
Conv1d-47            [-1, 196, 1000]         426,496<br>
Conv1d-48            [-1, 196, 1000]         426,496<br>
BatchNorm1d-49            [-1, 196, 1000]             392<br>
BatchNorm1d-50            [-1, 196, 1000]             392<br>
ReLU-51            [-1, 196, 1000]               0<br>
ReLU-52            [-1, 196, 1000]               0<br>
Dropout-53            [-1, 196, 1000]               0<br>
Dropout-54            [-1, 196, 1000]               0<br>
Conv1d-55             [-1, 196, 200]         653,072<br>
Conv1d-56             [-1, 196, 200]         653,072<br>
BatchNorm1d-57             [-1, 196, 200]             392<br>
BatchNorm1d-58             [-1, 196, 200]             392<br>
ReLU-59             [-1, 196, 200]               0<br>
ReLU-60             [-1, 196, 200]               0<br>
Dropout-61             [-1, 196, 200]               0<br>
Dropout-62             [-1, 196, 200]               0<br>
ResBlock1d-63  [[-1, 196, 200], [-1, 196, 200]]               0<br>
ResBlock1d-64  [[-1, 196, 200], [-1, 196, 200]]               0<br>
MaxPool1d-65              [-1, 196, 40]               0<br>
MaxPool1d-66              [-1, 196, 40]               0<br>
Conv1d-67              [-1, 256, 40]          50,176<br>
Conv1d-68              [-1, 256, 40]          50,176<br>
Conv1d-69             [-1, 256, 200]         852,992<br>
Conv1d-70             [-1, 256, 200]         852,992<br>
BatchNorm1d-71             [-1, 256, 200]             512<br>
BatchNorm1d-72             [-1, 256, 200]             512<br>
ReLU-73             [-1, 256, 200]               0<br>
ReLU-74             [-1, 256, 200]               0<br>
Dropout-75             [-1, 256, 200]               0<br>
Dropout-76             [-1, 256, 200]               0<br>
Conv1d-77              [-1, 256, 40]       1,114,112<br>
Conv1d-78              [-1, 256, 40]       1,114,112<br>
BatchNorm1d-79              [-1, 256, 40]             512<br>
BatchNorm1d-80              [-1, 256, 40]             512<br>
ReLU-81              [-1, 256, 40]               0<br>
ReLU-82              [-1, 256, 40]               0<br>
Dropout-83              [-1, 256, 40]               0<br>
Dropout-84              [-1, 256, 40]               0<br>
ResBlock1d-85  [[-1, 256, 40], [-1, 256, 40]]               0<br>
ResBlock1d-86  [[-1, 256, 40], [-1, 256, 40]]               0<br>
MaxPool1d-87               [-1, 256, 8]               0<br>
MaxPool1d-88               [-1, 256, 8]               0<br>
Conv1d-89               [-1, 320, 8]          81,920<br>
Conv1d-90               [-1, 320, 8]          81,920<br>
Conv1d-91              [-1, 320, 40]       1,392,640<br>
Conv1d-92              [-1, 320, 40]       1,392,640<br>
BatchNorm1d-93              [-1, 320, 40]             640<br>
BatchNorm1d-94              [-1, 320, 40]             640<br>
ReLU-95              [-1, 320, 40]               0<br>
ReLU-96              [-1, 320, 40]               0<br>
Dropout-97              [-1, 320, 40]               0<br>
Dropout-98              [-1, 320, 40]               0<br>
Conv1d-99               [-1, 320, 8]       1,740,800<br>
Conv1d-100               [-1, 320, 8]       1,740,800<br>
BatchNorm1d-101               [-1, 320, 8]             640<br>
BatchNorm1d-102               [-1, 320, 8]             640<br>
ReLU-103               [-1, 320, 8]               0<br>
ReLU-104               [-1, 320, 8]               0<br>
Dropout-105               [-1, 320, 8]               0<br>
Dropout-106               [-1, 320, 8]               0<br>
ResBlock1d-107  [[-1, 320, 8], [-1, 320, 8]]               0<br>
ResBlock1d-108  [[-1, 320, 8], [-1, 320, 8]]               0<br>
Linear-109                    [-1, 1]           2,561</h1>
<h2>
<a name="total-params-13824193-trainable-params-13824193-non-trainable-params-0-3" class="anchor" href="#total-params-13824193-trainable-params-13824193-non-trainable-params-0-3"></a>Total params: 13,824,193<br>
Trainable params: 13,824,193<br>
Non-trainable params: 0</h2>
<h2>
<a name="input-size-mb-015-forwardbackward-pass-size-mb-183753314-params-size-mb-5274-estimated-total-size-mb-183758603-4" class="anchor" href="#input-size-mb-015-forwardbackward-pass-size-mb-183753314-params-size-mb-5274-estimated-total-size-mb-183758603-4"></a>Input size (MB): 0.15<br>
Forward/backward pass size (MB): 1837533.14<br>
Params size (MB): 52.74<br>
Estimated Total Size (MB): 1837586.03</h2>
<p>Thanks.</p>

ptrblck: <aside class="quote no-group" data-username="sujeongEOM" data-post="11" data-topic="157684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sujeongeom/48/51844_2.png" class="avatar"> sujeongEOM:</div>
<blockquote>
<p>With module list at the front and individual blocks at the back again. I’ve already trained huge dataset to this model, but i just found out. I wonder, how does this work, is this same with only individual blocks architecture?</p>
</blockquote>
</aside>
<p>Yes, you are registering the same module using different approaches. The registered modules are the same and use “parameter sharing”, i.e. you should not see more memory usage etc.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>