Why self.parameters() become empty when running models?

ojipadeson: <p>My bert model raise a StopIteration error when running the code <code>next(self.parameters()).dtype</code><br>
I first check the model simply by:</p>
<pre><code class="lang-auto">print(list(self.parameters())) 
# []
# []
# [][]
# []
</code></pre>
<p>It mean parameters are empty.</p>
<p>My code is like:</p>
<pre><code class="lang-auto">class MyModel(nn.Module)
    def __init__(self, hyper_param):
        super(MyModel, self).__init__()
        self.bert = Model(hyper_param)
    def forward(batch):
        # check attributes -- (2)
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        cx = batch["cx"]
        cy = batch["cy"]
        height = batch["height"]
        # next(self.parameters()).dtype is in self.bert
        output = self.bert(input_ids, attention_mask, cx, cy, height)
        return output
model = MyBertModel()
batch = next(dataset_iter)
# check attributes -- (1)
loss = model(batch)
</code></pre>
<p>I further check all the protected attributes of the model at position (1) and (2) commented in above code, and get following result:</p>
<p>At position (1)</p>
<pre><code class="lang-auto">_backward_hooks = {OrderedDict: 0} OrderedDict()
_buffers = {OrderedDict: 0} OrderedDict()
_forward_hooks = {OrderedDict: 0} OrderedDict()
_forward_pre_hooks = {OrderedDict: 0} OrderedDict()
_is_full_backward_hook = {NoneType} None
_load_state_dict_pre_hooks = {OrderedDict: 0} OrderedDict()
_modules = {OrderedDict: 0} OrderedDict()
_non_persistent_buffers_set = {set: 0} set()
_parameters = {OrderedDict: 1} OrderedDict([('weight', Parameter containing:\ntensor([[-0.0394,  0.0065,  0.0195,  ..., -0.0082, -0.0145,  0.0046],\n        [ 0.0124, 	 0.0060,  0.0094,  ..., -0.0219,  0.0053, -0.0220],\n        [ 0.0086, -0.0022,  0.0252,  ..., -0.0312, -0.0307,  0.0214],\n        ...,\n        [ 0.0200,  0.0115,  	0.0103,  ..., -0.0153,  0.0163, -0.0371],\n        [ 0.0301, -0.0143,  0.0047,  ..., -0.0138, -0.0130,  0.0120],\n        [-0.0115,  0.0102, -0.0111,  ..., -0.0081, -	0.0122, -0.0312]],\n       device='cuda:0', requires_grad=True))])

	 'weight' = {Parameter: (1000, 768)} Parameter containing:\ntensor([[-0.0394,  0.0065,  0.0195,  ..., -0.0082, -0.0145,  0.0046],\n        [ 0.0124,  0.0060,  	0.0094,  ..., -0.0219,  0.0053, -0.0220],\n        [ 0.0086, -0.0022,  0.0252,  ..., -0.0312, -0.0307,  0.0214],\n        ...,\n        [ 0.0200,  0.0115,  0.0103,  ..., -		0.0153,  0.0163, -0.0371],\n        [ 0.0301, -0.0143,  0.0047,  ..., -0.0138, -0.0130,  0.0120],\n        [-0.0115,  0.0102, -0.0111,  ..., -0.0081, -0.0122, -		0.0312]],\n       device='cuda:0', requires_grad=True)

	 __len__ = {int} 1
_state_dict_hooks = {OrderedDict: 0} OrderedDict()
_version = {int} 1
</code></pre>
<p>At positon (1), parameters seem normal</p>
<p>And At position (2):</p>
<pre><code class="lang-auto">_backward_hooks = {OrderedDict: 0} OrderedDict()
_buffers = {OrderedDict: 0} OrderedDict()
_former_parameters = {OrderedDict: 1} OrderedDict([('weight', tensor([[-0.0394,  0.0065,  0.0195,  ..., -0.0082, -0.0145,  0.0046],\n        [ 0.0124,  0.0060,  	0.0094,  ..., -0.0219,  0.0053, -0.0220],\n        [ 0.0086, -0.0022,  0.0252,  ..., -0.0312, -0.0307,  0.0214],\n        ...,\n        [

	 'weight' = {Tensor: (1000, 768)} tensor([[-0.0394,  0.0065,  0.0195,  ..., -0.0082, -0.0145,  0.0046],\n        [ 0.0124,  0.0060,  0.0094,  ..., -0.0219,  		0.0053, -0.0220],\n        [ 0.0086, -0.0022,  0.0252,  ..., -0.0312, -0.0307,  0.0214],\n        ...,\n        [ 0.0200,  0.0115,  0.0103,  ..., -0.0153,  0.0163, -		0.0371],\n        [ 0.0301, -0.0143,  0.0047,  ..., -0.0138, -0.0130,  0.0120],\n        [-0.0115,  0.0102, -0.0111,  ..., -0.0081, -0.0122, -0.0312]],\n       		device='cuda:7', grad_fn=&lt;BroadcastBackward&gt;)

	 __len__ = {int} 1
_forward_hooks = {OrderedDict: 0} OrderedDict()
_forward_pre_hooks = {OrderedDict: 0} OrderedDict()
_is_replica = {bool} True
_is_full_backward_hook = {NoneType} None
_load_state_dict_pre_hooks = {OrderedDict: 0} OrderedDict()
_modules = {OrderedDict: 0} OrderedDict()
_non_persistent_buffers_set = {set: 0} set()
_parameters = {OrderedDict: 0} OrderedDict()
	__len__ = {int} 0
_state_dict_hooks = {OrderedDict: 0} OrderedDict()
_version = {int} 1
</code></pre>
<p>What I can find is that the <code>_parameter</code> simply disappears after the call of <code>forward()</code> method, and with an additional <code>_former_parameters</code> attribute appear, but I can’t find this attribute in <code>nn.Modules</code> source code. How to explain this?</p>

ptrblck: <p>Are you using <code>nn.DataParallel</code>? If so, I think the replica models move the parameters to <code>_former_parameters</code> to avoid conflicts with leaf nodes.<br>
However, the recommendation is to use <code>DistributedDataParallel</code> for the better performance and I haven’t taken a look at <code>nn.DataParallel</code> in a while.</p>