Distributed Data Parallel .module attribute

username: kumar.s
cooked: <p>I am working with <code>torch.nn.parallel.DistributedDataParallel</code> class and I saw that it assigns the input <code>module</code> under <code>self.module</code>.<br>
From the source code at <code>line 599</code> in <code>torch/nn/parallel/distributed.py</code>:</p>
<pre><code class="lang-auto">self.module = module
</code></pre>
<p>so for an instance of DDP,  let’s say:</p>
<pre><code class="lang-auto">ddp = parallel.DistributedDataParallel(module=mod)
</code></pre>
<p>When I try to access <code>ddp.__dict__['module']</code> the key is not found. However I can access the attribute  by simply doing a <code>ddp.module</code>. When I print <code>ddp.__dict__.keys()</code> I get the following:</p>
<pre><code class="lang-auto">dict_keys(['training', '_parameters', '_buffers', '_non_persistent_buffers_set', '_backward_hooks', 
'_is_full_backward_hook', '_forward_hooks', '_forward_pre_hooks', '_state_dict_hooks', 
'_load_state_dict_pre_hooks', '_modules', 'is_multi_device_module', 'device_type', 'device_ids', 
'output_device', 'process_group', 'static_graph', 'dim', 'device', 'broadcast_buffers', 
'find_unused_parameters', 'require_backward_grad_sync', 'require_forward_param_sync', 
'ddp_uneven_inputs_config', 'gradient_as_bucket_view', 'parameters_to_ignore', 
'broadcast_bucket_size', 'bucket_bytes_cap', 'use_side_stream_for_tensor_copies', 
 '_module_copies', 'modules_params', 'modules_buffers', 'num_iterations', 'reducer', 'logger'])   
</code></pre>
<p>I couldn’t find in the source code of how this masking was achieved and if there is a reason to do that. But I would like to know.</p>

username: H-Huang
cooked: <p>I believe this is a nuance with Python namespaces, not exactly distributed related. Calling a method/class on an instance such as <code>instance.some_func(arg)</code> means you are calling <code>class.some_func(instance, arg)</code>. Since <code>__dict__</code> is used to access instance attributes and <code>module</code> is a class then it will not show up. You can use <code>dir()</code> instead which accesses complete list of attributes. For your example you can try running <code>print("module" in dir(ddp))</code> which will print out <code>True</code>.</p>

username: kumar.s
cooked: <p>I figured it out. The answer is in the <code>__setattr__</code> of the class <code>Module</code> (from which DDP inherits). So the <code>Module</code> class <code>__setattr__</code> removes everything from <code>__dict__</code> which is of type <code>Parameter</code> or <code>Module</code> and stores them in the <code>self._parameters</code> and <code>self._modules</code> OrderedDict.</p>
<p>Here’s the corresponding snippet from the source code:</p>
<pre><code class="lang-auto">    def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -&gt; None:
        def remove_from(*dicts_or_sets):
            for d in dicts_or_sets:
                if name in d:
                    if isinstance(d, dict):
                        del d[name]
                    else:
                        d.discard(name)
</code></pre>
<p>and it is called for attributes of type <code>Module</code> later in the same function like so:</p>
<pre><code class="lang-auto">else:
        modules = self.__dict__.get('_modules')
        if isinstance(value, Module):
            if modules is None:
                raise AttributeError(
                    "cannot assign module before Module.__init__() call")
            remove_from(self.__dict__, self._parameters, self._buffers, self._non_persistent_buffers_set)
            modules[name] = value
</code></pre>