AttributeError: 'ModelParallelModel' object has no attribute '_forward_pre_hooks'

ProGamerGov: <p>The <code>chunks</code> input is a list of <code>nn.Sequential</code> networks from a model I have divided up to run on multiple GPUs/CPUs. The <code>device_list</code> input is a list of devices, like for example <code>['cuda:0', 'cuda:1', 'cuda:2', 'cuda:3']</code>, or <code>['cpu', 'cuda:0', 'cuda:1']</code>. Both lists will have the same number of values.</p>
<p>Unfortunately I don’t have multiple GPUs, and online services can be rather expensive for multiple GPUs. So, I want to make sure that I have things right before I try to run the code. I created the following class to run a single model with a batch size of 1 across multiple devices. Basically each device is supposed to run part of a model before passing the output to the next device.</p>
<p>I put the chunks onto their devices:</p>
<pre><code class="lang-auto">
    for i, chunk in enumerate(chunks):
        chunk.to(device_list[i])
</code></pre>
<p>And then I pass them to my class:</p>
<pre><code class="lang-auto">class ModelParallelModel(nn.Sequential):
    def __init__(self, chunks, device_list):
        super(ModelParallelModel, self)
        self.chunks = chunks
        self.device_list = device_list
        print(str(len(self.chunks)))
        print(self.device_list)

    def forward(self, input):
        for i, chunk in enumerate(chunks):
            if i &lt; len(chunks) -1:
               input = chunk(input.to(device_list[i]) ).to(device_list[i+1])
            else: 
               input = chunk(input.to(device_list[i]))
        return input
</code></pre>
<p>These lines of code are where I create the model and then try to run:</p>
<pre><code class="lang-auto"> net = ModelParallelModel(chunks, device_list)
</code></pre>
<pre><code class="lang-auto"> net(test_image)
</code></pre>
<p>Though I get this error:</p>
<pre><code class="lang-auto">4
['cuda:0', 'cuda:1', 'cuda:2', 'cuda:3']
Traceback (most recent call last):
  File "test.py", line 465, in &lt;module&gt;
    main()
  File "test.py", line 164, in main
    net(test_image)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 538, in __call__
    for hook in self._forward_pre_hooks.values():
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 591, in __getattr__
    type(self).__name__, name))
AttributeError: 'ModelParallelModel' object has no attribute '_forward_pre_hooks'
</code></pre>
<p>How do I fix this error? I can’t seem to find anything about what this error means, or how to fix it anywhere.</p>

ProGamerGov: <p>I was trying to follow this guide: <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html</a></p>

ptrblck: <p>You are missing the <code>__init__</code> call in <code>super</code>:</p>
<pre><code class="lang-python"># yours
super(ModelParallelModel, self)
# fix to
super(ModelParallelModel, self).__init__()
</code></pre>
<p>However, I’m currently not sure, why you are deriving from <code>nn.Sequential</code> instead of <code>nn.Module</code>, as you are not using any attributes from <code>nn.Sequential</code>.</p>

ProGamerGov: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the reply, I was able to get my code running on multiple GPUs!</p>
<p>I pull loss values from multiple layers (that can be on different devices) and add them together before running <code>backward()</code>. Currently, I have to use <code>.to(device)</code> to bring all these loss values to a single device to run backwards. It looks a bit ugly and it’s redundant when I’m using a single GPU/CPU, but I’m not sure if that’s something I could change? DataParallel is for duplicating a model across different GPUs and CPUs, but could I use it only on the loss values so that I don’t have to use to use <code>.to(device)</code>? If I can use it, then would it slow down my code?</p>
<p>I’m also having an issue where GPU:0 always shows at least some usage. In the few tests I did with the same inputs and changing what layers went on which GPUs, I also had at least 850MiB of usage on GPU:0, even when it shouldn’t have been used at all.</p>
<p>Currently I make sure that my inputs and model are CUDA with the following code:</p>
<pre><code class="lang-auto">dtype = torch.cuda.FloatTensor

input_image.type(dtype)
</code></pre>
<pre><code class="lang-auto">model = model.cuda()
</code></pre>
<p>Do I need to do something else in order to prevent GPU:0 from always having extra usage?</p>