IndexError: Caught IndexError in replica 0 on device 0

h_xi: <p>This error occurs when I train my model with 8-gpu, and I don’t know how to fix it.</p>
<p>Traceback (most recent call last):<br>
…<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 161, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 171, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py”, line 86, in parallel_apply<br>
output.reraise()<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/_utils.py”, line 428, in reraise<br>
raise self.exc_type(msg)<br>
IndexError: Caught IndexError in replica 0 on device 0.<br>
Original Traceback (most recent call last):<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker<br>
output = module(*input, **kwargs)<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/root/hxi/Gait Recognition/SSGait0.4/models/vgg_c3d.py”, line 187, in forward<br>
feature = feature.matmul(self.fc_bin[0])<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 426, in <strong>getitem</strong><br>
idx = self._get_abs_string_index(idx)<br>
File “/root/anaconda3/envs/SSGait/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 409, in _get_abs_string_index<br>
raise IndexError(‘index {} is out of range’.format(idx))<br>
IndexError: index 0 is out of range</p>
<p>I guess this error might be related to this statement in my model, but I don’t know how to fix it:<br>
def <strong>init</strong>(self):<br>
self.fc_bin = nn.ParameterList([<br>
nn.Parameter(<br>
nn.init.xavier_uniform_(<br>
torch.zeros(sum(self.bin_numgl), _set_channels[2], _set_channels[2])))<br>
])<br>
def forward(self, x):<br>
feature = feature.matmul(self.fc_bin[0])</p>
<p>When this code is commented out, it runs successfully: feature = feature.matmul(self.fc_bin[0])</p>
<h3>
<a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p>cuda ==11.0 cudatoolkit== 11.0 pytorch==1.7.1</p>

ptrblck: <aside class="quote no-group" data-username="h_xi" data-post="1" data-topic="148655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h_xi/48/48380_2.png" class="avatar"> h_xi:</div>
<blockquote>
<p>When this code is commented out, it runs successfully: feature = feature.matmul(self.fc_bin[0])</p>
</blockquote>
</aside>
<p>It seems the line of code is failing with:</p>
<pre data-code-wrap="python"><code class="lang-python">IndexError: index 0 is out of range
</code></pre>
<p>so make sure <code>self.fc_bin</code> has the correct shape to be indexed by <code>0</code>.</p>

h_xi: <p>self.fc_bin[0].shape is [64,128,128]<br>
feature.shape is [64,16,128]<br>
so  i think ‘feature = feature.matmul(self.fc_bin[0])‘<br>
<code>self.fc_bin</code> has the correct shape<br>
Note that I can train normally with a single gpu, but this error occurs when using 8-gpus</p>

ptrblck: <p>I guess the shape might be different (and invalid) in the 8 GPU case, which raises the error.<br>
If you get stuck, please post a minimal, executable code snippet to reproduce the error.</p>