Train jointly with Tesla P4 & Tesla T4

username: 111104
cooked: <p>Hi all.</p>
<p>I’m trying to train a model simultaneously on two types of cards: Tesla P4 &amp; Tesla T4.<br>
I’m using the codes from <code>https://github.com/kuangliu/pytorch-cifar</code>. However, it failed unfortunately. The error log is:</p>
<blockquote>
<p>==&gt; Preparing data…<br>
Files already downloaded and verified<br>
Files already downloaded and verified<br>
==&gt; Building model…<br>
/home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py:26: UserWarning:<br>
There is an imbalance between your GPUs. You may want to exclude GPU 1 which<br>
has less than 75% of the memory or cores of GPU 0. You can do so by setting<br>
the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES<br>
environment variable.<br>
warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))</p>
</blockquote>
<blockquote>
<p>Epoch: 0<br>
Traceback (most recent call last):<br>
File “main.py”, line 150, in <br>
train(epoch)<br>
File “main.py”, line 102, in train<br>
loss.backward()<br>
File “/home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 100, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM (operator() at /opt/conda/conda-bld/pytorch_1591914880026/work/aten/src/ATen/native/cudnn/Conv.cpp:1142)<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7fcb40ac6b5e in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0xd70c62 (0x7fcb41c90c62 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0xd6dbe5 (0x7fcb41c8dbe5 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0xd6f07f (0x7fcb41c8f07f in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0xd72cd0 (0x7fcb41c92cd0 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#5:</span> at::native::cudnn_convolution_backward_weight(c10::ArrayRef, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool) + 0x49 (0x7fcb41c92f29 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0xdd9880 (0x7fcb41cf9880 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#7:</span>  + 0xe1daf8 (0x7fcb41d3daf8 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#8:</span> at::native::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x2fc (0x7fcb41c93bdc in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#9:</span>  + 0xdd958b (0x7fcb41cf958b in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#10:</span>  + 0xe1db54 (0x7fcb41d3db54 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#11:</span>  + 0x29dee26 (0x7fcb6aab4e26 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#12:</span>  + 0x2a2e634 (0x7fcb6ab04634 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#13:</span> torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;&amp;) + 0x378 (0x7fcb6a6ccff8 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#14:</span>  + 0x2ae7df5 (0x7fcb6abbddf5 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#15:</span> torch::autograd::Engine::evaluate_function(std::shared_ptr<a>torch::autograd::GraphTask</a>&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) + 0x16f3 (0x7fcb6abbb0f3 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#16:</span> torch::autograd::Engine::thread_main(std::shared_ptr<a>torch::autograd::GraphTask</a> const&amp;, bool) + 0x3d2 (0x7fcb6abbbed2 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#17:</span> torch::autograd::Engine::thread_init(int) + 0x39 (0x7fcb6abb4549 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#18:</span> torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7fcb6e104638 in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#19:</span>  + 0xc819d (0x7fcb7095f19d in /home/yjh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/…/…/…/…/./libstdc++.so.6)<br>
frame <span class="hashtag-raw">#20:</span>  + 0x9609 (0x7fcb89219609 in /lib/x86_64-linux-gnu/libpthread.so.0)<br>
frame <span class="hashtag-raw">#21:</span> clone + 0x43 (0x7fcb89140103 in /lib/x86_64-linux-gnu/libc.so.6)</p>
</blockquote>
<p>However, I remember this works earlier. Also, everything is ok now if I run the codes on a single type of card. I don’t know whether there’re changes on pytorch source codes, could anyone be so kind to help me?</p>

username: ptrblck
cooked: <p>Are you directly using the linked code or did you add any changes to it?<br>
Also, could you post the PyTorch, CUDA, and cudnn versions you are using?</p>

username: 111104
cooked: <p>I’m using the linked codes without modification since I’m just testing a new machine.</p>
<p>After some test, I found:<br>
cudatoolkit               10.2.89 + cudnn                     7.6.5 + pytorch                   1.5.1 fails;<br>
cudatoolkit               10.0.130 + cudnn 7.4.2 + pytorch                   1.0.1 works.</p>

username: ptrblck
cooked: <p>Could you halve the batch size and rerun the script of these devices in isolation for the failing version, please?</p>

username: 111104
cooked: <p>It works in isolation.</p>

username: ptrblck
cooked: <p>Does it even work in isolation with <code>batch_size/2</code>, as this would be the new workload for each device, if you are using <code>nn.DataParallel</code>?</p>

username: 111104
cooked: <p>Yes it works in isolation with halved batch size.</p>

username: ptrblck
cooked: <p>That’s bad, as we won’t be able to try to reproduce it easily.<br>
Could you try to create the cudnn logs as described <a href="https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#api-logging">here</a> and upload them?</p>

username: 111104
cooked: <p>The log is large so I upload it  <a href="https://drive.google.com/file/d/1SgdMn5cdwQ56IsxHN3YzZ9A2EaKn0HDR/view?usp=sharing" rel="nofollow noopener">here</a>.</p>