Waining:There is an imbalance between your GPUs

username: xu_wang
cooked: <p>When I was using Dataparallel with multi GPUs,a warning below appear:</p>
<p>There is an imbalance between your GPUs. You may want to exclude GPU 0 which<br>
has less than 75% of the memory or cores of GPU 1. You can do so by setting<br>
the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES<br>
environment variable.</p>
<p>what doesit mean?</p>

username: ptrblck
cooked: <p>It means you machine has different GPUs which differ a lot regarding their performance.<br>
If you would like to use <code>DataParallel</code> the weak GPUs will most likely be a bottleneck in your code. That’s why you can just ignore the weak GPUs using <code>device_ids</code> or <code>CUDA_VISIBLE_DEVICES</code>.</p>
<p>The code of the check and warning is defined <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py#L10">here</a>.</p>

username: Mata_Fu
cooked: <p>Hi, I have the same problem here. But I’m pretty sure that all of my GPU are the same.</p>
<pre><code class="lang-auto">Fri Jun  1 15:17:23 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.24                 Driver Version: 396.24                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:01:00.0 Off |                  N/A |
| 25%   55C    P8    17W / 250W |    887MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 00000000:02:00.0 Off |                  N/A |
| 33%   65C    P8    20W / 250W |    475MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 00000000:04:00.0 Off |                  N/A |
| 22%   41C    P8    13W / 250W |     11MiB / 12211MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
</code></pre>

username: Mata_Fu
cooked: <p>By the way, have you solved this problem?</p>

username: liankf
cooked: <p>I have the same warning, and I solve it by ''from <strong>future</strong> import division ‘’. because in python 2.*,     1/2  =  0.<br>
so in the function" warn_imbalance"<br>
def warn_imbalance(get_prop):<br>
values = [get_prop(props) for props in dev_props]<br>
min_pos, min_val = min(enumerate(values), key=operator.itemgetter(1))<br>
max_pos, max_val = max(enumerate(values), key=operator.itemgetter(1))<br>
if min_val / max_val &lt; 0.75:<br>
warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))<br>
return True<br>
return False</p>
<p>min_val / max_val is directly zero</p>

username: seyeeet
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>can you please help me with this issue.<br>
I have 3 GPUs, 1 is 24GB and 2 are 11GB.<br>
when I have batch size 4, it gives me gpu cuda memory error. However, I am sure the total gpu memory of 3 gpus is capable of handleing it.<br>
How I can make sure to send 2 batches to gpu 0 and 1 batch to gpu 1 and one batch to gpu 2?</p>

username: ptrblck
cooked: <p>You could manipulate or write a custom scatter and gather strategy based on <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py">this code</a>.<br>
I’m not aware of a simple method to specify the chunk sizes.</p>

username: seyeeet
cooked: <p>I’m a beginner about this whole thing. are you aware of a simple code that do that so I can learn from it?</p>