Multiple GPU with os CUDA_VISIBLE_DEVICES does not work

username: thancaocuong
cooked: <p>Hi, I’ve tried to set CUDA_VISIBLE_DEVICES = ‘1’ in main function but when I move the model to cuda, It does not move to GPU1 but GPU0 instead (result in OOM due to GPU0 is in use). Please tell me if I’m wrong.<br>
here is my code:<br>
in train.py:</p>
<pre><code class="lang-auto">def main(config_file_path):
    config = SspmYamlConfig(config_file_path)

    dataloader_cfg = config.get_dataloader_cfg()
    trainer_cfg = config.get_trainer_cfg()
    logger_cfg = config.get_logger_cfg()
    model_cfg = config.get_model_cfg()
    pose_dataset_cfg = config.get_pose_dataset_cfg()
    data_augmentation_cfg = config.get_augmentation_cfg()
    target_generator_cfg = config.get_target_generator_cfg()

    learning_rate = trainer_cfg['optimizer']['learning_rate']
    # parsing device = [1] by config
    device = ','.join(list(map(str, trainer_cfg['device'])))
    os.environ['CUDA_DEVICE_ORDER']= 'PCI_BUS_ID'
    os.environ['CUDA_VISIBLE_DEVICES'] = device 
    model = getModel(model_cfg)
    train_loader = DataLoader(train_dataset, **dataloader_cfg['train'])
    val_loader = DataLoader(val_dataset, **dataloader_cfg['val'])
    trainer = Trainer(
                      model, optimizer, logger,
                      writer, config, train_loader, val_loader
                      )
</code></pre>
<p>Trainer class is inherited from BaseTrainer where the model was transferd to cuda</p>
<pre><code class="lang-auto">class BaseTrainer(ABC):
    def __init__(self, model, optimizer, logger, writer, config):
        self.config = config
        self.logger = logger
        self.writer = writer
        self.optimizer = optimizer
        self.trainer_config = config.get_trainer_cfg()
        self.device_list = self.trainer_config['device'] #device list is [1]
        self.device_type = self._check_gpu(self.device_list)
        self.device = torch.device(self.device_type)
        self.model = model
        self.model = self.model.to(self.device)
        self.model = torch.nn.DataParallel(self.model)
    def _check_gpu(self, gpus):
        if len(gpus) &gt; 0 and torch.cuda.is_available():
            pynvml.nvmlInit()
            for i in gpus:
                handle = pynvml.nvmlDeviceGetHandleByIndex(i)
                meminfo = pynvml.nvmlDeviceGetMemoryInfo(handle)
                memused = meminfo.used / 1024 / 1024
                self.logger.info('GPU{} used: {}M'.format(i, memused))
                if memused &gt; 1000:
                    pynvml.nvmlShutdown()
                    raise ValueError('GPU{} is occupied!'.format(i))
            pynvml.nvmlShutdown()
            return 'cuda'
        else:
            self.logger.info('Using CPU!')
            return 'cpu'
</code></pre>

username: ptrblck
cooked: <p>If you are masking devices via <code>CUDA_VISIBLE_DEVICES</code> all visible devices will be mapped to device ids in the range <code>[0, nb_visible_devices]</code>.<br>
E.g. if your system has two GPUs and you are using <code>CUDA_VISIBLE_DEVICES=1</code>, you would have to access it inside the script as <code>cuda:0</code>.</p>

username: thancaocuong
cooked: <p>thank you for your quick reply. but I have a question:<br>
I have 3 GPUs, when I want to use only GPU1 and 2 (GPU0 is in use). how should I do?</p>

username: ptrblck
cooked: <p>If all devices are the same, use<br>
<code>CUDA_VISIBLE_DEVICES=1,2 python script.py args</code><br>
to run the script and inside the script use <code>cuda:0</code> and <code>cuda:1</code> (or the equivalent <code>.cuda(0)</code>, <code>.cuda(1)</code> commands).</p>
<p>However, if the mapping is not what you expect via <code>nvidia-smi</code>, you could force the PCI bus order order via <code>CUDA_DEVICE_ORDER=PCI_BUS_ID</code> in front of the aforementioned command.</p>

username: thancaocuong
cooked: <p>I’ve found that I need to set VISIBLE device at the begining of my script. I’s my mistake, thank you for your help. I will close this topic</p>

username: r00bi
cooked: <aside class="quote no-group" data-username="thancaocuong" data-post="6" data-topic="77373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thancaocuong/48/17681_2.png" class="avatar"> thancaocuong:</div>
<blockquote>
<p>und that I need to set VISIBLE device at the begining of my script. I’s my mistake, thank you for your help. I will close this topic</p>
</blockquote>
</aside>
<p>Hi,<br>
I have 8 gpus, suddenly the vm couldn’t recognize them and returns the error of   “raise RuntimeError(“No GPUs available.”)<br>
RuntimeError: No GPUs available.”</p>
<p>when I  <em>export CUDA_VISIBLE_DEVICES=1</em> , it could find the gpu#1 and run code on it, however, I need to use all 8 gpus so set export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 but again getting the same error of no gpus available.<br>
How can I fix it.</p>

username: ptrblck
cooked: <p>I don’t know what might be causing your issues and you might need to check if your script is explicitly setting <code>CUDA_VISIBLE_DEVICES</code> somewhere.</p>

username: Manh_Do_Duy
cooked: <p>I am facing the same issue.<br>
I tried:</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"] = "1,2" 
os.environ["WORLD_SIZE"] = "1"

device = torch.device("cuda:1")

model.load_state_dict(torch.load(model_path, map_location=device))
</code></pre>
<p>but it didn’t work.<br>
Do I have to do any config?</p>