RuntimeError: Device index must not be negative

mohamed_laabydy: <p>Hello,</p>
<p>I’m trying to train a YOLOv7 model on Databricks using a GPU, but I’m encountering an error when running the following training command:</p>
<pre><code class="lang-auto">!python train.py --batch-size 2 --epochs 8 --data data.yaml --weights 'yolov7_training.pt' --workers 2
</code></pre>
<p>The error message is:</p>
<pre><code class="lang-auto">File "/Workspace/Users/Yolov7/train.py", line 624, in &lt;module&gt;
    train(hyp, opt, device, tb_writer)
  File "/Workspace/Yolov7/train.py", line 368, in train
    pred = model(imgs)  # forward
  File "/databricks/python/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/databricks/python/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/databricks/python/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 993, in _run_ddp_forward
    inputs, kwargs = _to_kwargs(
  File "/databricks/python/lib/python3.10/site-packages/torch/distributed/utils.py", line 94, in _to_kwargs
    _recursive_to(inputs, device_id, use_side_stream_for_tensor_copies)
  File "/databricks/python/lib/python3.10/site-packages/torch/distributed/utils.py", line 86, in _recursive_to
    res = to_map(inputs)
  File "/databricks/python/lib/python3.10/site-packages/torch/distributed/utils.py", line 77, in to_map
    return list(zip(*map(to_map, obj)))
  File "/databricks/python/lib/python3.10/site-packages/torch/distributed/utils.py", line 55, in to_map
    if obj.device == torch.device("cuda", target_gpu):
RuntimeError: Device index must not be negative
</code></pre>
<p>I want to avoid using Distributed Data Parallel (DDP) because I’ve encountered issues with it before. When I print the <code>device</code> variable right before the error, it returns <code>"cuda:0"</code>.</p>
<p>Here are the details about the environment I’m using:</p>
<ul>
<li>Databricks Runtime: 13.3 LTS ML (includes Apache Spark 3.4.1, GPU, Scala 2.12)</li>
</ul>
<p>Any insights into what might be causing this issue and how I can resolve it would be greatly appreciated.</p>
<p>Thank you!</p>

Eduardo_Lawson: <p>Hi mohamed! If you share the part of the train.py that is giving this error, it will be easier to help you</p>

mohamed_laabydy: <p>Thank you, Eduardo, for your reply.</p>
<p>I’m using the code directly from the official <a href="https://github.com/WongKinYiu/yolov7?tab=readme-ov-file" rel="noopener nofollow ugc">YOLOv7 repository</a> with the following initialization:</p>
<p><code>dist.init_process_group(backend='nccl')</code><br>
I’m doing this to bypass the DDP protocol.</p>
<p>Here’s the relevant part of my code:</p>
<pre><code class="lang-auto">    opt.total_batch_size = opt.batch_size
    device = select_device(opt.device, batch_size=opt.batch_size)
    
    # Hyperparameters
    with open(opt.hyp) as f:
        hyp = yaml.load(f, Loader=yaml.SafeLoader)  # load hyps

    # Train
    logger.info(opt)
    if not opt.evolve:
        tb_writer = None  # init loggers
        if opt.global_rank in [-1, 0]:
            prefix = colorstr('tensorboard: ')
            logger.info(f"{prefix}Start with 'tensorboard --logdir {opt.project}', view at http://localhost:6006/")
            tb_writer = SummaryWriter(opt.save_dir)  # Tensorboard
        train(hyp, opt, device, tb_writer)
</code></pre>
<p>For the <code>device</code> argument, I’m specifying it as follows:</p>
<p><code>parser.add_argument('--device', default='0', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')</code></p>