Error in use multiple gpu in my source

setare: <p>Hi I want to run my project on two gpu parallel so after I write my code, I write this command<br>
in this commands I write my model in paralell and then in pl.trainer in add parameter of gpus=2<br>
’<br>
def main():<br>
datamodule = DataModule(train_ds, val_ds)<br>
mymodel = mymodel(config)<br>
trainer = pl.Trainer(<br>
accelerator=“gpu”,<br>
gpus=2,<br>
)<br>
trainer.fit(model, datamodule)</p>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
main()</p>
<p>but when I run it I see this error how to edit my source code?( I’d appreciate it if you could help me with it)<br>
‘TypeError: <code>Trainer.fit()</code> requires a <code>LightningModule</code>, got: DataParallel’</p>

fduwjj: <aside class="quote no-group" data-username="setare" data-post="1" data-topic="165622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/48.png" class="avatar"> setare:</div>
<blockquote>
<p>parallel so after I write my code, I write this command<br>
in this commands I write my model in</p>
</blockquote>
</aside>
<p>Looks like this is something more related to lightning. Maybe can you post the all of your code here so that we can better help you? Thanks!</p>

setare: <p>Hi, yes sure. Thank you for your responce.<br>
my source code is…</p>
<p>I send my source if it could be better show my error</p>
<p>This is data module</p>
<pre><code class="lang-auto">class DataModule(pl.LightningDataModule):

  def __init__(self, train_dataset, val_dataset,  batch_size = 1):
    super(DataModule, self).__init__()
    self.train_dataset = train_dataset
    self.val_dataset = val_dataset
    self.batch_size = batch_size
  def train_dataloader(self):
    return DataLoader(self.train_dataset, batch_size = self.batch_size, collate_fn = collate_fn, shuffle = True, num_workers = 0, pin_memory = True)
  
  def val_dataloader(self):
    return DataLoader(self.val_dataset, batch_size = self.batch_size,collate_fn = collate_fn, shuffle = False, num_workers = 0, pin_memory = True)
</code></pre>
<p>and this is part of model</p>
<pre><code class="lang-auto">class LaTrForVQA(pl.LightningModule):
  def __init__(self, config , learning_rate = 1e-4, max_steps = 100000//2):
    super(LaTrForVQA, self).__init__()   
    self.config = config
    self.save_hyperparameters()
    self.latr =  LaTr_for_finetuning(config)
    self.training_losses = []
    self.validation_losses = []
    self.max_steps = max_steps

  def configure_optimizers(self):
    return torch.optim.AdamW(self.parameters(), lr = self.hparams['learning_rate'])

  def forward(self, batch_dict):
    boxes =   batch_dict['boxes']
    img =     batch_dict['img']
    question = batch_dict['question']
    words =   batch_dict['tokenized_words']
    answer_vector = self.latr(lang_vect = words, 
                               spatial_vect = boxes, 
                               img_vect = img, 
                               quest_vect = question
                               )
    return answer_vector

  def calculate_metrics(self, prediction, labels):

      ## Calculate the accuracy score between the prediction and ground label for a batch, with considering the pad sequence
      batch_size = len(prediction)
      ac_score = 0

      for (pred, gt) in zip(prediction, labels):
        ac_score+= calculate_acc_score(pred.detach().cpu(), gt.detach().cpu())
      ac_score = ac_score/batch_size
      return ac_score

  def training_step(self, batch, batch_idx):
    answer_vector = self.forward(batch)

    ## https://discuss.huggingface.co/t/bertformaskedlm-s-loss-and-scores-how-the-loss-is-computed/607/2
    loss = nn.CrossEntropyLoss()(answer_vector.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(answer_vector, dim = -1)

    ## Calculating the accuracy score
    train_acc = self.calculate_metrics(preds, batch['answer'])
    train_acc = torch.tensor(train_acc)

    ## Logging
    self.log('train_ce_loss', loss,prog_bar = True)
    self.log('train_acc', train_acc, prog_bar = True)
    self.training_losses.append(loss.item())

    return loss

  def validation_step(self, batch, batch_idx):
    logits = self.forward(batch)
    loss = nn.CrossEntropyLoss()(logits.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(logits, dim = -1)

    ## Validation Accuracy
    val_acc = self.calculate_metrics(preds.cpu(), batch['answer'].cpu())
    val_acc = torch.tensor(val_acc)

    ## Logging
    self.log('val_ce_loss', loss, prog_bar = True)
    self.log('val_acc', val_acc, prog_bar = True)    
    return {'val_loss': loss, 'val_acc': val_acc}

  def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i, opt_closure = None, on_tpu=False,
    using_native_amp=False, using_lbfgs=False):
        if self.trainer.global_step &lt; 1000:
            lr_scale = min(1., float(self.trainer.global_step + 1) / 1000.)
            for pg in optimizer.param_groups:
                pg['lr'] = lr_scale * self.hparams.learning_rate
        else:
            for pg in optimizer.param_groups:
                pg['lr'] = polynomial(self.hparams.learning_rate, self.trainer.global_step, max_iter = self.max_steps)
        optimizer.step(opt_closure)
        optimizer.zero_grad()

  def validation_epoch_end(self, outputs):       
        val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        val_acc = torch.stack([x['val_acc'] for x in outputs]).mean()
        self.log('val_loss_epoch_end', val_loss, on_epoch=True, sync_dist=True)
        self.log('val_acc_epoch_end', val_acc, on_epoch=True, sync_dist=True)
        
        self.val_prediction = []

model=LaTrForVQA(config)
</code></pre>
<p>and this is for trainer</p>
<pre><code class="lang-auto">trainer = pl.Trainer(
    max_steps = max_steps,
    default_root_dir="runs",
    gpus=2,
    deterministic=True,
) 
</code></pre>
<p>Now I fit my model</p>
<pre><code class="lang-auto">datamodule = DataModule(train_ds, val_ds)
trainer.fit(model,datamodule)   
</code></pre>
<p>If you have any other questions, be sure to ask…<br>
I change my code according some suggestion to don’t use parallel and only use pl.trainer(gpus=2)  but with this code I still can’t use both of GPU in parallel and only one GPU work in source code and I dont know what other change I should apply in this source.</p>

fduwjj: <aside class="quote no-group" data-username="setare" data-post="3" data-topic="165622">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/48.png" class="avatar"> setare:</div>
<blockquote>
<p><code>LightningDataModule</code></p>
</blockquote>
</aside>
<p>To enable parallelization in Lightning training, you need to use strategy here:<br>
<a href="https://pytorch-lightning.readthedocs.io/en/stable/api_references.html#strategies" class="inline-onebox" rel="noopener nofollow ugc">accelerators — PyTorch Lightning 2.1.2 documentation</a>.</p>

setare: <p>Hi,<br>
I use different type of strategy and anyone don’t work and give me different type of errors and I don’t know what to do and can’t find any solution<br>
I mention some of this error</p>
<p>When I set strategy=“ddp_spawn” (or ddp).It gives me this error</p>
<p>MisconfigurationException: <code>Trainer(strategy='ddp_spawn')</code> is not compatible with an interactive environment. Run your code as a script, or choose one of the compatible strategies: Trainer(strategy=None|dp|ddp_fork). In case you are spawning processes yourself, make sure to include the Trainer creation inside the worker function</p>
<p>When I set strategy=“ddp_fork” .It gives me this error<br>
"Cannot re-initialize CUDA in forked subprocess. To use CUDA with "<br>
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>
<p>When I set strategy=“dp” . It gives me this error<br>
AssertionError: Gather function not implemented for CPU tensors</p>
<p>When I set strategy=" ddp_fork " (or ddp_notebook).It gives me this error<br>
ValueError: ‘ddp_notebook’ is not a valid DistributedType</p>
<p>If it is my wrong in source code what to do?<br>
I’d so appreciate it if you could help me with this wrong.</p>

fduwjj: <p>How did you set your multi-process env? I mean from these errors looks it is complaining the way you set up multi-process env.</p>

setare: <p>If I have understood your point, first of all I create a virtual environment in anaconda navigator with python 3.7 and my source could identify both of GPU</p>
<p>import torch<br>
print(‘__CUDNN VERSION:’, torch.backends.cudnn.version())<br>
print(‘__Number CUDA Devices:’, torch.cuda.device_count())<br>
print(‘__CUDA Device Name 1:’,torch.cuda.get_device_name(0))<br>
print(‘__CUDA Device Name 2:’,torch.cuda.get_device_name(1))<br>
print(‘__CUDA Device current:’,torch.cuda.get_device_name(torch.cuda.current_device()))<br>
print(‘__CUDA Device Total Memory [GB]:’,torch.cuda.get_device_properties(0).total_memory/1e9)<br>
print(‘__CUDA Device Total Memory [GB]:’,torch.cuda.get_device_properties(1).total_memory/1e9)<br>
print(‘__CUDA Device Total Memory [GB]:’,torch.cuda.get_device_properties(0).total_memory/1e9+torch.cuda.get_device_properties(1).total_memory/1e9)</p>
<p>_CUDNN VERSION: 8302<br>
__Number CUDA Devices: 2<br>
__CUDA Device Name 1: NVIDIA GeForce GTX 1080 Ti<br>
__CUDA Device Name 2: NVIDIA GeForce GTX 1070 Ti<br>
__CUDA Device Total Memory [GB]: 11.721572352<br>
__CUDA Device Total Memory [GB]: 8.514043904<br>
__CUDA Device Total Memory [GB]: 20.235616256<br>
​<br>
version of pytorch-lightning  is     1.8.0.post1</p>
<p>and I set for different kinds of pytorch package . All of them have these errors.<br>
It is worth mentioning that with the same settings I can use both of my GPU in dataparallel only but I can’t use multi gpu in pytorch-lightning .</p>
<p>If I have not understood your point, may you give more guidance.</p>

setare: <p>Hi <a class="mention" href="/u/fduwjj">@fduwjj</a> ,<br>
This problem is very important for me. I would appreciate it if you gave your opinion about this problem. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

harshas: <p>try replacing pl.LightningDataModule in class DataModule() with pl.LightningModule</p>

Tali_M: <p>Did you find an answer? I’m having similar problems…</p>

setare: <p>Unfortunately, no. I use gpu that have more RAM.</p>