question: Is it possible for SLURM auto submit to work on DP?
context: In my experience, it never works. I looked to the trainer code and saw that the code managing this works only on DDP.
  def configure_slurm_ddp(self, num_gpu_nodes):
      self.is_slurm_managing_tasks = False
  
      ### !!HERE!!
      if self.use_ddp:
          self.num_requested_gpus = self.num_gpus * num_gpu_nodes
          self.num_slurm_tasks = 0
          try:
              self.num_slurm_tasks = int(os.environ['SLURM_NTASKS'])
              self.is_slurm_managing_tasks = self.num_slurm_tasks == self.num_requested_gpus
  
              # in interactive mode we don't manage tasks
              job_name = os.environ['SLURM_JOB_NAME']
              if job_name == 'bash':
                  self.is_slurm_managing_tasks = False
  
          except Exception:
              # likely not on slurm, so set the slurm managed flag to false
              self.is_slurm_managing_tasks = False
  However, sometimes we are not using the distributed computing on slurm (only DP). It would be nice to have the auto resubmit feature still working in this situation.
  
  OS: Linux
  Packaging conda
  Version 16
answer: actually, lightning supports slurm no matter what backend you use...
      def register_slurm_signal_handlers(self):
          # see if we're using slurm (not interactive)
          on_slurm = False
          try:
              job_name = os.environ['SLURM_JOB_NAME']
              if job_name != 'bash':
                  on_slurm = True
          except Exception as e:
              pass
  
          if on_slurm:
              log.info('Set SLURM handle signals.')
              signal.signal(signal.SIGUSR1, self.sig_handler)
              signal.signal(signal.SIGTERM, self.term_handler)