question: Is it possible to disable CheckpointConnector?
context: Even with checkpoint_callback=False, Trainer appears to be using CheckpointConnector for some reason. Since very occasionally (once every ~30 runs) the checkpoint is either deleted too early or is never created in the first place (no idea which one), the whole experiment fails, as shown in the log below. Since CheckpointConnector does not appear to be doing anything important when running locally, is it possible to eliminate it without breaking the training process?
  Traceback (most recent call last):
    File ".\my_code\run_automated.py", line 95, in <module>
      main(experiment, config, dataset)
    File "D:\GIT\my_code\processing.py", line 117, in main
      trainer.fit(model, dataloader_train, dataloader_val)
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 513, in fit
      self.dispatch()
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 553, in dispatch
      self.accelerator.start_training(self)
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\accelerators\accelerator.py", line 74, in start_training
      self.training_type_plugin.start_training(trainer)
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 111, in start_training
      self._results = trainer.run_train()
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 609, in run_train
      self._pre_training_routine()
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 600, in _pre_training_routine
      self.checkpoint_connector.restore_weights()
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py", line 65, in restore_weights
      max_suffix = self.max_ckpt_in_folder(dir_path_hpc, "hpc_ckpt_")
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py", line 372, in max_ckpt_in_folder
      files = [os.path.basename(f["name"]) for f in fs.listdir(dir_path)]
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\spec.py", line 1122, in listdir
      return self.ls(path, detail=detail, **kwargs)
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\implementations\local.py", line 51, in ls
      return [self.info(f) for f in paths]
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\implementations\local.py", line 51, in <listcomp>
      return [self.info(f) for f in paths]
    File "c:\users\pluczak\.conda\envs\pytorch\lib\site-packages\fsspec\implementations\local.py", line 61, in info
      out = os.stat(path, follow_symlinks=False)
  FileNotFoundError: [WinError 2] Nie można odnaleźć określonego pliku: 'D:/GIT/my_code/04d2a63662b34828946b5545646c063f.pt'
answer: So are you saying the file returned by fs.listdir(dir_path) gets removed?
  If you can reproduce this, please open an Issue about it. Seems like a bug.
  In the meantime, you can patch the problematic function like this:
  trainer.checkpoint_connector.max_ckpt_in_folder = lambda *args, **kwargs: None