tensorboard.compat.tensorflow_stub.errors.AlreadyExistsError: Directory already exists
Author: rabeehkarimimahabadi
Pull request: no
Labels: name: wontfix

Hi
I am running finetune_trainer.py on cloud with TPUs, here is the error, I appreciate your help thanks
```json
{
  "textPayload": "Traceback (most recent call last):\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 330, in _mp_start_fn\n    _start_fn(index, pf_cfg, fn, args)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 324, in _start_fn\n    fn(gindex, *args)\n  File \"/workdir/seq2seq/finetune_trainer.py\", line 303, in _mp_fn\n    app.run(main, flags_parser=parse_flags)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py\", line 300, in run\n    _run_main(main, args)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py\", line 251, in _run_main\n    sys.exit(main(argv))\n  File \"/workdir/seq2seq/finetune_trainer.py\", line 246, in main\n    data_args=data_args,\n  File \"/workdir/seq2seq/seq2seq_trainer.py\", line 37, in __init__\n    super().__init__(*args, **kwargs)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer.py\", line 318, in __init__\n    self.control = self.callback_handler.on_init_end(self.args, self.state, self.control)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer_callback.py\", line 325, in on_init_end\n    return self.call_event(\"on_init_end\", args, state, control)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/trainer_callback.py\", line 376, in call_event\n    **kwargs,\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/integrations.py\", line 213, in on_init_end\n    self.tb_writer = SummaryWriter(log_dir=args.logging_dir)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py\", line 221, in __init__\n    self._get_file_writer()\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py\", line 252, in _get_file_writer\n    self.flush_secs, self.filename_suffix)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py\", line 62, in __init__\n    log_dir, max_queue, flush_secs, filename_suffix)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/tensorboard/summary/writer/event_file_writer.py\", line 77, in __init__\n    tf.io.gfile.makedirs(logdir)\n  File \"/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/io/gfile.py\", line 673, in makedirs\n    return get_filesystem(path).makedirs(path)\ntensorboard.compat.tensorflow_stub.errors.AlreadyExistsError: Directory already exists\n",
  "insertId": "5rl5rhwn8m5sobp9q",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "seq2seq",
      "location": "europe-west4-a",
      "project_id": "try-ideas-for-rmi",
      "namespace_name": "ruse-xgcp",
      "cluster_name": "xcloud-v3-donut-europe-west4-a",
      "pod_name": "20201105.df.e2753.0-7mxdc"
    }
  },
  "timestamp": "2020-11-05T11:28:19.975413564Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/job-name": "20201105.df.e2753.0",
    "k8s-pod/controller-uid": "4974cfc1-b3df-4882-9fd1-9095f4a944d9",
    "k8s-pod/jobowner": "ruse-xgcp",
    "k8s-pod/app": "xcloud",
    "k8s-pod/serviceName": "xc-20201105-df-e2753-0"
  },
  "logName": "projects/try-ideas-for-rmi/logs/stderr",
  "receiveTimestamp": "2020-11-05T11:28:23.810886436Z"
}
```

Comments:
Well it seems to tell you that your tensorboard directory already exists? Try with a different directory?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.