diff --git a/docs/source/package_reference/utilities.mdx b/docs/source/package_reference/utilities.mdx
index 3c1a1065e..9ebb60d34 100644
--- a/docs/source/package_reference/utilities.mdx
+++ b/docs/source/package_reference/utilities.mdx
@@ -24,6 +24,8 @@ These are basic dataclasses used throughout 🤗 Accelerate and they can be pass
 
 [[autodoc]] utils.PrecisionType
 
+[[autodoc]] utils.ProjectConfiguration
+
 ## Data Manipulation and Operations
 
 These include data operations that mimic the same `torch` ops but can be used on distributed processes.
diff --git a/docs/source/quicktour.mdx b/docs/source/quicktour.mdx
index 6d92e7958..6ea99f9bd 100644
--- a/docs/source/quicktour.mdx
+++ b/docs/source/quicktour.mdx
@@ -370,7 +370,11 @@ Note that since all the model parameters are references to tensors, this will lo
 ## Saving/loading entire states
 
 When training your model, you may want to save the current state of the model, optimizer, random generators, and potentially LR schedulers to be restored in the _same script_.
-You can use [`~Accelerator.save_state`] and [`~Accelerator.load_state`] respectively to do so, just by simply passing in a save location. 
+You can use [`~Accelerator.save_state`] and [`~Accelerator.load_state`] respectively to do so.
+
+To further customize where and how states saved through [`~Accelerator.save_state`] the [`~utils.ProjectConfiguration`] class can be used. For example 
+if `automatic_checkpoint_naming` is enabled each saved checkpoint will be located then at `Accelerator.project_dir/checkpoints/checkpoint_{checkpoint_number}`.
+
 If you have registered any other stateful items to be stored through [`~Accelerator.register_for_checkpointing`] they will also be saved and/or loaded.
 
 <Tip>
diff --git a/docs/source/usage_guides/checkpoint.mdx b/docs/source/usage_guides/checkpoint.mdx
index f2684971b..354549205 100644
--- a/docs/source/usage_guides/checkpoint.mdx
+++ b/docs/source/usage_guides/checkpoint.mdx
@@ -17,6 +17,9 @@ saving and loading the model, optimizer, RNG generators, and the GradScaler. Ins
 - Use [`~Accelerator.save_state`] for saving everything mentioned above to a folder location
 - Use [`~Accelerator.load_state`] for loading everything stored from an earlier `save_state`
 
+To further customize where and how states saved through [`~Accelerator.save_state`] the [`~utils.ProjectConfiguration`] class can be used. For example 
+if `automatic_checkpoint_naming` is enabled each saved checkpoint will be located then at `Accelerator.project_dir/checkpoints/checkpoint_{checkpoint_number}`.
+
 It should be noted that the expectation is that those states come from the same training script, they should not be from two separate scripts.
 
 - By using [`~Accelerator.register_for_checkpointing`], you can register custom objects to be automatically stored or loaded from the two prior functions,
@@ -28,7 +31,7 @@ Below is a brief example using checkpointing to save and reload a state during t
 from accelerate import Accelerator
 import torch
 
-accelerator = Accelerator()
+accelerator = Accelerator(project_dir="my/save/path")
 
 my_scheduler = torch.optim.lr_scheduler.StepLR(my_optimizer, step_size=1, gamma=0.99)
 my_model, my_optimizer, my_training_dataloader = accelerator.prepare(my_model, my_optimizer, my_training_dataloader)
@@ -37,7 +40,7 @@ my_model, my_optimizer, my_training_dataloader = accelerator.prepare(my_model, m
 accelerator.register_for_checkpointing(my_scheduler)
 
 # Save the starting state
-accelerator.save_state("my/save/path")
+accelerator.save_state()
 
 device = accelerator.device
 my_model.to(device)
@@ -56,5 +59,5 @@ for epoch in range(num_epochs):
     my_scheduler.step()
 
 # Restore previous state
-accelerator.load_state("my/save/path")
+accelerator.load_state("my/save/path/checkpointing/checkpoint_0")
 ```
diff --git a/docs/source/usage_guides/tracking.mdx b/docs/source/usage_guides/tracking.mdx
index b22a4dd14..3ff40a4a4 100644
--- a/docs/source/usage_guides/tracking.mdx
+++ b/docs/source/usage_guides/tracking.mdx
@@ -83,6 +83,12 @@ for iteration in config["num_iterations"]:
 accelerator.end_training()
 ```
 
+If a tracker requires a directory to save data to such as `TensorBoard` then a `logging_dir` or `project_dir` can be passed in. `project_dir` is useful 
+if there are other further configurations such as those which can be combined with the [`~utils.ProjectConfiguration`] dataclass.
+
+```python
+accelerator = Accelerator(log_with="tensorboard", logging_dir=".")
+```
 
 ## Implementing Custom Trackers
 
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index 1c3782067..151d1db35 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -16,6 +16,7 @@
 import gc
 import math
 import os
+import shutil
 import sys
 import warnings
 from contextlib import contextmanager
@@ -44,6 +45,7 @@
     LoggerType,
     MegatronLMPlugin,
     PrecisionType,
+    ProjectConfiguration,
     RNGType,
     compare_versions,
     convert_outputs_to_fp32,
@@ -157,8 +159,11 @@ class Accelerator:
             - `"comet_ml"`
             If `"all"` is selected, will pick up all available trackers in the environment and initialize them. Can
             also accept implementations of `GeneralTracker` for custom trackers, and can be combined with `"all"`.
-        logging_dir (`str`, `os.PathLike`, *optional*):
-            A path to a directory for storing logs of locally-compatible loggers.
+        project_config (`ProjectConfiguration`, *optional*):
+            A configuration for how saving the state can be handled.
+        project_dir (`str`, `os.PathLike`, *optional*):
+            A path to a directory for storing data such as logs of locally-compatible loggers and potentially saved
+            checkpoints.
         dispatch_batches (`bool`, *optional*):
             If set to `True`, the dataloader prepared by the Accelerator is only iterated through on the main process
             and then the batches are split and broadcast to each process. Will default to `True` for `DataLoader` whose
@@ -205,6 +210,8 @@ def __init__(
         megatron_lm_plugin: MegatronLMPlugin = None,
         rng_types: Optional[List[Union[str, RNGType]]] = None,
         log_with: Optional[List[Union[str, LoggerType, GeneralTracker]]] = None,
+        project_dir: Optional[Union[str, os.PathLike]] = None,
+        project_config: Optional[ProjectConfiguration] = None,
         logging_dir: Optional[Union[str, os.PathLike]] = None,
         dispatch_batches: Optional[bool] = None,
         even_batches: bool = True,
@@ -212,7 +219,19 @@ def __init__(
         kwargs_handlers: Optional[List[KwargsHandler]] = None,
         dynamo_backend: Union[DynamoBackend, str] = None,
     ):
-        self.logging_dir = logging_dir
+        if project_config is not None:
+            self.project_configuration = project_config
+        else:
+            self.project_configuration = ProjectConfiguration(project_dir=project_dir)
+
+        if logging_dir is not None:
+            warnings.warn(
+                "`logging_dir` is deprecated and will be removed in version 0.18.0 of 🤗 Accelerate. Use `project_dir` instead.",
+                FutureWarning,
+            )
+            self.project_configuration.logging_dir = logging_dir
+        if project_dir is not None and self.project_dir is None:
+            self.project_configuration.project_dir = project_dir
         if mixed_precision is not None:
             mixed_precision = str(mixed_precision)
             if mixed_precision not in PrecisionType:
@@ -420,6 +439,18 @@ def local_process_index(self):
     def device(self):
         return self.state.device
 
+    @property
+    def project_dir(self):
+        return self.project_configuration.project_dir
+
+    @property
+    def logging_dir(self):
+        return self.project_configuration.logging_dir
+
+    @property
+    def save_iteration(self):
+        return self.project_configuration.iteration
+
     @property
     def is_main_process(self):
         """True for one process only."""
@@ -1596,9 +1627,16 @@ def save(self, obj, f):
         """
         save(obj, f)
 
-    def save_state(self, output_dir: str):
+    def save_state(self, output_dir: str = None):
         """
-        Saves the current states of the model, optimizer, scaler, RNG generators, and registered objects.
+        Saves the current states of the model, optimizer, scaler, RNG generators, and registered objects to a folder.
+
+        If a `ProjectConfiguration` was passed to the `Accelerator` object with `automatic_checkpoint_naming` enabled
+        then checkpoints will be saved to `self.project_dir/checkpoints`. If the number of current saves is greater
+        than `total_limit` then the oldest save is deleted. Each checkpoint is saved in seperate folders named
+        `checkpoint_<iteration>`.
+
+        Otherwise they are just saved to `output_dir`.
 
         <Tip>
 
@@ -1611,8 +1649,25 @@ def save_state(self, output_dir: str):
             output_dir (`str` or `os.PathLike`):
                 The name of the folder to save all relevant weights and states.
         """
-        # Check if folder exists
-        output_dir = os.path.expanduser(output_dir)
+        if self.project_configuration.automatic_checkpoint_naming:
+            output_dir = os.path.join(self.project_dir, "checkpoints")
+        os.makedirs(output_dir, exist_ok=True)
+        if self.project_configuration.automatic_checkpoint_naming:
+            folders = [os.path.join(output_dir, folder) for folder in os.listdir(output_dir)]
+            if self.project_configuration.total_limit is not None and (
+                len(folders) + 1 > self.project_configuration.total_limit
+            ):
+                folders.sort()
+                logger.warning(
+                    f"Deleting {len(folders) + 1 - self.project_configuration.total_limit} checkpoints to make room for new checkpoint."
+                )
+                for folder in folders[: len(folders) + 1 - self.project_configuration.total_limit]:
+                    shutil.rmtree(folder)
+            output_dir = os.path.join(output_dir, f"checkpoint_{self.save_iteration}")
+            if os.path.exists(output_dir):
+                raise ValueError(
+                    f"Checkpoint directory {output_dir} ({self.save_iteration}) already exists. Please manually override `self.save_iteration` with what iteration to start with."
+                )
         os.makedirs(output_dir, exist_ok=True)
         logger.info(f"Saving current state to {output_dir}")
 
@@ -1660,6 +1715,7 @@ def save_state(self, output_dir: str):
         )
         for i, obj in enumerate(self._custom_objects):
             save_custom_state(obj, output_dir, i)
+        self.project_configuration.iteration += 1
         return save_location
 
     def load_state(self, input_dir: str):
diff --git a/src/accelerate/tracking.py b/src/accelerate/tracking.py
index 75694ba72..7afebea48 100644
--- a/src/accelerate/tracking.py
+++ b/src/accelerate/tracking.py
@@ -147,7 +147,7 @@ class TensorBoardTracker(GeneralTracker):
     name = "tensorboard"
     requires_logging_directory = True
 
-    def __init__(self, run_name: str, logging_dir: Optional[Union[str, os.PathLike]], **kwargs):
+    def __init__(self, run_name: str, logging_dir: Optional[Union[str, os.PathLike]] = None, **kwargs):
         self.run_name = run_name
         self.logging_dir = os.path.join(logging_dir, run_name)
         self.writer = tensorboard.SummaryWriter(self.logging_dir, **kwargs)
@@ -451,7 +451,6 @@ def __init__(
         run_name: Optional[str] = None,
         description: Optional[str] = None,
     ):
-
         experiment_name = os.getenv("MLFLOW_EXPERIMENT_NAME", experiment_name)
         run_id = os.getenv("MLFLOW_RUN_ID", run_id)
         tags = os.getenv("MLFLOW_TAGS", tags)
diff --git a/src/accelerate/utils/__init__.py b/src/accelerate/utils/__init__.py
index be78d0089..d3fb916e8 100644
--- a/src/accelerate/utils/__init__.py
+++ b/src/accelerate/utils/__init__.py
@@ -16,6 +16,7 @@
     LoggerType,
     MegatronLMPlugin,
     PrecisionType,
+    ProjectConfiguration,
     RNGType,
     SageMakerDistributedType,
     TensorInformation,
diff --git a/src/accelerate/utils/dataclasses.py b/src/accelerate/utils/dataclasses.py
index eafad8a09..dc8883425 100644
--- a/src/accelerate/utils/dataclasses.py
+++ b/src/accelerate/utils/dataclasses.py
@@ -313,6 +313,39 @@ class TensorInformation:
     dtype: torch.dtype
 
 
+@dataclass
+class ProjectConfiguration:
+    """
+    Configuration for the Accelerator object based on inner-project needs.
+    """
+
+    project_dir: str = field(default=None, metadata={"help": "A path to a directory for storing data."})
+    logging_dir: str = field(
+        default=None,
+        metadata={
+            "help": "A path to a directory for storing logs of locally-compatible loggers. If None, defaults to `project_dir`."
+        },
+    )
+    automatic_checkpoint_naming: bool = field(
+        default=False,
+        metadata={"help": "Whether saved states should be automatically iteratively named."},
+    )
+
+    total_limit: int = field(
+        default=None,
+        metadata={"help": "The maximum number of total saved states to keep."},
+    )
+
+    iteration: int = field(
+        default=0,
+        metadata={"help": "The current save iteration."},
+    )
+
+    def __post_init__(self):
+        if self.logging_dir is None:
+            self.logging_dir = self.project_dir
+
+
 @dataclass
 class DeepSpeedPlugin:
     """
diff --git a/tests/test_state_checkpointing.py b/tests/test_state_checkpointing.py
index 87b2d3b06..dd2f6a98b 100644
--- a/tests/test_state_checkpointing.py
+++ b/tests/test_state_checkpointing.py
@@ -23,7 +23,7 @@
 from torch.utils.data import DataLoader, TensorDataset
 
 from accelerate import Accelerator
-from accelerate.utils import set_seed
+from accelerate.utils import ProjectConfiguration, set_seed
 
 
 logger = logging.getLogger(__name__)
@@ -75,7 +75,26 @@ def forward(self, x):
 
 
 class CheckpointTest(unittest.TestCase):
-    def test_can_resume_training(self):
+    def test_with_save_limit(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            set_seed(42)
+            model = DummyModel()
+            optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)
+            train_dataloader, valid_dataloader = dummy_dataloaders()
+            project_config = ProjectConfiguration(total_limit=1, project_dir=tmpdir, automatic_checkpoint_naming=True)
+            # Train baseline
+            accelerator = Accelerator(project_config=project_config)
+            model, optimizer, train_dataloader, valid_dataloader = accelerator.prepare(
+                model, optimizer, train_dataloader, valid_dataloader
+            )
+            # Save initial
+            accelerator.save_state()
+
+            # Save second state
+            accelerator.save_state()
+            self.assertEqual(len(os.listdir(accelerator.project_dir)), 1)
+
+    def test_can_resume_training_with_folder(self):
         with tempfile.TemporaryDirectory() as tmpdir:
             set_seed(42)
             model = DummyModel()
@@ -126,6 +145,58 @@ def test_can_resume_training(self):
             self.assertEqual(opt_state1, opt_state3)
             self.assertEqual(ground_truth_rands, test_rands)
 
+    def test_can_resume_training(self):
+        with tempfile.TemporaryDirectory() as tmpdir:
+            set_seed(42)
+            model = DummyModel()
+            optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)
+            train_dataloader, valid_dataloader = dummy_dataloaders()
+            project_config = ProjectConfiguration(automatic_checkpoint_naming=True)
+
+            # Train baseline
+            accelerator = Accelerator(project_dir=tmpdir, project_config=project_config)
+            model, optimizer, train_dataloader, valid_dataloader = accelerator.prepare(
+                model, optimizer, train_dataloader, valid_dataloader
+            )
+            # Save initial
+            accelerator.save_state()
+            (a, b) = model.a.item(), model.b.item()
+            opt_state = optimizer.state_dict()
+            ground_truth_rands = train(3, model, train_dataloader, optimizer, accelerator)
+            (a1, b1) = model.a.item(), model.b.item()
+            opt_state1 = optimizer.state_dict()
+
+            # Train partially
+            set_seed(42)
+            model = DummyModel()
+            optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)
+            train_dataloader, valid_dataloader = dummy_dataloaders()
+            project_config = ProjectConfiguration(iteration=1, automatic_checkpoint_naming=True)
+            accelerator = Accelerator(project_dir=tmpdir, project_config=project_config)
+            model, optimizer, train_dataloader, valid_dataloader = accelerator.prepare(
+                model, optimizer, train_dataloader, valid_dataloader
+            )
+            accelerator.load_state(os.path.join(tmpdir, "checkpoints", "checkpoint_0"))
+            (a2, b2) = model.a.item(), model.b.item()
+            opt_state2 = optimizer.state_dict()
+            self.assertEqual(a, a2)
+            self.assertEqual(b, b2)
+            self.assertEqual(opt_state, opt_state2)
+
+            test_rands = train(2, model, train_dataloader, optimizer, accelerator)
+            # Save everything
+            accelerator.save_state()
+
+            # Load everything back in and make sure all states work
+            accelerator.load_state(os.path.join(tmpdir, "checkpoints", "checkpoint_1"))
+            test_rands += train(1, model, train_dataloader, optimizer, accelerator)
+            (a3, b3) = model.a.item(), model.b.item()
+            opt_state3 = optimizer.state_dict()
+            self.assertEqual(a1, a3)
+            self.assertEqual(b1, b3)
+            self.assertEqual(opt_state1, opt_state3)
+            self.assertEqual(ground_truth_rands, test_rands)
+
     def test_invalid_registration(self):
         t = torch.tensor([1, 2, 3])
         t1 = torch.tensor([2, 3, 4])
@@ -147,19 +218,18 @@ def test_with_scheduler(self):
             optimizer = torch.optim.Adam(params=model.parameters(), lr=1e-3)
             scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.99)
             train_dataloader, valid_dataloader = dummy_dataloaders()
+            project_config = ProjectConfiguration(automatic_checkpoint_naming=True)
             # Train baseline
-            accelerator = Accelerator()
-            model, optimizer, train_dataloader, valid_dataloader = accelerator.prepare(
-                model, optimizer, train_dataloader, valid_dataloader
+            accelerator = Accelerator(project_dir=tmpdir, project_config=project_config)
+            model, optimizer, train_dataloader, valid_dataloader, scheduler = accelerator.prepare(
+                model, optimizer, train_dataloader, valid_dataloader, scheduler
             )
-            accelerator.register_for_checkpointing(scheduler)
             # Save initial
-            initial = os.path.join(tmpdir, "initial")
-            accelerator.save_state(initial)
+            accelerator.save_state()
             scheduler_state = scheduler.state_dict()
             train(3, model, train_dataloader, optimizer, accelerator, scheduler)
             self.assertNotEqual(scheduler_state, scheduler.state_dict())
 
             # Load everything back in and make sure all states work
-            accelerator.load_state(initial)
+            accelerator.load_state(os.path.join(tmpdir, "checkpoints", "checkpoint_0"))
             self.assertEqual(scheduler_state, scheduler.state_dict())
diff --git a/tests/test_tracking.py b/tests/test_tracking.py
index d6ff4908a..1aa341f9b 100644
--- a/tests/test_tracking.py
+++ b/tests/test_tracking.py
@@ -70,9 +70,11 @@ def test_log(self):
             log = list(filter(lambda x: x.is_file(), Path(f"{dirpath}/{project_name}").iterdir()))[0]
             self.assertNotEqual(str(log), "")
 
-    def test_logging_dir(self):
+    def test_project_dir(self):
         with self.assertRaisesRegex(ValueError, "Logging with `tensorboard` requires a `logging_dir`"):
             _ = Accelerator(log_with="tensorboard")
+        with tempfile.TemporaryDirectory() as dirpath:
+            _ = Accelerator(log_with="tensorboard", project_dir=dirpath)
         with tempfile.TemporaryDirectory() as dirpath:
             _ = Accelerator(log_with="tensorboard", logging_dir=dirpath)
 


Review comments: body: This is starting to make too many arguments.
diff_hunk: @@ -158,7 +159,16 @@ class Accelerator:
             If `"all"` is selected, will pick up all available trackers in the environment and initialize them. Can
             also accept implementations of `GeneralTracker` for custom trackers, and can be combined with `"all"`.
         logging_dir (`str`, `os.PathLike`, *optional*):
-            A path to a directory for storing logs of locally-compatible loggers.
+            A path to a directory for storing logs of locally-compatible loggers. If not passed will save in
+            `project_dir` by default.
+        project_dir (`str`, `os.PathLike`, *optional*):
+            A path to a directory for storing data such as logs of locally-compatible loggers and potentially saved
+            checkpoints.
+        automatic_checkpoint_naming (`bool`, *optional*, defaults to `False`):
+            Whether saved states should be stored in `project_location` and be automatically iteratively named.
+        save_total_limit (`int`, *optional*):
+            The maximum number of checkpoints to keep if performing `automatic_checkpoint_naming`, will default to all
+            of them.
body: Fixed with a `SaveConfiguration` class
diff_hunk: @@ -158,7 +159,16 @@ class Accelerator:
             If `"all"` is selected, will pick up all available trackers in the environment and initialize them. Can
             also accept implementations of `GeneralTracker` for custom trackers, and can be combined with `"all"`.
         logging_dir (`str`, `os.PathLike`, *optional*):
-            A path to a directory for storing logs of locally-compatible loggers.
+            A path to a directory for storing logs of locally-compatible loggers. If not passed will save in
+            `project_dir` by default.
+        project_dir (`str`, `os.PathLike`, *optional*):
+            A path to a directory for storing data such as logs of locally-compatible loggers and potentially saved
+            checkpoints.
+        automatic_checkpoint_naming (`bool`, *optional*, defaults to `False`):
+            Whether saved states should be stored in `project_location` and be automatically iteratively named.
+        save_total_limit (`int`, *optional*):
+            The maximum number of checkpoints to keep if performing `automatic_checkpoint_naming`, will default to all
+            of them.
body: Let's also deprecate logging_dir to include it in the SaveConfig? Wdyt?
diff_hunk: @@ -205,13 +213,23 @@ def __init__(
         megatron_lm_plugin: MegatronLMPlugin = None,
         rng_types: Optional[List[Union[str, RNGType]]] = None,
         log_with: Optional[List[Union[str, LoggerType, GeneralTracker]]] = None,
+        project_dir: Optional[Union[str, os.PathLike]] = None,
+        save_config: Optional[SaveConfiguration] = None,
         logging_dir: Optional[Union[str, os.PathLike]] = None,
         dispatch_batches: Optional[bool] = None,
         even_batches: bool = True,
         step_scheduler_with_optimizer: bool = True,
         kwargs_handlers: Optional[List[KwargsHandler]] = None,
         dynamo_backend: Union[DynamoBackend, str] = None,
     ):
+        if save_config is not None:
+            self.save_configuration = save_config
+        else:
+            self.save_configuration = SaveConfiguration()
+        if project_dir is not None and self.project_dir is None:
+            self.save_configuration.project_dir = project_dir
+        if self.project_dir is not None and logging_dir is None:
body: I was debating on that, if we did so I think it'd be better to name it `ProjectConfiguration` instead since it's all relative to the total "project" at that point. But can make adjustments for it. The only "con" I see is given the conversation above for the user the code would change, but the end-result would not so I think it'd be fine
diff_hunk: @@ -205,13 +213,23 @@ def __init__(
         megatron_lm_plugin: MegatronLMPlugin = None,
         rng_types: Optional[List[Union[str, RNGType]]] = None,
         log_with: Optional[List[Union[str, LoggerType, GeneralTracker]]] = None,
+        project_dir: Optional[Union[str, os.PathLike]] = None,
+        save_config: Optional[SaveConfiguration] = None,
         logging_dir: Optional[Union[str, os.PathLike]] = None,
         dispatch_batches: Optional[bool] = None,
         even_batches: bool = True,
         step_scheduler_with_optimizer: bool = True,
         kwargs_handlers: Optional[List[KwargsHandler]] = None,
         dynamo_backend: Union[DynamoBackend, str] = None,
     ):
+        if save_config is not None:
+            self.save_configuration = save_config
+        else:
+            self.save_configuration = SaveConfiguration()
+        if project_dir is not None and self.project_dir is None:
+            self.save_configuration.project_dir = project_dir
+        if self.project_dir is not None and logging_dir is None:
body: ```suggestion
if there are other further configurations such as those which can be combined with the [`~utils.ProjectConfiguration`] dataclass.
```
diff_hunk: @@ -83,6 +83,12 @@ for iteration in config["num_iterations"]:
 accelerator.end_training()
 ```
 
+If a tracker requires a directory to save data to such as `TensorBoard` then a `logging_dir` or `project_dir` can be passed in. `project_dir` is useful 
+for if there are other further configurations such as those which can be combined with the [`~utils.ProjectConfiguration`] dataclass.

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: In my setup I have a VM for training and a LXC that hosts TensorBoard linked by a NFS share. I set `logging_dir` to the NFS share so the VM pushes its logs directly to TB as they are generated. Since this PR centralizes logging and output I would no longer be able to dump my logs directly to the TB server and instead have to do some complicated rsync watchdog service.

Can we keep `logging_dir` as an option? Just set it to `project_dir` by default, no need to depreciate it.
body: @Cyberes there is a deprecation cycle for `logging_dir` of two full versions and when it comes to trackers `project_dir` acts the exact same as the `logging_dir` parameter, so for your particular setup if I understand correctly would just need to change `logging_dir` to `project_dir` and nothing else.  

Is this reasonable enough for you? Or is there something more specific/drastic I've misunderstood.
body: Just to make sure I'm understanding the new functionality, the checkpoints are saved to `project_dir/checkpoints`  and the logs are saved to `project_dir/logs/[name]`?
body: Checkpoints are saved to `project_dir/checkpoints`, logs are saved to `project_dir/[name]`
body: Yeah, I'd like to have the option to put them in completely separate directories. For example, save the checkpoint states to `project_dir/checkpoints` and TB logs to `/mnt/tensorboard`.

If you're against the seperate directories thing I could always override `TensorBoardTracker`, I guess.
body: That use case makes sense, will do as you suggested and override only if not passed. Thanks @Cyberes :) 
body: Sweeeeeeeeeet, thanks @muellerzr.