diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index aa04063d8..c213a9167 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -12,6 +12,6 @@ jobs:
       with:
         python-version: 3.6
     - name: Install Python dependencies
-      run: pip install -e .[test]
+      run: pip install setuptools==59.5.0; pip install -e .[test,test_trackers]
     - name: Run Tests
       run: make test
\ No newline at end of file
diff --git a/Makefile b/Makefile
index 943191f62..e5b557f10 100644
--- a/Makefile
+++ b/Makefile
@@ -25,4 +25,4 @@ style:
 	
 # Run tests for the library
 test:
-	python -m pytest -n auto --dist=loadfile -s -v ./tests/
+	python -m pytest -n auto --dist=loadfile -s -v ./tests/
\ No newline at end of file
diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml
index 8f266c600..0a6d45892 100644
--- a/docs/source/_toctree.yml
+++ b/docs/source/_toctree.yml
@@ -21,4 +21,6 @@
     title: Internals
   - local: checkpoint
     title: Checkpointing
+  - local: tracking
+    title: Experiment Tracking
   title: API Reference
diff --git a/docs/source/internal.mdx b/docs/source/internal.mdx
index 3acaaad87..bd6924990 100644
--- a/docs/source/internal.mdx
+++ b/docs/source/internal.mdx
@@ -44,6 +44,10 @@ The main work on your PyTorch `DataLoader` is done by the following function:
 
 [[autodoc]] state.DistributedType
 
+## Tracking
+
+[[autodoc]] tracking.GeneralTracker
+
 ## Utilities
 
 [[autodoc]] utils.extract_model_from_parallel
diff --git a/docs/source/tracking.mdx b/docs/source/tracking.mdx
new file mode 100644
index 000000000..256e2848b
--- /dev/null
+++ b/docs/source/tracking.mdx
@@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Tracking
+
+There are a large number of experiment tracking API's available, however getting them all to work with in a multi-processing environment can oftentimes be complex.
+Accelerate provides a general tracking API that can be used to log useful items during your script through [`~Accelerator.log`]
+
+## Integrated Trackers
+
+Currently `Accelerate` supports three trackers out-of-the-box:
+
+
+[[autodoc]] tracking.TensorBoardTracker
+
+[[autodoc]] tracking.WandBTracker
+
+[[autodoc]] tracking.CometMLTracker
+
+To use any of them, pass in the selected type(s) to the `log_with` parameter in [`Accelerate`]:
+```python
+from accelerate import Accelerate
+from accelerate.utils import LoggerType
+
+accelerator = Accelerate(log_with="all")  # For all available trackers in the environment
+accelerator = Accelerate(log_with="wandb")
+accelerator = Accelerate(log_with=["wandb", LoggerType.TENSORBOARD])
+```
+
+At the start of your experiment [`~Accelerator.init_trackers`] should be used to setup your project, and potentially add any experiment hyperparameters to be logged:
+```python
+hps = {"num_iterations": 5, "learning_rate": 1e-2}
+accelerator.init_trackers("my_project", config=hps)
+```
+
+When you are ready to log any data, [`~Accelerator.log`] should be used.
+A `step` can also be passed in to correlate the data with a particular step in the training loop.
+```python
+accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=1)
+```
+
+Once you've finished training, make sure to run [`~Accelerator.end_training`] so that all the trackers can run their finish functionalities if they have any.
+```python
+accelerator.end_training()
+```
+
+
+A full example is below:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator(log_with="all")
+config = {
+    "num_iterations": 5,
+    "learning_rate": 1e-2,
+    "loss_function": str(my_loss_function),
+}
+
+accelerator.init_trackers("example_project", config=config)
+
+my_model, my_optimizer, my_training_dataloader = accelerate.prepare(my_model, my_optimizer, my_training_dataloader)
+device = accelerator.device
+my_model.to(device)
+
+for iteration in config["num_iterations"]:
+    for step, batch in my_training_dataloader:
+        my_optimizer.zero_grad()
+        inputs, targets = batch
+        inputs = inputs.to(device)
+        targets = targets.to(device)
+        outputs = my_model(inputs)
+        loss = my_loss_function(outputs, targets)
+        accelerator.backward(loss)
+        my_optimizer.step()
+        accelerator.log({"training_loss": loss}, step=step)
+accelerator.end_training()
+```
+
+
+## Implementing Custom Trackers
+
+To implement a new tracker to be used in `Accelerator`, a new one can be made through implementing the [`~GeneralTracker`] class.
+Every tracker must implement three functions:
+  - `__init__`: 
+    - Should store a `run_name` and initialize the tracker API of the integrated library. 
+    - If a tracker stores their data locally (such as TensorBoard), a `logging_dir` parameter can be added.
+  - `store_init_configuration`: 
+    - Should take in a `values` dictionary and store them as a one-time experiment configuration
+  - `log`: 
+    - Should take in a `values` dictionary and a `step`, and should log them to the run
+
+A brief example can be seen below with an integration with Weights and Biases, containing only the relevent information:
+```python
+from accelerate.tracking import GeneralTracker
+from typing import Optional
+
+import wandb
+
+
+class WandBTracker(GeneralTracker):
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        wandb.init(self.run_name)
+
+    def store_init_configuration(self, values: dict):
+        wandb.config(values)
+
+    def log(self, values: dict, step: Optional[int] = None):
+        wandb.log(values, step=step)
+```
+
+## When a wrapper cannot work
+
+If a library has an API that does not follow a strict `.log` with an overall dictionary such as Neptune.AI, logging can be done manually under an `if accelerator.is_main_process` statement:
+```diff
+from accelerate import Accelerator
++ import neptune.new as neptune
+
+accelerator = Accelerator()
++ run = neptune.init(...)
+
+my_model, my_optimizer, my_training_dataloader = accelerate.prepare(my_model, my_optimizer, my_training_dataloader)
+device = accelerator.device
+my_model.to(device)
+
+for iteration in config["num_iterations"]:
+    for batch in my_training_dataloader:
+        my_optimizer.zero_grad()
+        inputs, targets = batch
+        inputs = inputs.to(device)
+        targets = targets.to(device)
+        outputs = my_model(inputs)
+        loss = my_loss_function(outputs, targets)
+        total_loss += loss
+        accelerator.backward(loss)
+        my_optimizer.step()
++       if accelerator.is_main_process:
++           run["logs/training/batch/loss"].log(loss)
+```
diff --git a/setup.py b/setup.py
index c898eb5a1..0ce1e5af4 100644
--- a/setup.py
+++ b/setup.py
@@ -20,8 +20,9 @@
 extras["docs"] = []
 extras["test"] = [
     "pytest",
-    "pytest-xdist",
+    "pytest-xdist"
 ]
+extras["test_trackers"] = ["wandb", "comet-ml", "tensorflow"]
 extras["dev"] = extras["quality"] + extras["test"]
 
 extras["sagemaker"] = [
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index 2b623db34..f52cf7ca8 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -28,14 +28,19 @@
 from .kwargs_handlers import DistributedDataParallelKwargs, GradScalerKwargs, InitProcessGroupKwargs, KwargsHandler
 from .optimizer import AcceleratedOptimizer
 from .state import AcceleratorState, DistributedType, is_deepspeed_available
+from .tracking import CometMLTracker, TensorBoardTracker, WandBTracker, get_available_trackers
 from .utils import (
     DeepSpeedPlugin,
+    LoggerType,
     PrecisionType,
     RNGType,
     convert_outputs_to_fp32,
     extract_model_from_parallel,
     gather,
     get_pretty_name,
+    is_comet_ml_available,
+    is_tensorboard_available,
+    is_wandb_available,
     pad_across_processes,
     save,
     wait_for_everyone,
@@ -88,6 +93,17 @@ class Accelerator:
               dataloader) or of the iterable dataset (if it exists) if the underlying dataset is of that type.
 
             Will default to `["torch"]` for PyTorch versions <=1.5.1 and `["generator"]` for PyTorch versions >= 1.6.
+        log_with (list of `str` or [`~utils.LoggerType`], *optional*):
+            A list of loggers to be setup for experiment tracking. Should be one or several of:
+
+            - `"all"`
+            - `"tensorboard"`
+            - `"wandb"`
+            - `"comet_ml"`
+
+            If `"all`" is selected, will pick up all available trackers in the environment and intialize them.
+        logging_dir (`str`, `os.PathLike`, *optional*):
+            A path to a directory for storing logs of locally-compatible loggers.
         dispatch_batches (`bool`, *optional*):
             If set to `True`, the dataloader prepared by the Accelerator is only iterated through on the main process
             and then the batches are split and broadcast to each process. Will default to `True` for `DataLoader` whose
@@ -111,9 +127,27 @@ def __init__(
         cpu: bool = False,
         deepspeed_plugin: DeepSpeedPlugin = None,
         rng_types: Optional[List[Union[str, RNGType]]] = None,
+        log_with: Optional[List[Union[str, LoggerType]]] = None,
+        logging_dir: Optional[Union[str, os.PathLike]] = "",
         dispatch_batches: Optional[bool] = None,
         kwargs_handlers: Optional[List[KwargsHandler]] = None,
     ):
+        if log_with is not None:
+            if not isinstance(log_with, list):
+                log_with = [log_with]
+            if "all" in log_with or LoggerType.ALL in log_with:
+                log_with = get_available_trackers()
+            else:
+                for i, log_type in enumerate(log_with):
+                    if log_type not in LoggerType:
+                        raise ValueError(
+                            f"Unsupported logging capability: {log_type}. Choose between {LoggerType.list()}"
+                        )
+                    log_with[i] = LoggerType(log_type)
+                log_with = list(set(log_with))
+        self.log_with = log_with
+        self.logging_dir = logging_dir
+
         if mixed_precision is not None:
             mixed_precision = str(mixed_precision)
             if mixed_precision not in PrecisionType:
@@ -556,6 +590,50 @@ def wait_for_everyone(self):
         """
         wait_for_everyone()
 
+    def init_trackers(self, project_name: str, config: Optional[dict] = None):
+        """
+        Initializes a run for all trackers stored in `self.log_with`, potentially with starting configurations
+
+        Args:
+            project_name (`str`):
+                The name of the project. All trackers will save their data based on this
+            config (`dict`, *optional*):
+                Optional starting configuration to be logged.
+        """
+        self.trackers = []
+        for tracker in self.log_with:
+            if str(tracker).lower() == "tensorboard" and is_tensorboard_available():
+                self.trackers.append(TensorBoardTracker(project_name, self.logging_dir))
+            elif str(tracker).lower() == "wandb" and is_wandb_available():
+                self.trackers.append(WandBTracker(project_name))
+            elif str(tracker).lower() == "comet_ml" and is_comet_ml_available():
+                self.trackers.append(CometMLTracker(project_name))
+        if config is not None:
+            for tracker in self.trackers:
+                tracker.store_init_configuration(config)
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to all stored trackers in `self.trackers`.
+
+        Args:
+            values (`dict`):
+                Values should be a dictionary-like object containing only types `int`, `float`, or `str`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        if self.is_main_process:
+            for tracker in self.trackers:
+                tracker.log(values, step=step)
+
+    def end_training(self):
+        """
+        Runs any special end training behaviors, such as stopping trackers
+        """
+        if self.is_main_process:
+            for tracker in self.trackers:
+                tracker.finish()
+
     def save(self, obj, f):
         """
         Save the object passed to disk once per machine. Use in place of `torch.save`.
diff --git a/src/accelerate/test_utils/testing.py b/src/accelerate/test_utils/testing.py
index 507f16eb8..cce7347f6 100644
--- a/src/accelerate/test_utils/testing.py
+++ b/src/accelerate/test_utils/testing.py
@@ -19,7 +19,7 @@
 import torch
 
 from ..state import AcceleratorState, is_tpu_available
-from ..utils import gather
+from ..utils import gather, is_tensorflow_available
 
 
 def are_the_same_tensors(tensor):
@@ -64,6 +64,17 @@ def require_multi_gpu(test_case):
         return test_case
 
 
+def require_tensorflow(test_case):
+    """
+    Decorator marking a test that requires TensorFlow installed. These tests are skipped when TensorFlow isn't
+    installed
+    """
+    if not is_tensorflow_available():
+        return unittest.skip("test requires TensorFlow")(test_case)
+    else:
+        return test_case
+
+
 class _RunOutput:
     def __init__(self, returncode, stdout, stderr):
         self.returncode = returncode
diff --git a/src/accelerate/tracking.py b/src/accelerate/tracking.py
new file mode 100644
index 000000000..65a5caec2
--- /dev/null
+++ b/src/accelerate/tracking.py
@@ -0,0 +1,245 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+import os
+from abc import ABCMeta, abstractmethod
+from typing import Optional, Union
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    @abstractmethod
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Implementations should use the experiment configuration
+        functionality of a tracking API.
+
+        Args:
+            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
+                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`,
+                `str`, `float`, `int`, or `None`.
+        """
+        pass
+
+    @abstractmethod
+    def log(self, values: dict, step: Optional[int]):
+        """
+        Logs `values` to the current run. Base `log` implementations of a tracking API should go in here, along with
+        special behavior for the `step parameter.
+
+        Args:
+            values (Dictionary `str` to `str`, `float`, or `int`):
+                Values to be logged as key-value pairs. The values need to have type `str`, `float`, or `int`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        pass
+
+    def finish(self):
+        """
+        Should run any finalizing functions within the tracking API. If the API should not have one, just don't
+        overwrite that method.
+        """
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run
+        logging_dir (`str`, `os.PathLike`):
+            Location for TensorBoard logs to be stored.
+    """
+
+    def __init__(self, run_name: str, logging_dir: Optional[Union[str, os.PathLike]] = ""):
+        self.run_name = run_name
+        self.logging_dir = os.path.join(logging_dir, run_name)
+        self.writer = tensorboard.SummaryWriter(self.logging_dir)
+        logger.info(f"Initialized TensorBoard project {self.run_name} logging to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
+                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`,
+                `str`, `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (Dictionary `str` to `str`, `float`, or `int`):
+                Values to be logged as key-value pairs. The values need to have type `str`, `float`, or `int`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        for k, v in values.items():
+            if isinstance(v, (int, float)):
+                self.writer.add_scalar(k, v, global_step=step)
+            elif isinstance(v, str):
+                self.writer.add_text(k, v, global_step=step)
+        self.writer.flush()
+        logger.info("Successfully logged to TensorBoard")
+
+    def finish(self):
+        """
+        Closes `TensorBoard` writer
+        """
+        self.writer.close()
+        logger.info("TensorBoard writer closed")
+
+
+class WandBTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `wandb`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        self.run = wandb.init(self.run_name)
+        logger.info(f"Initialized WandB project {self.run_name}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
+                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`,
+                `str`, `float`, `int`, or `None`.
+        """
+        wandb.config.update(values)
+        logger.info("Stored initial configuration hyperparameters to WandB")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (Dictionary `str` to `str`, `float`, or `int`):
+                Values to be logged as key-value pairs. The values need to have type `str`, `float`, or `int`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        self.run.log(values, step=step)
+        logger.info("Successfully logged to WandB")
+
+    def finish(self):
+        """
+        Closes `wandb` writer
+        """
+        self.run.finish()
+        logger.info("WandB run closed")
+
+
+class CometMLTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `comet_ml`. Should be initialized at the start of your script.
+
+    API keys must be stored in a Comet config file.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        self.writer = Experiment(project_name=run_name)
+        logger.info(f"Initialized CometML project {self.run_name}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
+                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`,
+                `str`, `float`, `int`, or `None`.
+        """
+        self.writer.log_parameters(values)
+        logger.info("Stored initial configuration hyperparameters to CometML")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (Dictionary `str` to `str`, `float`, or `int`):
+                Values to be logged as key-value pairs. The values need to have type `str`, `float`, or `int`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        if step is not None:
+            self.writer.set_step(step)
+        self.writer.log_others(values)
+        logger.info("Successfully logged to CometML")
diff --git a/src/accelerate/utils.py b/src/accelerate/utils.py
index af9299a4f..c5a291ab8 100644
--- a/src/accelerate/utils.py
+++ b/src/accelerate/utils.py
@@ -34,6 +34,22 @@
     import torch_xla.core.xla_model as xm
 
 
+def is_tensorflow_available():
+    return importlib.util.find_spec("tensorflow") is not None
+
+
+def is_tensorboard_available():
+    return importlib.util.find_spec("tensorboard") is not None or importlib.util.find_spec("tensorboardX") is not None
+
+
+def is_wandb_available():
+    return importlib.util.find_spec("wandb") is not None
+
+
+def is_comet_ml_available():
+    return importlib.util.find_spec("comet_ml") is not None
+
+
 def is_boto3_available():
     return importlib.util.find_spec("boto3") is not None
 
@@ -74,6 +90,13 @@ def list(cls):
         return list(map(lambda item: str(item), cls))
 
 
+class LoggerType(BaseEnum):
+    ALL = "all"
+    TENSORBOARD = "tensorboard"
+    WANDB = "wandb"
+    COMETML = "comet_ml"
+
+
 class PrecisionType(BaseEnum):
     NO = "no"
     FP16 = "fp16"
diff --git a/tests/test_tracking.py b/tests/test_tracking.py
new file mode 100644
index 000000000..8d5ebc6bc
--- /dev/null
+++ b/tests/test_tracking.py
@@ -0,0 +1,176 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import os
+import re
+import shutil
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+# We use TF to parse the logs
+from accelerate import Accelerator
+from accelerate.test_utils.testing import require_tensorflow
+from accelerate.utils import is_tensorflow_available
+
+
+if is_tensorflow_available():
+    import tensorflow as tf
+    from tensorboard.plugins.hparams import plugin_data_pb2
+    from tensorflow.core.util import event_pb2
+    from tensorflow.python.summary.summary_iterator import summary_iterator
+
+
+logger = logging.getLogger(__name__)
+
+
+class TensorBoardTrackingTest(unittest.TestCase):
+    @require_tensorflow
+    def test_init_trackers(self):
+        hps = None
+        project_name = "test_project_with_config"
+        with tempfile.TemporaryDirectory() as dirpath:
+            oldpwd = os.getcwd()
+            os.chdir(dirpath)
+            accelerator = Accelerator(log_with="tensorboard")
+            config = {"num_iterations": 12, "learning_rate": 1e-2, "some_boolean": False, "some_string": "some_value"}
+            accelerator.init_trackers(project_name, config)
+            accelerator.end_training()
+            for child in Path(project_name).glob("*/**"):
+                log = list(filter(lambda x: x.is_file(), child.iterdir()))[0]
+                # The config log is stored one layer deeper in the logged directory
+                # And names are randomly generated each time
+            si = summary_iterator(str(log))
+            # Pull HPS through careful parsing
+            for event in si:
+                for value in event.summary.value:
+                    proto_bytes = value.metadata.plugin_data.content
+                    plugin_data = plugin_data_pb2.HParamsPluginData.FromString(proto_bytes)
+                    if plugin_data.HasField("session_start_info"):
+                        hps = dict(plugin_data.session_start_info.hparams)
+            os.chdir(oldpwd)
+
+        self.assertTrue(isinstance(hps, dict))
+        keys = list(hps.keys())
+        keys.sort()
+        self.assertEqual(keys, ["learning_rate", "num_iterations", "some_boolean", "some_string"])
+        self.assertEqual(hps["num_iterations"].number_value, 12)
+        self.assertEqual(hps["learning_rate"].number_value, 0.01)
+        self.assertEqual(hps["some_boolean"].bool_value, False)
+        self.assertEqual(hps["some_string"].string_value, "some_value")
+
+    @require_tensorflow
+    def test_log(self):
+        step = None
+        project_name = "test_project_with_log"
+        with tempfile.TemporaryDirectory() as dirpath:
+            oldpwd = os.getcwd()
+            os.chdir(dirpath)
+            accelerator = Accelerator(log_with="tensorboard")
+            accelerator.init_trackers(project_name)
+            values = {"total_loss": 0.1, "iteration": 1, "my_text": "some_value"}
+            accelerator.log(values, step=0)
+            accelerator.end_training()
+            # Logged values are stored in the outermost-tfevents file and can be read in as a TFRecord
+            # Names are randomly generated each time
+            log = list(filter(lambda x: x.is_file(), Path(project_name).iterdir()))[0]
+            serialized_examples = tf.data.TFRecordDataset(log)
+            for e in serialized_examples:
+                event = event_pb2.Event.FromString(e.numpy())
+                if step is None:
+                    step = event.step
+                for value in event.summary.value:
+                    if value.tag == "total_loss":
+                        total_loss = value.simple_value
+                    elif value.tag == "iteration":
+                        iteration = value.simple_value
+                    elif value.tag == "my_text/text_summary":  # Append /text_summary to the key
+                        my_text = value.tensor.string_val[0].decode()
+            os.chdir(oldpwd)
+        self.assertAlmostEqual(total_loss, values["total_loss"])
+        self.assertEqual(iteration, values["iteration"])
+        self.assertEqual(my_text, values["my_text"])
+
+
+@mock.patch.dict(os.environ, {"WANDB_MODE": "offline"})
+class WandBTrackingTest(unittest.TestCase):
+    @staticmethod
+    def get_value_from_log(key: str, log: str, key_occurance: int = 0):
+        """
+        Parses wandb log for `key` and returns the value.
+        If parsing through multiple calls to .log, pass in a `key_occurance`
+        """
+        res = re.findall(rf"(?<={key} )[^\s]+", log)[key_occurance]
+        if '"' in res:
+            return re.findall(r'"([^"]*)"', res)[0]
+        else:
+            return res
+
+    def test_init_trackers(self):
+        project_name = "test_project_with_config"
+        accelerator = Accelerator(log_with="wandb")
+        config = {"num_iterations": 12, "learning_rate": 1e-2, "some_boolean": False, "some_string": "some_value"}
+        accelerator.init_trackers(project_name, config)
+        accelerator.end_training()
+        # The latest offline log is stored at wandb/latest-run/*.wandb
+        for child in Path("wandb/latest-run").glob("*"):
+            logger.info(child)
+            if child.is_file() and child.suffix == ".wandb":
+                with open(child, "rb") as f:
+                    content = f.read()
+                break
+
+        # Check HPS through careful parsing and cleaning
+        cleaned_log = re.sub(r"[\x00-\x1f]+", " ", content.decode("utf8", "ignore"))
+        self.assertEqual(self.get_value_from_log("num_iterations", cleaned_log), "12")
+        self.assertEqual(self.get_value_from_log("learning_rate", cleaned_log), "0.01")
+        self.assertEqual(self.get_value_from_log("some_boolean", cleaned_log), "false")
+        self.assertEqual(self.get_value_from_log("some_string", cleaned_log), "some_value")
+
+    def test_log(self):
+        project_name = "test_project_with_log"
+        accelerator = Accelerator(log_with="wandb")
+        accelerator.init_trackers(project_name)
+        values = {"total_loss": 0.1, "iteration": 1, "my_text": "some_value"}
+        accelerator.log(values, step=0)
+        accelerator.end_training()
+        # The latest offline log is stored at wandb/latest-run/*.wandb
+        for child in Path("wandb/latest-run").glob("*"):
+            if child.is_file() and child.suffix == ".wandb":
+                with open(child, "rb") as f:
+                    content = f.read()
+                break
+        # Check HPS through careful parsing and cleaning
+        cleaned_log = re.sub(r"[\x00-\x1f]+", " ", content.decode("utf8", "ignore"))
+        self.assertEqual(self.get_value_from_log("total_loss", cleaned_log), "0.1")
+        self.assertEqual(self.get_value_from_log("iteration", cleaned_log), "1")
+        self.assertEqual(self.get_value_from_log("my_text", cleaned_log), "some_value")
+        self.assertEqual(self.get_value_from_log("_step", cleaned_log), "0")
+
+    def setUp(self):
+        os.mkdir(".wandb_tests")
+        os.chdir(".wandb_tests")
+
+    def tearDown(self):
+        if os.getcwd().endswith(".wandb_tests"):
+            os.chdir("..")
+        if os.path.exists(".wandb_tests"):
+            shutil.rmtree(".wandb_tests")
+
+    @classmethod
+    def setUpClass(cls):
+        if os.path.exists(".wandb_tests"):
+            shutil.rmtree(".wandb_tests")


Review comments: body: ```suggestion
from .tracking import TensorBoardTracker
```
:scream: 
diff_hunk: @@ -17,10 +17,12 @@
 import sys
 import warnings
 from contextlib import contextmanager
+from pathlib import Path
 from typing import List, Optional, Union
 
 import torch
 
+from accelerate.accelerate.src.accelerate.tracking import TensorBoardTracker
body: The `"all"` should be dealt with around here too.
diff_hunk: @@ -111,9 +115,17 @@ def __init__(
         cpu: bool = False,
         deepspeed_plugin: DeepSpeedPlugin = None,
         rng_types: Optional[List[Union[str, RNGType]]] = None,
+        log_with: Optional[List[Union[LoggerType, str]]] = None,
         dispatch_batches: Optional[bool] = None,
         kwargs_handlers: Optional[List[KwargsHandler]] = None,
     ):
+        if log_with is not None:
+            if not isinstance(log_with, list):
body: Nice!
diff_hunk: @@ -556,6 +568,30 @@ def wait_for_everyone(self):
         """
         wait_for_everyone()
 
+    def init_trackers(self, project_name: str, config: dict = None):
+        """
+        Initializes a run for all trackers stored in `self.log_with`, potentially with starting configurations
+
+        Args:
+            project_name (`str`):
+        """
+        project_location = Path(project_name)
+        self.trackers = []
+        for tracker in self.log_with:
+            if tracker.lower() == "tensorboard" and is_tensorboard_available():
+                self.trackers.append(TensorBoardTracker(project_location))
+        if config is not None:
+            for tracker in self.trackers:
+                tracker.store_init_configuration(config)
body: Should be inside an `is_tensorboard_available` flag.
diff_hunk: @@ -0,0 +1,72 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+
+from torch.utils import tensorboard
body: Accelerate has no idea about the global step though. We could add it in an optional argument to `log`? Or add something that keeps track of the number of time the `optimizer.step` is called internally.
diff_hunk: @@ -0,0 +1,72 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+
+from torch.utils import tensorboard
+
+
+logger = logging.getLogger(__name__)
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    log_directory = "tensorboard"
+
+    def __init__(self, run_name=""):
+        self.run_name = Path(run_name)
+        self.writer = tensorboard.SummaryWriter(self.logging_dir)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values):
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    # add_scalar has the option for `global_step`, should we include this?
body: That's more or less what my thinking is, having it be an optional argument. I don't think there's a need for us to keep track of `optimizer.step` calls internally for this yet, but we'll see what changes as this develops
diff_hunk: @@ -0,0 +1,72 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+
+from torch.utils import tensorboard
+
+
+logger = logging.getLogger(__name__)
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    log_directory = "tensorboard"
+
+    def __init__(self, run_name=""):
+        self.run_name = Path(run_name)
+        self.writer = tensorboard.SummaryWriter(self.logging_dir)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values):
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    # add_scalar has the option for `global_step`, should we include this?
body: *That is what we call VSCode not being friendly and me missing it!*
diff_hunk: @@ -17,10 +17,12 @@
 import sys
 import warnings
 from contextlib import contextmanager
+from pathlib import Path
 from typing import List, Optional, Union
 
 import torch
 
+from accelerate.accelerate.src.accelerate.tracking import TensorBoardTracker
body: Decided to make it an optional argument. If folks seem to find it annoying then we can go ahead and track `optimizer.step` next
diff_hunk: @@ -0,0 +1,72 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+
+from torch.utils import tensorboard
+
+
+logger = logging.getLogger(__name__)
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    log_directory = "tensorboard"
+
+    def __init__(self, run_name=""):
+        self.run_name = Path(run_name)
+        self.writer = tensorboard.SummaryWriter(self.logging_dir)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values):
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    # add_scalar has the option for `global_step`, should we include this?
body: ```suggestion
            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`, `str`,
                `float`, `int`, or `None`.
```
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
body: Seems to be optional and has a default ;-)
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
body: ```suggestion
            step (`int`, *optional*):
```
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (`dict`):
+                Values to be logged as key-value pairs. Value be of type `int`, `float`, or `str`.
+            step (`int`, Optional):
body: Same comment as above for the `run_name` of Tensorboard.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (`dict`):
+                Values to be logged as key-value pairs. Value be of type `int`, `float`, or `str`.
+            step (`int`, Optional):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        for k, v in values.items():
+            if isinstance(v, (int, float)):
+                self.writer.add_scalar(k, v, global_step=step)
+            elif isinstance(v, str):
+                self.writer.add_text(k, v, global_step=step)
+        self.writer.flush()
+        logger.info("Successfully logged to TensorBoard")
+
+
+class WandBTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `wandb`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
body: Same comments as above.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (`dict`):
+                Values to be logged as key-value pairs. Value be of type `int`, `float`, or `str`.
+            step (`int`, Optional):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        for k, v in values.items():
+            if isinstance(v, (int, float)):
+                self.writer.add_scalar(k, v, global_step=step)
+            elif isinstance(v, str):
+                self.writer.add_text(k, v, global_step=step)
+        self.writer.flush()
+        logger.info("Successfully logged to TensorBoard")
+
+
+class WandBTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `wandb`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    log_directory = None
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        wandb.init(self.run_name)
+        logger.info(f"Initialized WandB project {self.run_name}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
body: And here too ;-)
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (`dict`):
+                Values to be logged as key-value pairs. Value be of type `int`, `float`, or `str`.
+            step (`int`, Optional):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        for k, v in values.items():
+            if isinstance(v, (int, float)):
+                self.writer.add_scalar(k, v, global_step=step)
+            elif isinstance(v, str):
+                self.writer.add_text(k, v, global_step=step)
+        self.writer.flush()
+        logger.info("Successfully logged to TensorBoard")
+
+
+class WandBTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `wandb`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    log_directory = None
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        wandb.init(self.run_name)
+        logger.info(f"Initialized WandB project {self.run_name}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
+        """
+        wandb.config(values)
+        logger.info("Stored initial configuration hyperparameters to WandB")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (`dict`):
+                Values to be logged as key-value pairs.
+            step (`int`, Optional):
+                The run step. If included, the log will be affiliated with this step.
body: Same comments on the docstrings.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (`dict`):
+                Values to be logged as key-value pairs. Value be of type `int`, `float`, or `str`.
+            step (`int`, Optional):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        for k, v in values.items():
+            if isinstance(v, (int, float)):
+                self.writer.add_scalar(k, v, global_step=step)
+            elif isinstance(v, str):
+                self.writer.add_text(k, v, global_step=step)
+        self.writer.flush()
+        logger.info("Successfully logged to TensorBoard")
+
+
+class WandBTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `wandb`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    log_directory = None
+
+    def __init__(self, run_name: str = ""):
+        self.run_name = run_name
+        wandb.init(self.run_name)
+        logger.info(f"Initialized WandB project {self.run_name}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (`dict`):
+                Values to be stored as initial hyperparameters as key-value pairs. Value be of type `bool`, `str`,
+                `float`, `int`, or `None`.
+        """
+        wandb.config(values)
+        logger.info("Stored initial configuration hyperparameters to WandB")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (`dict`):
+                Values to be logged as key-value pairs.
+            step (`int`, Optional):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        wandb.log(values, step=step)
+        logger.info("Successfully logged to WandB")
+
+
+class CometMLTracker(GeneralTracker):
+    """
body: You'll need to add the `tracking.` prefix to each for the links to work.
diff_hunk: @@ -0,0 +1,142 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Tracking
+
+There are a large number of experiment tracking API's available, however getting them all to work with in a multi-processing environment can oftentimes be complex.
+Accelerate provides a general tracking API that can be used to log useful items during your script through [`~Accelerator.log`]
+
+## Integrated Trackers
+
+Currently `Accelerate` supports three trackers out-of-the-box:
+- [`~TensorBoardTracker`]
+- [`~WandBTracker`]
+- [`~CometMLTracker`]
body: ```suggestion
+           run["logs/training/batch/loss"].log(loss)
```
4 spaces please :-)
diff_hunk: @@ -0,0 +1,142 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Tracking
+
+There are a large number of experiment tracking API's available, however getting them all to work with in a multi-processing environment can oftentimes be complex.
+Accelerate provides a general tracking API that can be used to log useful items during your script through [`~Accelerator.log`]
+
+## Integrated Trackers
+
+Currently `Accelerate` supports three trackers out-of-the-box:
+- [`~TensorBoardTracker`]
+- [`~WandBTracker`]
+- [`~CometMLTracker`]
+
+To use any of them, pass in the selected type(s) to the `log_with` parameter in [`~Accelerate.__init__`]:
+```python
+from accelerate import Accelerate
+from accelerate.utils import LoggerType
+
+accelerator = Accelerate(log_with="all")  # For all available trackers in the environment
+accelerator = Accelerate(log_with="wandb")
+accelerator = Accelerate(log_with=["wandb", LoggerType.TENSORBOARD])
+```
+
+At the start of your experiment [`~Accelerator.init_tracker`] should be used to setup your project, and potentially add any experiment hyperparameters to be logged:
+```python
+hps = {"num_iterations": 5, "learning_rate": 1e-2}
+accelerator.init_trackers("my_project", config=hps)
+```
+
+When you are ready to log any data, [`~Accelerator.log`] should be used.
+A `step` can also be passed in to correlate the data with a particular step in the training loop.
+```python
+accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=1)
+```
+
+A full example is below:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator(log_with="all")
+config = {
+    "num_iterations": 5,
+    "learning_rate": 1e-2,
+    "loss_function": str(my_loss_function),
+}
+
+accelerator.init_trackers("example_project", config=config)
+
+my_model, my_optimizer, my_training_dataloader = accelerate.prepare(my_model, my_optimizer, my_training_dataloader)
+device = accelerator.device
+my_model.to(device)
+
+for iteration in config["num_iterations"]:
+    for step, batch in my_training_dataloader:
+        my_optimizer.zero_grad()
+        inputs, targets = batch
+        inputs = inputs.to(device)
+        targets = targets.to(device)
+        outputs = my_model(inputs)
+        loss = my_loss_function(outputs, targets)
+        accelerator.backward(loss)
+        my_optimizer.step()
+        accelerator.log({"training_loss": loss}, step=step)
+```
+
+[[autodoc]] tracking.TensorBoardTracker
+
+[[autodoc]] tracking.WandBTracker
+
+[[autodoc]] tracking.CometMLTracker
+
+
+## Implementing Custom Trackers
+
+To implement a new tracker to be used in `Accelerator`, a new one can be made through implementing the [`~GeneralTracker`] class.
+Every tracker must implement three functions:
+  - `__init__`: Should store a `run_name` and initialize the tracker API of the integrated library
+  - `store_init_configuration`: Should take in a `values` dictionary and store them as a one-time experiment configuration
+  - `log`: Should take in a `values` dictionary and a `step`, and should log them to the run
+As well as contain a `log_directory` attribute. If the integrated library does not store anything locally, should be set to `None`.
+
+A brief example can be seen below with an integration with Weights and Biases, containing only the relevent information:
+```python
+from accelerate.tracking import GeneralTracker
+from typing import Optional
+
+import wandb
+
+
+class WandBTracker(GeneralTracker):
+    log_directory = None
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        wandb.init(self.run_name)
+
+    def store_init_configuration(self, values: dict):
+        wandb.config(values)
+
+    def log(self, values: dict, step: Optional[int] = None):
+        wandb.log(values, step=step)
+```
+
+## When a wrapper cannot work
+
+If a library has an API that does not follow a strict `.log` with an overall dictionary such as Neptune.AI, logging can be done manually under an `if accelerator.is_main_process` statement:
+```diff
+from accelerate import Accelerator
++ import neptune.new as neptune
+
+accelerator = Accelerator()
++ run = neptune.init(...)
+
+my_model, my_optimizer, my_training_dataloader = accelerate.prepare(my_model, my_optimizer, my_training_dataloader)
+device = accelerator.device
+my_model.to(device)
+
+for iteration in config["num_iterations"]:
+    for batch in my_training_dataloader:
+        my_optimizer.zero_grad()
+        inputs, targets = batch
+        inputs = inputs.to(device)
+        targets = targets.to(device)
+        outputs = my_model(inputs)
+        loss = my_loss_function(outputs, targets)
+        total_loss += loss
+        accelerator.backward(loss)
+        my_optimizer.step()
++       if accelerator.is_main_process:
++         run["logs/training/batch/loss"].log(loss)
body: This call should be done at the Tracker level.
diff_hunk: @@ -556,6 +585,39 @@ def wait_for_everyone(self):
         """
         wait_for_everyone()
 
+    def init_trackers(self, project_name: str, config: Optional[dict] = None):
+        """
+        Initializes a run for all trackers stored in `self.log_with`, potentially with starting configurations
+
+        Args:
+            project_name (`str`):
+                The name of the project. All trackers will save their data based on this
+            config (`dict`, *optional*):
+                Optional starting configuration to be logged.
+        """
+        project_location = Path(project_name)
body: The logging_dir should also be something that could be customized and default to the usual PyTorch default.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
body: This variable doesn't seem used.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
body: This doesn't seem used either.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
+                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`,
+                `str`, `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (Dictionary `str` to `str`, `float`, or `int`):
+                Values to be logged as key-value pairs. The values need to have type `str`, `float`, or `int`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        for k, v in values.items():
+            if isinstance(v, (int, float)):
+                self.writer.add_scalar(k, v, global_step=step)
+            elif isinstance(v, str):
+                self.writer.add_text(k, v, global_step=step)
+        self.writer.flush()
+        logger.info("Successfully logged to TensorBoard")
+
+
+class WandBTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `wandb`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    log_directory = None
body: Same here.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
+        logger.info(f"Initialized TensorBoard project {self.run_name} writing to {self.logging_dir}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
+                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`,
+                `str`, `float`, `int`, or `None`.
+        """
+        self.writer.add_hparams(values, metric_dict={})
+        self.writer.flush()
+        logger.info("Stored initial configuration hyperparameters to TensorBoard")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (Dictionary `str` to `str`, `float`, or `int`):
+                Values to be logged as key-value pairs. The values need to have type `str`, `float`, or `int`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        for k, v in values.items():
+            if isinstance(v, (int, float)):
+                self.writer.add_scalar(k, v, global_step=step)
+            elif isinstance(v, str):
+                self.writer.add_text(k, v, global_step=step)
+        self.writer.flush()
+        logger.info("Successfully logged to TensorBoard")
+
+
+class WandBTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `wandb`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    log_directory = None
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        wandb.init(self.run_name)
+        logger.info(f"Initialized WandB project {self.run_name}")
+        logger.info(
+            "Make sure to log any initial configurations with `self.store_init_configuration` before training!"
+        )
+
+    def store_init_configuration(self, values: dict):
+        """
+        Logs `values` as hyperparameters for the run. Should be run at the beginning of your experiment.
+
+        Args:
+            values (Dictionary `str` to `bool`, `str`, `float` or `int`):
+                Values to be stored as initial hyperparameters as key-value pairs. The values need to have type `bool`,
+                `str`, `float`, `int`, or `None`.
+        """
+        wandb.config(values)
+        logger.info("Stored initial configuration hyperparameters to WandB")
+
+    def log(self, values: dict, step: Optional[int] = None):
+        """
+        Logs `values` to the current run.
+
+        Args:
+            values (Dictionary `str` to `str`, `float`, or `int`):
+                Values to be logged as key-value pairs. The values need to have type `str`, `float`, or `int`.
+            step (`int`, *optional*):
+                The run step. If included, the log will be affiliated with this step.
+        """
+        wandb.log(values, step=step)
+        logger.info("Successfully logged to WandB")
+
+
+class CometMLTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `comet_ml`. Should be initialized at the start of your script.
+
+    API keys must be stored in a Comet config file.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run.
+    """
+
+    log_directory = None
body: We should remove this from the abtract class as it's not required for all of the trackers.
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
body: `logging_dir` and `run_name` are actually the same when it comes to `tensorboard`, and we can't have both. So something like this won't work: 
```python
writerC = SummaryWriter("Testme", comment="LR_0.1_BATCH_16")
```
So we can just remove this completely. 
diff_hunk: @@ -0,0 +1,210 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Expectation:
+# Provide a project dir name, then each type of logger gets stored in project/{`logging_dir`}
+
+import logging
+from abc import ABCMeta, abstractmethod
+from pathlib import Path
+from typing import Optional
+
+from .utils import LoggerType, is_comet_ml_available, is_tensorboard_available, is_wandb_available
+
+
+_available_trackers = []
+
+if is_tensorboard_available():
+    from torch.utils import tensorboard
+
+    _available_trackers.append(LoggerType.TENSORBOARD)
+
+if is_wandb_available():
+    import wandb
+
+    _available_trackers.append(LoggerType.WANDB)
+
+if is_comet_ml_available():
+    from comet_ml import Experiment
+
+    _available_trackers.append(LoggerType.COMETML)
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_available_trackers():
+    "Returns a list of all supported available trackers in the system"
+    return _available_trackers
+
+
+class GeneralTracker(object, metaclass=ABCMeta):
+    """
+    A base Tracker class to be used for all logging integration implementations.
+    """
+
+    log_directory: str
+
+    @abstractmethod
+    def store_init_configuration(self, values):
+        pass
+
+    @abstractmethod
+    def log(self, values, step: Optional[int]):
+        pass
+
+
+class TensorBoardTracker(GeneralTracker):
+    """
+    A `Tracker` class that supports `tensorboard`. Should be initialized at the start of your script.
+
+    Args:
+        run_name (`str`):
+            The name of the experiment run. Logs are then stored in `tensorboard/{run_name}`
+    """
+
+    log_directory = Path("tensorboard")
+
+    def __init__(self, run_name: str):
+        self.run_name = run_name
+        self.writer = tensorboard.SummaryWriter(self.logging_dir / self.run_name)
body: Not following why we have to add TensoFlow here. Can't it be just `tensorboard`?
diff_hunk: @@ -21,6 +21,8 @@
 extras["test"] = [
     "pytest",
     "pytest-xdist",
+    "tensorflow",
body: It would probably be best to use a temporary directory here, to avoid users running the tests locally getting this folder.
diff_hunk: @@ -0,0 +1,89 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import shutil
+import unittest
+from pathlib import Path
+
+# We use TF to parse the logs
+import tensorflow as tf
+from accelerate import Accelerator
+from tensorboard.plugins.hparams import plugin_data_pb2
+from tensorflow.core.util import event_pb2
+from tensorflow.python.summary.summary_iterator import summary_iterator
+
+
+logger = logging.getLogger(__name__)
+
+
+class TensorBoardTrackingTest(unittest.TestCase):
+    @classmethod
+    def teardown_class(cls):
+        shutil.rmtree("test_project_with_config")
+        shutil.rmtree("test_project_with_log")
+
+    def test_init_trackers(self):
+        hps = None
+        project_name = "test_project_with_config"
body: Same here.
diff_hunk: @@ -0,0 +1,89 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import shutil
+import unittest
+from pathlib import Path
+
+# We use TF to parse the logs
+import tensorflow as tf
+from accelerate import Accelerator
+from tensorboard.plugins.hparams import plugin_data_pb2
+from tensorflow.core.util import event_pb2
+from tensorflow.python.summary.summary_iterator import summary_iterator
+
+
+logger = logging.getLogger(__name__)
+
+
+class TensorBoardTrackingTest(unittest.TestCase):
+    @classmethod
+    def teardown_class(cls):
+        shutil.rmtree("test_project_with_config")
+        shutil.rmtree("test_project_with_log")
+
+    def test_init_trackers(self):
+        hps = None
+        project_name = "test_project_with_config"
+        accelerator = Accelerator(log_with="tensorboard")
+        config = {"num_iterations": 12, "learning_rate": 1e-2, "some_boolean": False, "some_string": "some_value"}
+        accelerator.init_trackers(project_name, config)
+        for child in Path(project_name).glob("*/**"):
+            log = list(filter(lambda x: x.is_file(), child.iterdir()))[0]
+            # The config log is stored one layer deeper in the logged directory
+            # And names are randomly generated each time
+        si = summary_iterator(str(log))
+        # Pull HPS through careful parsing
+        for event in si:
+            for value in event.summary.value:
+                proto_bytes = value.metadata.plugin_data.content
+                plugin_data = plugin_data_pb2.HParamsPluginData.FromString(proto_bytes)
+                if plugin_data.HasField("session_start_info"):
+                    hps = dict(plugin_data.session_start_info.hparams)
+
+        self.assertTrue(isinstance(hps, dict))
+        keys = list(hps.keys())
+        keys.sort()
+        self.assertEqual(keys, ["learning_rate", "num_iterations", "some_boolean", "some_string"])
+        self.assertEqual(hps["num_iterations"].number_value, 12)
+        self.assertEqual(hps["learning_rate"].number_value, 0.01)
+        self.assertEqual(hps["some_boolean"].bool_value, False)
+        self.assertEqual(hps["some_string"].string_value, "some_value")
+
+    def test_log(self):
+        step = None
+        project_name = "test_project_with_log"
body: Is it why the TF dep is necessary? There is really no other way to load a TensorBoard log?
diff_hunk: @@ -0,0 +1,89 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import shutil
+import unittest
+from pathlib import Path
+
+# We use TF to parse the logs
+import tensorflow as tf
+from accelerate import Accelerator
+from tensorboard.plugins.hparams import plugin_data_pb2
+from tensorflow.core.util import event_pb2
+from tensorflow.python.summary.summary_iterator import summary_iterator
+
+
+logger = logging.getLogger(__name__)
+
+
+class TensorBoardTrackingTest(unittest.TestCase):
+    @classmethod
+    def teardown_class(cls):
+        shutil.rmtree("test_project_with_config")
+        shutil.rmtree("test_project_with_log")
+
+    def test_init_trackers(self):
+        hps = None
+        project_name = "test_project_with_config"
+        accelerator = Accelerator(log_with="tensorboard")
+        config = {"num_iterations": 12, "learning_rate": 1e-2, "some_boolean": False, "some_string": "some_value"}
+        accelerator.init_trackers(project_name, config)
+        for child in Path(project_name).glob("*/**"):
+            log = list(filter(lambda x: x.is_file(), child.iterdir()))[0]
+            # The config log is stored one layer deeper in the logged directory
+            # And names are randomly generated each time
+        si = summary_iterator(str(log))
+        # Pull HPS through careful parsing
+        for event in si:
+            for value in event.summary.value:
+                proto_bytes = value.metadata.plugin_data.content
+                plugin_data = plugin_data_pb2.HParamsPluginData.FromString(proto_bytes)
+                if plugin_data.HasField("session_start_info"):
+                    hps = dict(plugin_data.session_start_info.hparams)
+
+        self.assertTrue(isinstance(hps, dict))
+        keys = list(hps.keys())
+        keys.sort()
+        self.assertEqual(keys, ["learning_rate", "num_iterations", "some_boolean", "some_string"])
+        self.assertEqual(hps["num_iterations"].number_value, 12)
+        self.assertEqual(hps["learning_rate"].number_value, 0.01)
+        self.assertEqual(hps["some_boolean"].bool_value, False)
+        self.assertEqual(hps["some_string"].string_value, "some_value")
+
+    def test_log(self):
+        step = None
+        project_name = "test_project_with_log"
+        accelerator = Accelerator(log_with="tensorboard")
+        accelerator.init_trackers(project_name)
+        values = {"total_loss": 0.1, "iteration": 1, "my_text": "some_value"}
+        accelerator.log(values, step=0)
+        # Logged values are stored in the outermost-tfevents file and can be read in as a TFRecord
+        # Names are randomly generated each time
+        log = list(filter(lambda x: x.is_file(), Path(project_name).iterdir()))[0]
+        serialized_examples = tf.data.TFRecordDataset(log)
body: Yup, sadly: https://github.com/tensorflow/tensorboard#how-can-i-export-data-from-tensorboard

We have to use `tf` to parse it back in. 
diff_hunk: @@ -0,0 +1,89 @@
+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import shutil
+import unittest
+from pathlib import Path
+
+# We use TF to parse the logs
+import tensorflow as tf
+from accelerate import Accelerator
+from tensorboard.plugins.hparams import plugin_data_pb2
+from tensorflow.core.util import event_pb2
+from tensorflow.python.summary.summary_iterator import summary_iterator
+
+
+logger = logging.getLogger(__name__)
+
+
+class TensorBoardTrackingTest(unittest.TestCase):
+    @classmethod
+    def teardown_class(cls):
+        shutil.rmtree("test_project_with_config")
+        shutil.rmtree("test_project_with_log")
+
+    def test_init_trackers(self):
+        hps = None
+        project_name = "test_project_with_config"
+        accelerator = Accelerator(log_with="tensorboard")
+        config = {"num_iterations": 12, "learning_rate": 1e-2, "some_boolean": False, "some_string": "some_value"}
+        accelerator.init_trackers(project_name, config)
+        for child in Path(project_name).glob("*/**"):
+            log = list(filter(lambda x: x.is_file(), child.iterdir()))[0]
+            # The config log is stored one layer deeper in the logged directory
+            # And names are randomly generated each time
+        si = summary_iterator(str(log))
+        # Pull HPS through careful parsing
+        for event in si:
+            for value in event.summary.value:
+                proto_bytes = value.metadata.plugin_data.content
+                plugin_data = plugin_data_pb2.HParamsPluginData.FromString(proto_bytes)
+                if plugin_data.HasField("session_start_info"):
+                    hps = dict(plugin_data.session_start_info.hparams)
+
+        self.assertTrue(isinstance(hps, dict))
+        keys = list(hps.keys())
+        keys.sort()
+        self.assertEqual(keys, ["learning_rate", "num_iterations", "some_boolean", "some_string"])
+        self.assertEqual(hps["num_iterations"].number_value, 12)
+        self.assertEqual(hps["learning_rate"].number_value, 0.01)
+        self.assertEqual(hps["some_boolean"].bool_value, False)
+        self.assertEqual(hps["some_string"].string_value, "some_value")
+
+    def test_log(self):
+        step = None
+        project_name = "test_project_with_log"
+        accelerator = Accelerator(log_with="tensorboard")
+        accelerator.init_trackers(project_name)
+        values = {"total_loss": 0.1, "iteration": 1, "my_text": "some_value"}
+        accelerator.log(values, step=0)
+        # Logged values are stored in the outermost-tfevents file and can be read in as a TFRecord
+        # Names are randomly generated each time
+        log = list(filter(lambda x: x.is_file(), Path(project_name).iterdir()))[0]
+        serialized_examples = tf.data.TFRecordDataset(log)
body: ```suggestion
        Should run any finalizing functions within the tracking API. If the API should not have one, just don't overwrite that method.
```
diff_hunk: @@ -80,6 +80,16 @@ def log(self, values: dict, step: Optional[int]):
         """
         pass
 
+    @abstractmethod
+    def finish(self):
+        """
+        Should run any finalizing functions within the tracking API. If the API should not have one, just return:
+        ```python
+        super().finish()
+        ```
body: ```suggestion
```
No need to write something as calling the super method is then the default ;-)
diff_hunk: @@ -218,3 +242,7 @@ def log(self, values: dict, step: Optional[int] = None):
             self.writer.set_step(step)
         self.writer.log_others(values)
         logger.info("Successfully logged to CometML")
+
+    def finish(self):
+        """Do nothing"""
+        super().finish()
body: I'd like to avoid this. The API should accept any logging dir for TensorBoard at least, no?
diff_hunk: @@ -29,29 +32,29 @@
 
 
 class TensorBoardTrackingTest(unittest.TestCase):
-    @classmethod
-    def teardown_class(cls):
-        shutil.rmtree("test_project_with_config")
-        shutil.rmtree("test_project_with_log")
-
     def test_init_trackers(self):
         hps = None
         project_name = "test_project_with_config"
-        accelerator = Accelerator(log_with="tensorboard")
-        config = {"num_iterations": 12, "learning_rate": 1e-2, "some_boolean": False, "some_string": "some_value"}
-        accelerator.init_trackers(project_name, config)
-        for child in Path(project_name).glob("*/**"):
-            log = list(filter(lambda x: x.is_file(), child.iterdir()))[0]
-            # The config log is stored one layer deeper in the logged directory
-            # And names are randomly generated each time
-        si = summary_iterator(str(log))
-        # Pull HPS through careful parsing
-        for event in si:
-            for value in event.summary.value:
-                proto_bytes = value.metadata.plugin_data.content
-                plugin_data = plugin_data_pb2.HParamsPluginData.FromString(proto_bytes)
-                if plugin_data.HasField("session_start_info"):
-                    hps = dict(plugin_data.session_start_info.hparams)
+        with tempfile.TemporaryDirectory() as dirpath:
+            oldpwd = os.getcwd()
+            os.chdir(dirpath)
body: Would be nice to have a `logging_dir` argument in the `Accelerator` I realize.
diff_hunk: @@ -64,26 +67,90 @@ def test_init_trackers(self):
 
     def test_log(self):
         step = None
+        project_name = "test_project_with_log"
+        with tempfile.TemporaryDirectory() as dirpath:
+            oldpwd = os.getcwd()
+            os.chdir(dirpath)
+            accelerator = Accelerator(log_with="tensorboard")
+            accelerator.init_trackers(project_name)
+            values = {"total_loss": 0.1, "iteration": 1, "my_text": "some_value"}
+            accelerator.log(values, step=0)
+            accelerator.end_training()
+            # Logged values are stored in the outermost-tfevents file and can be read in as a TFRecord
+            # Names are randomly generated each time
+            log = list(filter(lambda x: x.is_file(), Path(project_name).iterdir()))[0]
+            serialized_examples = tf.data.TFRecordDataset(log)
+            for e in serialized_examples:
+                event = event_pb2.Event.FromString(e.numpy())
+                if step is None:
+                    step = event.step
+                for value in event.summary.value:
+                    if value.tag == "total_loss":
+                        total_loss = value.simple_value
+                    elif value.tag == "iteration":
+                        iteration = value.simple_value
+                    elif value.tag == "my_text/text_summary":  # Append /text_summary to the key
+                        my_text = value.tensor.string_val[0].decode()
+            os.chdir(oldpwd)
+        self.assertAlmostEqual(total_loss, values["total_loss"])
+        self.assertEqual(iteration, values["iteration"])
+        self.assertEqual(my_text, values["my_text"])
+
+
+@mock.patch.dict(os.environ, {"WANDB_MODE": "offline"})
+class WandBTrackingTest(unittest.TestCase):
+    @staticmethod
+    def get_value_from_log(key: str, log: str, key_occurance: int = 0):
+        """
+        Parses wandb log for `key` and returns the value.
+        If parsing through multiple calls to .log, pass in a `key_occurance`
+        """
+        res = re.findall(rf"(?<={key} )[^\s]+", log)[key_occurance]
+        if '"' in res:
+            return re.findall(r'"([^"]*)"', res)[0]
+        else:
+            return res
+
+    def test_init_trackers(self):
+        project_name = "test_project_with_config"
+        with tempfile.TemporaryDirectory() as dirpath:
+            oldpwd = os.getcwd()
+            os.chdir(dirpath)
body: No need to do a separate job, they can all go in the same one!
diff_hunk: @@ -12,6 +12,19 @@ jobs:
       with:
         python-version: 3.6
     - name: Install Python dependencies
-      run: pip install -e .[test]
+      run: pip install -e .[test,trackers]
     - name: Run Tests
-      run: make test
\ No newline at end of file
+      run: make test
+
+  test-tf:
body: We should mention the train end method in this example, since it's relevant for trackers.
diff_hunk: @@ -0,0 +1,137 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Tracking
+
+There are a large number of experiment tracking API's available, however getting them all to work with in a multi-processing environment can oftentimes be complex.
+Accelerate provides a general tracking API that can be used to log useful items during your script through [`~Accelerator.log`]
+
+## Integrated Trackers
+
+Currently `Accelerate` supports three trackers out-of-the-box:
+
+
+[[autodoc]] tracking.TensorBoardTracker
+
+[[autodoc]] tracking.WandBTracker
+
+[[autodoc]] tracking.CometMLTracker
+
+To use any of them, pass in the selected type(s) to the `log_with` parameter in [`~Accelerate.__init__`]:
+```python
+from accelerate import Accelerate
+from accelerate.utils import LoggerType
+
+accelerator = Accelerate(log_with="all")  # For all available trackers in the environment
+accelerator = Accelerate(log_with="wandb")
+accelerator = Accelerate(log_with=["wandb", LoggerType.TENSORBOARD])
+```
+
+At the start of your experiment [`~Accelerator.init_tracker`] should be used to setup your project, and potentially add any experiment hyperparameters to be logged:
+```python
+hps = {"num_iterations": 5, "learning_rate": 1e-2}
+accelerator.init_trackers("my_project", config=hps)
+```
+
+When you are ready to log any data, [`~Accelerator.log`] should be used.
+A `step` can also be passed in to correlate the data with a particular step in the training loop.
+```python
+accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=1)
+```
+
+A full example is below:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator(log_with="all")
+config = {
+    "num_iterations": 5,
+    "learning_rate": 1e-2,
+    "loss_function": str(my_loss_function),
+}
+
+accelerator.init_trackers("example_project", config=config)
+
+my_model, my_optimizer, my_training_dataloader = accelerate.prepare(my_model, my_optimizer, my_training_dataloader)
+device = accelerator.device
+my_model.to(device)
+
+for iteration in config["num_iterations"]:
+    for step, batch in my_training_dataloader:
+        my_optimizer.zero_grad()
+        inputs, targets = batch
+        inputs = inputs.to(device)
+        targets = targets.to(device)
+        outputs = my_model(inputs)
+        loss = my_loss_function(outputs, targets)
+        accelerator.backward(loss)
+        my_optimizer.step()
+        accelerator.log({"training_loss": loss}, step=step)
+```
body: I would group those together.
diff_hunk: @@ -20,9 +20,11 @@
 extras["docs"] = []
 extras["test"] = [
     "pytest",
-    "pytest-xdist",
+    "pytest-xdist"
 ]
-extras["dev"] = extras["quality"] + extras["test"]
+extras["trackers"] = ["wandb", "comet-ml"]
+extras["test_trackers"] = ["tensorflow"]
body: ```suggestion
extras["dev"] = extras["quality"] + extras["test"]
```
Please let's not include all the trackers in dev ;-)
diff_hunk: @@ -20,9 +20,11 @@
 extras["docs"] = []
 extras["test"] = [
     "pytest",
-    "pytest-xdist",
+    "pytest-xdist"
 ]
-extras["dev"] = extras["quality"] + extras["test"]
+extras["trackers"] = ["wandb", "comet-ml"]
+extras["test_trackers"] = ["tensorflow"]
+extras["dev"] = extras["quality"] + extras["test"] + extras["trackers"]

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: Given https://github.com/huggingface/accelerate/issues/291#issuecomment-1080771134, this PR is feature-complete. Only thing missing is tests for TensorBoard and potential mocking, and other documentation. But otherwise it's ready for a review
body: Last review, this time for the docs! 😄 🚀 
body: There's a weird issue when doing `pip install tensorflow` where it will just install a higher `setuptools`, and that setuptools will break importing pytorch. pinning it in the dev requirements "fixes" it. Need to play with it more, but a test is there. Once that's solved we probably should mimic something like this for the TF tests as well, since this goes through and looks at the actual values stored