diff --git a/README.md b/README.md
index deddf2986..7b912dd80 100644
--- a/README.md
+++ b/README.md
@@ -243,6 +243,7 @@ pip install accelerate
 - FP16 with native AMP (apex on the roadmap)
 - DeepSpeed support (Experimental)
 - PyTorch Fully Sharded Data Parallel (FSDP) support (Experimental)
+- Megatron-LM support (Experimental)
 
 ## Citing 🤗 Accelerate
 
diff --git a/docs/source/usage_guides/megatron_lm.mdx b/docs/source/usage_guides/megatron_lm.mdx
index 188dddb32..bc0b90701 100644
--- a/docs/source/usage_guides/megatron_lm.mdx
+++ b/docs/source/usage_guides/megatron_lm.mdx
@@ -23,18 +23,45 @@ For detailed information and how things work behind the scene please refer the g
 Accelerate integrates following feature of Megatron-LM to enable large scale pre-training/finetuning
 of BERT (Encoder), GPT (Decoder) or T5 models (Encoder and Decoder):
 
-a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks
+a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks.
+Each tensor is split into multiple chunks with each shard residing on separate GPU. At each step, the same mini-batch of data is processed
+independently and in parallel by each shard followed by syncing across all GPUs (`all-reduce` operation). 
+In a simple transformer layer, this leads to 2 `all-reduces` in the forward path and 2 in the backward path.
+For more details, please refer research paper [Megatron-LM: Training Multi-Billion Parameter Language Models Using
+Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#tensor-parallelism).
+
 
 b. **Pipeline Parallelism (PP)**: Reduces memory footprint and enables large scale training via inter-node parallelization. 
 Reduces the bubble of naive PP via PipeDream-Flush schedule/1F1B schedule and Interleaved 1F1B schedule. 
-Layers are distributed uniformly across PP stages.
+Layers are distributed uniformly across PP stages. For example, if a model has `24` layers and we have `4` GPUs for
+pipeline parallelism, each GPU will have `6` layers (24/4). For more details on schedules to reduce the idle time of PP,
+please refer to the research paper [Efficient Large-Scale Language Model Training on GPU Clusters
+Using Megatron-LM](https://arxiv.org/pdf/2104.04473.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#pipeline-parallelism).
 
 c. **Sequence Parallelism (SP)**: Reduces memory footprint without any additional communication. Only applicable when using TP.
-
-d. **Data Parallelism (DP)** via Distributed Optimizer: Reduces memory footprint by sharding optimizer states and gradients across DP ranks.
+It reduces activation memory required as it prevents the same copies to be on the tensor parallel ranks 
+post `all-reduce` by replacing then with `reduce-scatter` and `no-op` operation would be replaced by `all-gather`. 
+As `all-reduce = reduce-scatter + all-gather`, this saves a ton of activation memory at no added communication cost. 
+To put it simply, it shards the outputs of each transformer layer along sequence dimension, e.g., 
+if the sequence length is `1024` and the TP size is `4`, each GPU will have `256` tokens (1024/4) for each sample. 
+This increases the batch size that can be supported for training. For more details, please refer to the research paper
+[Reducing Activation Recomputation in Large Transformer Models](https://arxiv.org/pdf/2205.05198.pdf). 
+
+d. **Data Parallelism (DP)** via Distributed Optimizer: Reduces the memory footprint by sharding optimizer states and gradients across DP ranks
+(versus the traditional method of replicating the optimizer state across data parallel ranks). 
+For example, when using Adam optimizer with mixed-precision training, each parameter accounts for 12 bytes of memory.
+This gets distributed equally across the GPUs, i.e., each parameter would account for 3 bytes (12/4) if we have 4 GPUs.
+For more details, please refer the research paper [ZeRO: Memory Optimizations Toward Training Trillion
+Parameter Models](https://arxiv.org/pdf/1910.02054.pdf) and following section of 🤗 blog 
+[The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#zero-data-parallelism).
 
 e. **Selective Activation Recomputation**: Reduces the memory footprint of activations significantly via smart activation checkpointing.
 It doesn't store activations occupying large memory while being fast to recompute thereby achieving great tradeoff between memory and recomputation.
+For example, for GPT-3, this leads to 70% reduction in required memory for activations at the expense of
+only 2.7% FLOPs overhead for recomputation of activations. For more details, please refer to the research paper 
+[Reducing Activation Recomputation in Large Transformer Models](https://arxiv.org/pdf/2205.05198.pdf).
 
 f. **Fused Kernels**: Fused Softmax, Mixed Precision Fused Layer Norm and  Fused gradient accumulation to weight gradient computation of linear layer.
 PyTorch JIT compiled Fused GeLU and Fused Bias+Dropout+Residual addition.
diff --git a/docs/source/usage_guides/training_zoo.mdx b/docs/source/usage_guides/training_zoo.mdx
index cc388acac..444784b49 100644
--- a/docs/source/usage_guides/training_zoo.mdx
+++ b/docs/source/usage_guides/training_zoo.mdx
@@ -36,6 +36,7 @@ These examples showcase specific features that the Accelerate framework offers
 - [Memory-aware batch size finder](https://github.com/huggingface/accelerate/blob/main/examples/by_feature/memory.py)
 - [Metric Computation](https://github.com/huggingface/accelerate/blob/main/examples/by_feature/multi_process_metrics.py)
 - [Using Trackers](https://github.com/huggingface/accelerate/blob/main/examples/by_feature/tracking.py)
+- [Using Megatron-LM](https://github.com/huggingface/accelerate/blob/main/examples/by_feature/megatron_lm_gpt_pretraining.py)
 
 ### Full Examples 
 
@@ -55,6 +56,8 @@ These examples showcase every feature in Accelerate at once that was shown in "F
 - [Named entity recognition fine-tuning example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner_no_trainer.py)
 - [Image classification fine-tuning example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification_no_trainer.py)
 - [Summarization fine-tuning example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization_no_trainer.py)
+- [End-to-end examples on how to use AWS SageMaker integration of Accelerate](https://github.com/huggingface/notebooks/blob/main/sagemaker/22_accelerate_sagemaker_examples/README.md)
+- [Megatron-LM examples for various NLp tasks](https://github.com/pacman100/accelerate-megatron-test) 
 
 ## Integration Examples 
 


Review comments: body: ```suggestion
Each tensor is split into multiple chunks with each shard residing on separate GPU. At each step, the same mini-batch of data is processed
```
diff_hunk: @@ -23,18 +23,45 @@ For detailed information and how things work behind the scene please refer the g
 Accelerate integrates following feature of Megatron-LM to enable large scale pre-training/finetuning
 of BERT (Encoder), GPT (Decoder) or T5 models (Encoder and Decoder):
 
-a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks
+a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks.
+Each tensor are split into multiple chunks with each shard residing on separate GPU. At each step, same minimbatch of data is processed
body: ```suggestion
It reduces activation memory required as it prevents the same copies to be on the tensor parallel ranks 
```
diff_hunk: @@ -23,18 +23,45 @@ For detailed information and how things work behind the scene please refer the g
 Accelerate integrates following feature of Megatron-LM to enable large scale pre-training/finetuning
 of BERT (Encoder), GPT (Decoder) or T5 models (Encoder and Decoder):
 
-a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks
+a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks.
+Each tensor are split into multiple chunks with each shard residing on separate GPU. At each step, same minimbatch of data is processed
+independently and in parallel by each shard followed by syncing across all GPUs (`all-reduce` operation). 
+In a simple transformer layer, this leads to 2 `all-reduces` in the forward path and 2 in the backward path.
+For more details, please refer research paper [Megatron-LM: Training Multi-Billion Parameter Language Models Using
+Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#tensor-parallelism).
+
 
 b. **Pipeline Parallelism (PP)**: Reduces memory footprint and enables large scale training via inter-node parallelization. 
 Reduces the bubble of naive PP via PipeDream-Flush schedule/1F1B schedule and Interleaved 1F1B schedule. 
-Layers are distributed uniformly across PP stages.
+Layers are distributed uniformly across PP stages. For example, if a model has `24` layers and we have `4` GPUs for
+pipeline parallelism, each GPU will have `6` layers (24/4). For more details on schedules to reduce the idle time of PP,
+please refer to the research paper [Efficient Large-Scale Language Model Training on GPU Clusters
+Using Megatron-LM](https://arxiv.org/pdf/2104.04473.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#pipeline-parallelism).
 
 c. **Sequence Parallelism (SP)**: Reduces memory footprint without any additional communication. Only applicable when using TP.
-
-d. **Data Parallelism (DP)** via Distributed Optimizer: Reduces memory footprint by sharding optimizer states and gradients across DP ranks.
+It reduced activation memory required as it prevents same copies to be on the tensor parallel ranks 
body: ```suggestion
post `all-reduce` by replacing then with `reduce-scatter` and `no-op` operation would be replaced by `all-gather`. 
```
diff_hunk: @@ -23,18 +23,45 @@ For detailed information and how things work behind the scene please refer the g
 Accelerate integrates following feature of Megatron-LM to enable large scale pre-training/finetuning
 of BERT (Encoder), GPT (Decoder) or T5 models (Encoder and Decoder):
 
-a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks
+a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks.
+Each tensor are split into multiple chunks with each shard residing on separate GPU. At each step, same minimbatch of data is processed
+independently and in parallel by each shard followed by syncing across all GPUs (`all-reduce` operation). 
+In a simple transformer layer, this leads to 2 `all-reduces` in the forward path and 2 in the backward path.
+For more details, please refer research paper [Megatron-LM: Training Multi-Billion Parameter Language Models Using
+Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#tensor-parallelism).
+
 
 b. **Pipeline Parallelism (PP)**: Reduces memory footprint and enables large scale training via inter-node parallelization. 
 Reduces the bubble of naive PP via PipeDream-Flush schedule/1F1B schedule and Interleaved 1F1B schedule. 
-Layers are distributed uniformly across PP stages.
+Layers are distributed uniformly across PP stages. For example, if a model has `24` layers and we have `4` GPUs for
+pipeline parallelism, each GPU will have `6` layers (24/4). For more details on schedules to reduce the idle time of PP,
+please refer to the research paper [Efficient Large-Scale Language Model Training on GPU Clusters
+Using Megatron-LM](https://arxiv.org/pdf/2104.04473.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#pipeline-parallelism).
 
 c. **Sequence Parallelism (SP)**: Reduces memory footprint without any additional communication. Only applicable when using TP.
-
-d. **Data Parallelism (DP)** via Distributed Optimizer: Reduces memory footprint by sharding optimizer states and gradients across DP ranks.
+It reduced activation memory required as it prevents same copies to be on the tensor parallel ranks 
+post `all-reduce` by replacing it with `reduce-scatter` and `no-op` operation would be replaced by `all-gather`. 
body: ```suggestion
d. **Data Parallelism (DP)** via Distributed Optimizer: Reduces the memory footprint by sharding optimizer states and gradients across DP ranks
```
diff_hunk: @@ -23,18 +23,45 @@ For detailed information and how things work behind the scene please refer the g
 Accelerate integrates following feature of Megatron-LM to enable large scale pre-training/finetuning
 of BERT (Encoder), GPT (Decoder) or T5 models (Encoder and Decoder):
 
-a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks
+a. **Tensor Parallelism (TP)**: Reduces memory footprint without much additional communication on intra-node ranks.
+Each tensor are split into multiple chunks with each shard residing on separate GPU. At each step, same minimbatch of data is processed
+independently and in parallel by each shard followed by syncing across all GPUs (`all-reduce` operation). 
+In a simple transformer layer, this leads to 2 `all-reduces` in the forward path and 2 in the backward path.
+For more details, please refer research paper [Megatron-LM: Training Multi-Billion Parameter Language Models Using
+Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#tensor-parallelism).
+
 
 b. **Pipeline Parallelism (PP)**: Reduces memory footprint and enables large scale training via inter-node parallelization. 
 Reduces the bubble of naive PP via PipeDream-Flush schedule/1F1B schedule and Interleaved 1F1B schedule. 
-Layers are distributed uniformly across PP stages.
+Layers are distributed uniformly across PP stages. For example, if a model has `24` layers and we have `4` GPUs for
+pipeline parallelism, each GPU will have `6` layers (24/4). For more details on schedules to reduce the idle time of PP,
+please refer to the research paper [Efficient Large-Scale Language Model Training on GPU Clusters
+Using Megatron-LM](https://arxiv.org/pdf/2104.04473.pdf) and 
+this section of 🤗 blogpost [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed#pipeline-parallelism).
 
 c. **Sequence Parallelism (SP)**: Reduces memory footprint without any additional communication. Only applicable when using TP.
-
-d. **Data Parallelism (DP)** via Distributed Optimizer: Reduces memory footprint by sharding optimizer states and gradients across DP ranks.
+It reduced activation memory required as it prevents same copies to be on the tensor parallel ranks 
+post `all-reduce` by replacing it with `reduce-scatter` and `no-op` operation would be replaced by `all-gather`. 
+As `all-reduce = reduce-scatter + all-gather`, this saves a ton of activation memory at no added communication cost. 
+To put it simply, it shards the outputs of each transformer layer along sequence dimension, e.g., 
+if the sequence length is `1024` and the TP size is `4`, each GPU will have `256` tokens (1024/4) for each sample. 
+This increases the batch size that can be supported for training. For more details, please refer to the research paper
+[Reducing Activation Recomputation in Large Transformer Models](https://arxiv.org/pdf/2205.05198.pdf). 
+
+d. **Data Parallelism (DP)** via Distributed Optimizer: Reduces memory footprint by sharding optimizer states and gradients across DP ranks

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._