elif self.fsdp is not None and self.args.fsdp_config["xla"]:
Author: duanzhenyu001
Pull request: no
Labels: name: solved

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.4.143.bsk.8-amd64-x86_64-with-glibc2.28
- Python version: 3.10.9
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

in trainer if self.args.fsdp_config["xla"] is true，trainer will wrap model layers，is this mean if i want to use fsdp to shard model，I must install torch-xla>2.0?

### Expected behavior

want to know if  i must install torch-xla for fspd trainning

Comments:
Hi @duanzhenyu001, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

That being said, checking the docs it's possible to see that [XLA supports FDSP](https://huggingface.co/docs/transformers/main/main_classes/trainer#pytorchxla-fully-sharded-data-parallel), but [FDSP can be used separately](https://huggingface.co/docs/transformers/main/main_classes/trainer#pytorch-fully-sharded-data-parallel). If you wish to use XLA with FDSP, it's necessary to install torch-xla >= 2.0.  

@amyeroberts thanks a lot for your reply. when I use huggingface trainer fsdp mode finetune a 6B model，I got oom even 8 V100-32G gpu used. I didn't find where trainer shard my model when I use fsdp without xla. could you please point it out in the trainer code. thanks very much.   

@amyeroberts  and here is my TrainingArguments when I finetune glm2 a 6B model.
TrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
bf16=False,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_backend=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=64,
eval_delay=0,
eval_steps=None,
evaluation_strategy=epoch,
fp16=True,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=full_shard auto_wrap,
fsdp_config={'fsdp_forward_prefetch': True, 'fsdp_sync_module_states': True, 'fsdp_use_orig_params': True, 'xla': False, 'fsdp_transformer_layer_cls_to_wrap': 'GLMBlock'},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=GLMBlock,
full_determinism=False,
gradient_accumulation_steps=128,
gradient_checkpointing=False,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=2e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=1,
log_level=debug,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/mnt/bn/mods-llm/duanzhenyu/llm/ds_llm_sft/data/models/finetune/runs/Jul14_06-57-19_mlxlab25ta1apm6482c909-20230609063905-cqjk4l-u3z93n-worker,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1,
logging_strategy=steps,
lr_scheduler_type=cosine,
max_grad_norm=None,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=2,
optim=adamw_torch,
optim_args=None,
output_dir=/mnt/bn/mods-llm/duanzhenyu/llm/ds_llm_sft/data/models/finetune,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=/mnt/bn/mods-llm/duanzhenyu/llm/ds_llm_sft/data/models/finetune,
save_on_each_node=False,
save_safetensors=False,
save_steps=50,
save_strategy=steps,
save_total_limit=1,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=1000,
weight_decay=0.1,
xpu_backend=None,
)

@amyeroberts and at function "create_accelerator_and_postprocess" , 
        self.accelerator = Accelerator(
            deepspeed_plugin=self.args.deepspeed_plugin,
            gradient_accumulation_steps=self.args.gradient_accumulation_steps,
        )
self.accelerator always be inited, if i don't set os env ACCELERATE_USE_FSDP to be true, "self.is_fsdp_enabled = getattr(self.accelerator.state, "fsdp_plugin", None) is not None"  self.is_fsdp_enabled will always be false,  is this what you realy wanted?  and I'm confused the intention of this function.

cc @pacman100 who will be able to comment on the OOM memory issues and whether this is expected. 

Please note that recently there was a large update with Trainer, and it now uses accelerate in the background. In the issue information, I see that you're using v4.24. I suggest updating to a more recent release to benefit from this update and any bug resolutions in between. 

> 

I'm now using v4.30.2，but still get oom memory errors when training on 8 V100-32g GPUs. @pacman100  can some one give any suggestions? thanks a lot

Hello @duanzhenyu001, how are you launching the training script and a minimal training script for deep dive is required

The following works on 4 A100 80GB GPUs with GPT-j (6B model), although the entire VRAM is occupied on each:

```
cd transformers/examples/pytorch/language-modeling

torchrun --nnodes 1 --nproc-per-node 4 run_clm.py   --model_name_or_path EleutherAI/gpt-j-6b --dataset_name wikitext   --dataset_config_name wikitext-2-raw-v1   --per_device_train_batch_size 1   --per_device_eval_batch_size 1   --do_train   --do_eval   --output_dir /tmp/test-clm   --gradient_accumulation_steps 8 --overwrite_output_dir --fsdp "full_shard auto_wrap" --fsdp_transformer_layer_cls_to_wrap "GPTJBlock" --bf16
```

The reason for OOM at your end is plausible due to large seq lengths such as >=1024. In such cases, gradient/activation checkpointing is recommended with `--gradient_checkpointing`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.