dataset.transform() hangs indefinitely while finetuning the stable diffusion XL
Opened by bhosalems (closed)

### Describe the bug

Multi-GPU fine-tuning the stable diffusion X by following https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/README_sdxl.md hangs indefinitely.

### Steps to reproduce the bug

accelerate launch train_text_to_image_sdxl.py --pretrained_model_name_or_path=$MODEL_NAME --pretrained_vae_model_name_or_path=$VAE_NAME --dataset_name=$DATASET_NAME --enable_xformers_memory_efficient_attention --resolution=512 --center_crop --random_flip --proportion_empty_prompts=0.2 --train_batch_size=1 --gradient_accumulation_steps=4 --gradient_checkpointing --max_train_steps=10000 --use_8bit_adam --learning_rate=1e-06 --lr_scheduler="constant" --lr_warmup_steps=0 --mixed_precision="fp16" --report_to="wandb" --validation_prompt="a cute Sundar Pichai creature" --validation_epochs 5 --checkpointing_steps=5000 --output_dir="sdxl-pokemon-model"

### Expected behavior

It should start the training as it does for the single GPU training. I opened the issue in diffusers **https://github.com/huggingface/diffusers/issues/5534 but it does seem to be an issue with the Pokemon dataset.

I added some debug prints

  ```
  print("==========HERE3=============")
    with accelerator.main_process_first():
        print(accelerator.is_main_process)
        print("===========Here3.1===========")
        if args.max_train_samples is not None:
            dataset["train"] = dataset["train"].shuffle(seed=args.seed).select(range(args.max_train_samples))
        print("===========Here3.2===========")
        # Set the training transforms
        train_dataset = dataset["train"].with_transform(preprocess_train)
    print("==========HERE4=============")


Corresponding Output

Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
10/25/2023 21:18:04 - INFO - main - Distributed environment: MULTI_GPU Backend: nccl
Num processes: 3
Process index: 1
Local process index: 1
Device: cuda:1

Mixed precision type: fp16

10/25/2023 21:18:04 - INFO - main - Distributed environment: MULTI_GPU Backend: nccl
Num processes: 3
Process index: 2
Local process index: 2
Device: cuda:2

Mixed precision type: fp16

10/25/2023 21:18:04 - INFO - main - Distributed environment: MULTI_GPU Backend: nccl
Num processes: 3
Process index: 0
Local process index: 0
Device: cuda:0

Mixed precision type: fp16

You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
{‘variance_type’, ‘clip_sample_range’, ‘thresholding’, ‘dynamic_thresholding_ratio’} was not found in config. Values will be initialized to default values.
{‘attention_type’, ‘reverse_transformer_layers_per_block’, ‘dropout’} was not found in config. Values will be initialized to default values.
==========HERE1=============
==========HERE1=============
==========HERE1=============
==========HERE2=============
==========HERE2=============
==========HERE2=============
==========HERE3=============
True
===========Here3.1===========
===========Here3.2===========
==========HERE3=============
==========HERE3=========
```

### Environment info

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
absl-py 2.0.0 pypi_0 pypi
accelerate 0.24.0 pypi_0 pypi
aiohttp 3.8.6 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
async-timeout 4.0.3 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
bitsandbytes 0.41.1 pypi_0 pypi
blas 1.0 mkl
blessings 1.7 py39h06a4308_1002
brotli-python 1.0.9 py39h6a678d5_7
bzip2 1.0.8 h7b6447c_0
ca-certificates 2023.08.22 h06a4308_0
cachetools 5.3.2 pypi_0 pypi
certifi 2023.7.22 py39h06a4308_0
cffi 1.15.1 py39h5eee18b_3
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.1.7 unix_pyh707e725_0 conda-forge
cryptography 41.0.3 py39hdda0065_0
cuda-cudart 11.7.99 0 nvidia
cuda-cupti 11.7.101 0 nvidia
cuda-libraries 11.7.1 0 nvidia
cuda-nvrtc 11.7.99 0 nvidia
cuda-nvtx 11.7.91 0 nvidia
cuda-runtime 11.7.1 0 nvidia
datasets 2.14.6 pypi_0 pypi
diffusers 0.22.0.dev0 pypi_0 pypi
dill 0.3.7 pypi_0 pypi
docker-pycreds 0.4.0 py_0 conda-forge
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.12.4 pypi_0 pypi
freetype 2.12.1 h4a9f257_0
frozenlist 1.4.0 pypi_0 pypi
fsspec 2023.10.0 pypi_0 pypi
ftfy 6.1.1 pypi_0 pypi
giflib 5.2.1 h5eee18b_3
gitdb 4.0.11 pyhd8ed1ab_0 conda-forge
gitpython 3.1.40 pyhd8ed1ab_0 conda-forge
gmp 6.2.1 h295c915_3
gnutls 3.6.15 he1e5248_0
google-auth 2.23.3 pypi_0 pypi
google-auth-oauthlib 1.1.0 pypi_0 pypi
gpustat 0.6.0 pyhd3eb1b0_1
grpcio 1.59.0 pypi_0 pypi
huggingface-hub 0.17.3 pypi_0 pypi
idna 3.4 py39h06a4308_0
importlib-metadata 6.8.0 pypi_0 pypi
intel-openmp 2023.1.0 hdb19cb5_46305
jinja2 3.1.2 pypi_0 pypi
jpeg 9e h5eee18b_1
lame 3.100 h7b6447c_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libcublas 11.10.3.66 0 nvidia
libcufft 10.7.2.124 h4fbf590_0 nvidia
libcufile 1.8.0.34 0 nvidia
libcurand 10.3.4.52 0 nvidia
libcusolver 11.4.0.1 0 nvidia
libcusparse 11.7.4.91 0 nvidia
libdeflate 1.17 h5eee18b_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 13.2.0 h807b86a_2 conda-forge
libgfortran-ng 13.2.0 h69a702a_2 conda-forge
libgfortran5 13.2.0 ha4646dd_2 conda-forge
libiconv 1.16 h7f8727e_2
libidn2 2.3.4 h5eee18b_0
libnpp 11.7.4.75 0 nvidia
libnvjpeg 11.8.0.2 0 nvidia
libpng 1.6.39 h5eee18b_0
libprotobuf 3.20.3 he621ea3_0
libstdcxx-ng 13.2.0 h7e041cc_2 conda-forge
libtasn1 4.19.0 h5eee18b_0
libtiff 4.5.1 h6a678d5_0
libunistring 0.9.10 h27cfd23_0
libwebp 1.3.2 h11a3e52_0
libwebp-base 1.3.2 h5eee18b_0
llvm-openmp 14.0.6 h9e868ea_0
lz4-c 1.9.4 h6a678d5_0
markdown 3.5 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
mkl 2023.1.0 h213fc3f_46343
mkl-service 2.4.0 py39h5eee18b_1
mkl_fft 1.3.8 py39h5eee18b_0
mkl_random 1.2.4 py39hdb19cb5_0
multidict 6.0.4 pypi_0 pypi
multiprocess 0.70.15 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nettle 3.7.3 hbbd107a_1
numpy 1.26.0 py39h5f9d8c6_0
numpy-base 1.26.0 py39hb5e798b_0
nvidia-ml 7.352.0 pyhd3eb1b0_0
oauthlib 3.2.2 pypi_0 pypi
openh264 2.1.1 h4ff587b_0
openjpeg 2.4.0 h3ad879b_0
openssl 3.0.11 h7f8727e_2
packaging 23.2 pypi_0 pypi
pandas 2.1.1 pypi_0 pypi
pathtools 0.1.2 py_1 conda-forge
pillow 10.0.1 py39ha6cbd5a_0
pip 23.3 py39h06a4308_0
protobuf 4.23.4 pypi_0 pypi
psutil 5.9.6 pypi_0 pypi
pyarrow 13.0.0 pypi_0 pypi
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pycparser 2.21 pyhd3eb1b0_0
pyopenssl 23.2.0 py39h06a4308_0
pysocks 1.7.1 py39h06a4308_0
python 3.9.18 h955ad1f_0
python-dateutil 2.8.2 pypi_0 pypi
python_abi 3.9 2_cp39 conda-forge
pytorch 1.13.1 py3.9_cuda11.7_cudnn8.5.0_0 pytorch
pytorch-cuda 11.7 h778d358_5 pytorch
pytorch-mutex 1.0 cuda pytorch
pytz 2023.3.post1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h5eee18b_0
regex 2023.10.3 pypi_0 pypi
requests 2.31.0 py39h06a4308_0
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
safetensors 0.4.0 pypi_0 pypi
scipy 1.11.3 py39h5f9d8c6_0
sentry-sdk 1.32.0 pyhd8ed1ab_0 conda-forge
setproctitle 1.1.10 py39h3811e60_1004 conda-forge
setuptools 68.0.0 py39h06a4308_0
six 1.16.0 pyh6c4a22f_0 conda-forge
smmap 5.0.0 pyhd8ed1ab_0 conda-forge
sqlite 3.41.2 h5eee18b_0
tbb 2021.8.0 hdb19cb5_0
tensorboard 2.15.0 pypi_0 pypi
tensorboard-data-server 0.7.2 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tokenizers 0.14.1 pypi_0 pypi
torchaudio 0.13.1 py39_cu117 pytorch
torchtriton 2.1.0 py39 pytorch
torchvision 0.14.1 py39_cu117 pytorch
tqdm 4.66.1 pypi_0 pypi
transformers 4.34.1 pypi_0 pypi
typing_extensions 4.7.1 py39h06a4308_0
tzdata 2023.3 pypi_0 pypi
urllib3 1.26.18 py39h06a4308_0
wandb 0.15.12 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.8 pypi_0 pypi
werkzeug 3.0.1 pypi_0 pypi
wheel 0.41.2 py39h06a4308_0
xformers 0.0.22.post7 py39_cu11.7.1_pyt1.13.1 xformers
xxhash 3.4.1 pypi_0 pypi
xz 5.4.2 h5eee18b_0
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.9.2 pypi_0 pypi
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0

Comments:
I think the code hangs on the `accelerator.main_process_first()` context manager exit. To verify this, you can append a print statement to the end of the  `accelerator.main_process_first()` block. 


If the problem is in `with_transform`, it would help if you could share the error stack trace printed when you interrupt the process (while it hangs)

@bhosalems Were you able to fix that ? I face this issue as well

@matankley No I am not able to resolve this issue yet.

@mariosasko yes the problem seems to be to exit from accelerator.main_process_first(). Is there any known problem?

NCCL debug info I get below output, if it helps.
```
11/09/2023 13:36:44 - INFO - __main__ - Distributed environment: MULTI_GPU  Backend: nccl
Num processes: 2
Process index: 1
Local process index: 1
Device: cuda:1

Mixed precision type: fp16

Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
11/09/2023 13:36:44 - INFO - __main__ - Distributed environment: MULTI_GPU  Backend: nccl
Num processes: 2
Process index: 0
Local process index: 0
Device: cuda:0

Mixed precision type: fp16

{'timestep_spacing', 'thresholding', 'variance_type', 'clip_sample_range', 'prediction_type', 'dynamic_thresholding_ratio', 'sample_max_value'} was not found in config. Values will be initialized to default values.
{'norm_num_groups', 'force_upcast'} was not found in config. Values will be initialized to default values.
{'num_attention_heads', 'projection_class_embeddings_input_dim', 'addition_embed_type_num_heads', 'mid_block_only_cross_attention', 'addition_embed_type', 'num_class_embeds', 'upcast_attention', 'cross_attention_norm', 'addition_time_embed_dim', 'time_embedding_dim', 'class_embeddings_concat', 'encoder_hid_dim', 'encoder_hid_dim_type', 'resnet_out_scale_factor', 'attention_type', 'conv_out_kernel', 'only_cross_attention', 'resnet_time_scale_shift', 'resnet_skip_time_act', 'reverse_transformer_layers_per_block', 'conv_in_kernel', 'time_cond_proj_dim', 'use_linear_projection', 'mid_block_type', 'time_embedding_act_fn', 'dropout', 'timestep_post_act', 'dual_cross_attention', 'class_embed_type', 'transformer_layers_per_block', 'time_embedding_type'} was not found in config. Values will be initialized to default values.
{'num_attention_heads', 'projection_class_embeddings_input_dim', 'addition_embed_type_num_heads', 'mid_block_only_cross_attention', 'addition_embed_type', 'num_class_embeds', 'upcast_attention', 'cross_attention_norm', 'addition_time_embed_dim', 'time_embedding_dim', 'class_embeddings_concat', 'encoder_hid_dim', 'encoder_hid_dim_type', 'resnet_out_scale_factor', 'attention_type', 'conv_out_kernel', 'only_cross_attention', 'resnet_time_scale_shift', 'resnet_skip_time_act', 'reverse_transformer_layers_per_block', 'conv_in_kernel', 'time_cond_proj_dim', 'use_linear_projection', 'mid_block_type', 'time_embedding_act_fn', 'dropout', 'timestep_post_act', 'dual_cross_attention', 'class_embed_type', 'transformer_layers_per_block', 'time_embedding_type'} was not found in config. Values will be initialized to default values.
deepbull5:1311249:1311249 [0] NCCL INFO Bootstrap : Using enp194s0f0:128.205.43.171<0>
deepbull5:1311249:1311249 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
deepbull5:1311249:1311249 [0] NCCL INFO cudaDriverVersion 11070
NCCL version 2.14.3+cuda11.7
deepbull5:1311250:1311250 [1] NCCL INFO cudaDriverVersion 11070
deepbull5:1311249:1311365 [0] NCCL INFO NET/IB : No device found.
deepbull5:1311249:1311365 [0] NCCL INFO NET/Socket : Using [0]enp194s0f0:128.205.43.171<0>
deepbull5:1311249:1311365 [0] NCCL INFO Using network Socket
deepbull5:1311250:1311250 [1] NCCL INFO Bootstrap : Using enp194s0f0:128.205.43.171<0>
deepbull5:1311250:1311250 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
deepbull5:1311250:1311366 [1] NCCL INFO NET/IB : No device found.
deepbull5:1311250:1311366 [1] NCCL INFO NET/Socket : Using [0]enp194s0f0:128.205.43.171<0>
deepbull5:1311250:1311366 [1] NCCL INFO Using network Socket
deepbull5:1311250:1311366 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffff0000,00ffffff
deepbull5:1311249:1311365 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffff0000,00ffffff
deepbull5:1311249:1311365 [0] NCCL INFO Channel 00/04 :    0   1
deepbull5:1311250:1311366 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
deepbull5:1311249:1311365 [0] NCCL INFO Channel 01/04 :    0   1
deepbull5:1311249:1311365 [0] NCCL INFO Channel 02/04 :    0   1
deepbull5:1311249:1311365 [0] NCCL INFO Channel 03/04 :    0   1
deepbull5:1311249:1311365 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
deepbull5:1311249:1311365 [0] NCCL INFO Channel 00/0 : 0[1000] -> 1[24000] via P2P/IPC
deepbull5:1311250:1311366 [1] NCCL INFO Channel 00/0 : 1[24000] -> 0[1000] via P2P/IPC
deepbull5:1311249:1311365 [0] NCCL INFO Channel 01/0 : 0[1000] -> 1[24000] via P2P/IPC
deepbull5:1311250:1311366 [1] NCCL INFO Channel 01/0 : 1[24000] -> 0[1000] via P2P/IPC
deepbull5:1311250:1311366 [1] NCCL INFO Channel 02/0 : 1[24000] -> 0[1000] via P2P/IPC
deepbull5:1311249:1311365 [0] NCCL INFO Channel 02/0 : 0[1000] -> 1[24000] via P2P/IPC
deepbull5:1311250:1311366 [1] NCCL INFO Channel 03/0 : 1[24000] -> 0[1000] via P2P/IPC
deepbull5:1311249:1311365 [0] NCCL INFO Channel 03/0 : 0[1000] -> 1[24000] via P2P/IPC
deepbull5:1311249:1311365 [0] NCCL INFO Connected all rings
deepbull5:1311249:1311365 [0] NCCL INFO Connected all trees
deepbull5:1311249:1311365 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
deepbull5:1311249:1311365 [0] NCCL INFO 4 coll channels, 4 p2p channels, 2 p2p channels per peer
deepbull5:1311250:1311366 [1] NCCL INFO Connected all rings
deepbull5:1311250:1311366 [1] NCCL INFO Connected all trees
deepbull5:1311250:1311366 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
deepbull5:1311250:1311366 [1] NCCL INFO 4 coll channels, 4 p2p channels, 2 p2p channels per peer
deepbull5:1311249:1311365 [0] NCCL INFO comm 0x88a84ee0 rank 0 nranks 2 cudaDev 0 busId 1000 - Init COMPLETE
deepbull5:1311250:1311366 [1] NCCL INFO comm 0x89a42f60 rank 1 nranks 2 cudaDev 1 busId 24000 - Init COMPLETE

```

Maybe @muellerzr can help as an `accelerate` maintainer.

I don't know what the issue was, but after going through the thread here I loved the issue with https://github.com/huggingface/accelerate/issues/314#issuecomment-1565259831