[DeepSpeed] DeepSpeed 0.4.4 does not run with Wav2Vec2 pretraining script
Author: patrickvonplaten
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.0.dev0
- Platform: Linux-5.11.0-25-generic-x86_64-with-glibc2.33
- Python version: 3.9.1
- PyTorch version (GPU?): 1.9.0.dev20210217 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes
- Deepspeed: 0.4.4
- CUDA Version: 11.2 
- GPU: 4 x TITAN RTX

### Who can help

@stas00 

## To reproduce

Running the Wav2Vec2 pre-training script: https://github.com/huggingface/transformers/blob/master/examples/research_projects/wav2vec2/README.md#pretraining-wav2vec2

with the versions as defined above (on vorace) yields the following error:
<details>
<summary>Click for error message</summary>
<br>

```
Using amp fp16 backend                                                                                                                                                                                                                                                                                            
[2021-08-08 14:05:34,113] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.4.4, git-hash=unknown, git-branch=unknown                                                                                                                                                                            
[2021-08-08 14:05:35,930] [INFO] [utils.py:11:_initialize_parameter_parallel_groups] data_parallel_size: 4, parameter_parallel_size: 4                                                                                                                                                                            
[2021-08-08 14:57:51,866] [INFO] [engine.py:179:__init__] DeepSpeed Flops Profiler Enabled: False                                                                                                                                                                                                                 
Using /home/patrick/.cache/torch_extensions as PyTorch extensions root...                                                                                                                                                                                                                                         
Creating extension directory /home/patrick/.cache/torch_extensions/cpu_adam...                                                                                                                                                                                                                                    
Using /home/patrick/.cache/torch_extensions as PyTorch extensions root...                                                                                                                                                                                                                                         
Using /home/patrick/.cache/torch_extensions as PyTorch extensions root...                                                                                                                                                                                                                                         
Using /home/patrick/.cache/torch_extensions as PyTorch extensions root...                                                                                                                                                                                                                                         
Detected CUDA files, patching ldflags                                                                                                                                                                                                                                                                             
Emitting ninja build file /home/patrick/.cache/torch_extensions/cpu_adam/build.ninja...                                                                                                                                                                                                                           
Building extension module cpu_adam...                                                                                                                                                                                                                                                                             
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)                                                                                                                                                                                                 
[1/3] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output custom_cuda_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/patrick/anaconda3/envs/hu
gging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/
patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/TH -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/THC -isystem /home/patrick/anaconda3/envs/hugging_face/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D_
_CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERS
IONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_75,code=compute_75 -c /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/adam/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o                                             
FAILED: custom_cuda_kernel.cuda.o                                                                                                                                                                                                                                                                                 
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output custom_cuda_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/patrick/anaconda3/envs/hugging_
face/lib/python3.9/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/patric
k/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/TH -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/THC -isystem /home/patrick/anaconda3/envs/hugging_face/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_
NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__
 -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_75,code=compute_75 -c /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/adam/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o                                                   
/usr/include/c++/10/chrono: In substitution of ‘template<class _Rep, class _Period> template<class _Period2> using __is_harmonic = std::__bool_constant<(std::ratio<((_Period2::num / std::chrono::duration<_Rep, _Period>::_S_gcd(_Period2::num, _Period::num)) * (_Period::den / std::chrono::duration<_Rep, _Pe
riod>::_S_gcd(_Period2::den, _Period::den))), ((_Period2::den / std::chrono::duration<_Rep, _Period>::_S_gcd(_Period2::den, _Period::den)) * (_Period::num / std::chrono::duration<_Rep, _Period>::_S_gcd(_Period2::num, _Period::num)))>::den == 1)> [with _Period2 = _Period2; _Rep = _Rep; _Period = _Period]’:
/usr/include/c++/10/chrono:473:154:   required from here                                                                                                                                                                                                                                                          
/usr/include/c++/10/chrono:428:27: internal compiler error: Segmentation fault                                                                                                                                                                                                                                    
  428 |  _S_gcd(intmax_t __m, intmax_t __n) noexcept                                                                                                                                                                                                                                                              
      |                           ^~~~~~                                                                                                                                                                                                                                                                          
Please submit a full bug report,                                                                                                                                                                                                                                                                                  
with preprocessed source if appropriate.                                                                                                                                                                                                                                                                          
See <file:///usr/share/doc/gcc-10/README.Bugs> for instructions.                                                                                                                                                                                                                                                  
[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/includes -I/usr
/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/inc
lude/TH -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/THC -isystem /home/patrick/anaconda3/envs/hugging_face/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -L/usr/lib64 -lcudart -lcublas -g -Wno-reorder -march=native -fopenmp -D_
_AVX512__ -c /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o                                                                                                                                                                             
ninja: build stopped: subcommand failed.                                                                                                                                                                                                                                                                          
Traceback (most recent call last):                                                                                                                                                                                                                                                                                
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1662, in _run_ninja_build                                                                                                                                                                       
    subprocess.run(                                                                                                                                                                                                                                                                                               
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/subprocess.py", line 524, in run                                                                                                                                                                                                                  
    raise CalledProcessError(retcode, process.args,                                                                                                                                                                                                                                                               
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                  
The above exception was the direct cause of the following exception:                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                      
Traceback (most recent call last):                                                                                                                                                                                                                                                                                
  File "/home/patrick/hugging_face/transformers/examples/research_projects/wav2vec2/run_pretrain.py", line 394, in <module>                                                                                                                                                                                       
    main()                                                                                                                                                                                                                                                                                                        
  File "/home/patrick/hugging_face/transformers/examples/research_projects/wav2vec2/run_pretrain.py", line 390, in main                                                                                                                                                                                           
    trainer.train()                                                                                                                                                                                                                                                                                               
  File "/home/patrick/hugging_face/transformers/src/transformers/trainer.py", line 1136, in train                                                                                                                                                                                                                 
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(                                                                                                                                                                                                                                                   
  File "/home/patrick/hugging_face/transformers/src/transformers/deepspeed.py", line 370, in deepspeed_init                                                                                                                                                                                                       
    model, optimizer, _, lr_scheduler = deepspeed.initialize(                                                                                            
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/__init__.py", line 126, in initialize                                                                                                                                                                                     
    engine = DeepSpeedEngine(args=args,                                     
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 194, in __init__                                                                                                                                                                                 
    self._configure_optimizer(optimizer, model_parameters)                                                                                               
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 709, in _configure_optimizer                                                                                                                                                                     
    basic_optimizer = self._configure_basic_optimizer(model_parameters)                                                                                  
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 778, in _configure_basic_optimizer                                                                                                                                                               
    optimizer = DeepSpeedCPUAdam(model_parameters,                          
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py", line 79, in __init__                                                                                                                                                                               
    self.ds_opt_adam = CPUAdamBuilder().load()                              
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 347, in load                                                                                                                                                                             
    return self.jit_load(verbose)                                           
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 379, in jit_load                                                                                                                                                                         
    op_module = load(                                                       
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1074, in load                                                                                                                                                                                   
    return _jit_compile(                                                    
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1287, in _jit_compile                                                                                                                                                                           
    _write_ninja_file_and_build_library(                                    
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1399, in _write_ninja_file_and_build_library                                                                                                                                                    
    _run_ninja_build(                                                       
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1678, in _run_ninja_build                                                                                                                                                                       
    raise RuntimeError(message) from e                                      
RuntimeError: Error building extension 'cpu_adam' 
Loading extension module cpu_adam...                                        
Traceback (most recent call last):                                          
  File "/home/patrick/hugging_face/transformers/examples/research_projects/wav2vec2/run_pretrain.py", line 394, in <module>                                                                                                                                                                                       
    main()                                                                  
  File "/home/patrick/hugging_face/transformers/examples/research_projects/wav2vec2/run_pretrain.py", line 390, in main                                                                                                                                                                                           
    trainer.train()                                                         
  File "/home/patrick/hugging_face/transformers/src/transformers/trainer.py", line 1136, in train                                                        
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(                                                                                          
  File "/home/patrick/hugging_face/transformers/src/transformers/deepspeed.py", line 370, in deepspeed_init                                                                                                                                                                                                       
    model, optimizer, _, lr_scheduler = deepspeed.initialize(                                                                                            
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/__init__.py", line 126, in initialize                                                                                                                                                                                     
    engine = DeepSpeedEngine(args=args,                                     
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 194, in __init__                                                                                                                                                                                 
    self._configure_optimizer(optimizer, model_parameters)                                                                                               
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 709, in _configure_optimizer                                                                                                                                                                     
    basic_optimizer = self._configure_basic_optimizer(model_parameters)                                                                                  
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 778, in _configure_basic_optimizer                                                                                                                                                               
    optimizer = DeepSpeedCPUAdam(model_parameters,                          
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py", line 79, in __init__                                                                                                                                                                               
    self.ds_opt_adam = CPUAdamBuilder().load()                              
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 347, in load                                                                                                                                                                             
    return self.jit_load(verbose)                                           
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 379, in jit_load                                                                                                                                                                         
    op_module = load(                                                       
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1074, in load                                                                                                                                                                                   
    return _jit_compile(                                                    
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1312, in _jit_compile                                                                                                                                                                           
    return _import_module_from_library(name, build_directory, is_python_module)                                                                          
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1694, in _import_module_from_library                                                                                                                                                            
    file, path, description = imp.find_module(module_name, [path])                                                                                       
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/imp.py", line 296, in find_module                                                        
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'cpu_adam'
Loading extension module cpu_adam...
Traceback (most recent call last):
  File "/home/patrick/hugging_face/transformers/examples/research_projects/wav2vec2/run_pretrain.py", line 394, in <module>
    main()
  File "/home/patrick/hugging_face/transformers/examples/research_projects/wav2vec2/run_pretrain.py", line 390, in main
    trainer.train()
  File "/home/patrick/hugging_face/transformers/src/transformers/trainer.py", line 1136, in train
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
  File "/home/patrick/hugging_face/transformers/src/transformers/deepspeed.py", line 370, in deepspeed_init
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/__init__.py", line 126, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 194, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 709, in _configure_optimizer
    basic_optimizer = self._configure_basic_optimizer(model_parameters)
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 778, in _configure_basic_optimizer
    optimizer = DeepSpeedCPUAdam(model_parameters,
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py", line 79, in __init__
    self.ds_opt_adam = CPUAdamBuilder().load()
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 347, in load
    return self.jit_load(verbose)
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 379, in jit_load
    op_module = load(
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1074, in load
    return _jit_compile(
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1312, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1694, in _import_module_from_library
    file, path, description = imp.find_module(module_name, [path])
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/imp.py", line 296, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'cpu_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f1a372f01f0>
Traceback (most recent call last):
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py", line 93, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f961a8561f0>
Traceback (most recent call last):
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py", line 93, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7fcc1f2be1f0>
Traceback (most recent call last):
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py", line 93, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f5bbf5cc1f0>
Traceback (most recent call last):
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/adam/cpu_adam.py", line 93, in __del__
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Killing subprocess 1135563
Killing subprocess 1135564
Killing subprocess 1135565
Killing subprocess 1135566
Traceback (most recent call last):
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/launcher/launch.py", line 171, in <module>
    main()
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/launcher/launch.py", line 161, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deep
```

</details>

## Expected behavior

The script should run without problem.


Comments:
@patrickvonplaten,

This is definitely something for Deepspeed and not our integration since you have a segfault in building the kernels:

```
[1/3] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output custom_cuda_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/patrick/anaconda3/envs/hu
gging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/
patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/TH -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/THC -isystem /home/patrick/anaconda3/envs/hugging_face/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D_
_CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERS
IONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_75,code=compute_75 -c /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/adam/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o                                             
FAILED: custom_cuda_kernel.cuda.o                                                                                                                                                                                                                                                                                 
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output custom_cuda_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/patrick/anaconda3/envs/hugging_
face/lib/python3.9/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/patric
k/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/TH -isystem /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/torch/include/THC -isystem /home/patrick/anaconda3/envs/hugging_face/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_
NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__
 -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_75,code=compute_75 -c /home/patrick/anaconda3/envs/hugging_face/lib/python3.9/site-packages/deepspeed/ops/csrc/adam/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o                                                   
/usr/include/c++/10/chrono: In substitution of ‘template<class _Rep, class _Period> template<class _Period2> using __is_harmonic = std::__bool_constant<(std::ratio<((_Period2::num / std::chrono::duration<_Rep, _Period>::_S_gcd(_Period2::num, _Period::num)) * (_Period::den / std::chrono::duration<_Rep, _Pe
riod>::_S_gcd(_Period2::den, _Period::den))), ((_Period2::den / std::chrono::duration<_Rep, _Period>::_S_gcd(_Period2::den, _Period::den)) * (_Period::num / std::chrono::duration<_Rep, _Period>::_S_gcd(_Period2::num, _Period::num)))>::den == 1)> [with _Period2 = _Period2; _Rep = _Rep; _Period = _Period]’:
/usr/include/c++/10/chrono:473:154:   required from here                                                                                                                                                                                                                                                          
/usr/include/c++/10/chrono:428:27: internal compiler error: Segmentation fault                                                                                                                                                                                                                                    
  428 |  _S_gcd(intmax_t __m, intmax_t __n) noexcept  
```

Could you please file a bug report at https://github.com/microsoft/DeepSpeed and tag @RezaYazdaniAminabadi

It probably has something to do with your specific environment, since deepspeed==0.4.4 passes all wav2vec2 tests on my setup:

```
$ RUN_SLOW=1 pyt examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py
====================================================================== test session starts ======================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface, configfile: pytest.ini
plugins: dash-1.20.0, forked-1.3.0, xdist-2.3.0, instafail-0.4.2
collected 16 items                                                                                                                                              

examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py ................                                                                           [100%]
==================================================================== short test summary info ====================================================================
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_distributed_zero2_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_distributed_zero2_robust
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_distributed_zero3_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_distributed_zero3_robust
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_non_distributed_zero2_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_non_distributed_zero2_robust
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_non_distributed_zero3_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp16_non_distributed_zero3_robust
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_distributed_zero2_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_distributed_zero2_robust
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_distributed_zero3_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_distributed_zero3_robust
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_non_distributed_zero2_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_non_distributed_zero2_robust
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_non_distributed_zero3_base
PASSED examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py::TestDeepSpeedWav2Vec2::test_fp32_non_distributed_zero3_robust
================================================================ 16 passed in 491.63s (0:08:11) =================================================================
```

Also yours is python-3.9, do you have access to 3.8 by chance to validate if perhaps it's a py39 incompatibility? Mine is 3.8.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

It's actually solved - thanks for the help :-)