text: 
  
  TITLE
  Pretrain on Wav2vec2 getting parameters did not receive grad for rank0.
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ### System Info
  
  
  Following [this example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-pretraining) and running with [base model](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-pretraining#base) seem ok.
  
  But when turning into the [large model](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-pretraining#large), the model name is './' so I guess this is a typo. Therefore I tried `facebook/wav2vec2-large-xlsr-53` and `facebook/wav2vec2-large-lv60`, both getting
  
  ```
  Parameters which did not receive grad for rank 0: wav2vec2.encoder.layers.16.final_layer_norm.bias, wav2vec2.encoder.layers.16.final_layer_norm.weight, wav2vec2.encoder.layers.16.feed_forward.output_dense.bias, wav2vec2.encoder.layers.16.feed_forward.output_dense.weight, wav2vec2.encoder.layers.16.feed_forward.intermediate_dense.bias, wav2vec2.encoder.layers.16.feed_forward.intermediate_dense.weight, wav2vec2.encoder.layers.16.layer_norm.bias, wav2vec2.encoder.layers.16.layer_norm.weight, wav2vec2.encoder.layers.16.attention.out_proj.bias, wav2vec2.encoder.layers.16.attention.out_proj.weight, wav2vec2.encoder.layers.16.attention.q_proj.bias, wav2vec2.encoder.layers.16.attention.q_proj.weight, wav2vec2.encoder.layers.16.attention.v_proj.bias, wav2vec2.encoder.layers.16.attention.v_proj.weight, wav2vec2.encoder.layers.16.attention.k_proj.bias, wav2vec2.encoder.layers.16.attention.k_proj.weight, wav2vec2.encoder.layers.15.final_layer_norm.bias, wav2vec2.encoder.layers.15.final_layer_norm.weight, wav2vec2.encoder.layers.15.feed_forward.output_dense.bias, wav2vec2.encoder.layers.15.feed_forward.output_dense.weight, wav2vec2.encoder.layers.15.feed_forward.intermediate_dense.bias, wav2vec2.encoder.layers.15.feed_forward.intermediate_dense.weight, wav2vec2.encoder.layers.15.layer_norm.bias, wav2vec2.encoder.layers.15.layer_norm.weight, wav2vec2.encoder.layers.15.attention.out_proj.bias, wav2vec2.encoder.layers.15.attention.out_proj.weight, wav2vec2.encoder.layers.15.attention.q_proj.bias, wav2vec2.encoder.layers.15.attention.q_proj.weight, wav2vec2.encoder.layers.15.attention.v_proj.bias, wav2vec2.encoder.layers.15.attention.v_proj.weight, wav2vec2.encoder.layers.15.attention.k_proj.bias, wav2vec2.encoder.layers.15.attention.k_proj.weight, wav2vec2.encoder.layers.14.final_layer_norm.bias, wav2vec2.encoder.layers.14.final_layer_norm.weight, wav2vec2.encoder.layers.14.feed_forward.output_dense.bias, wav2vec2.encoder.layers.14.feed_forward.output_dense.weight, wav2vec2.encoder.layers.14.feed_forward.intermediate_dense.bias, wav2vec2.encoder.layers.14.feed_forward.intermediate_dense.weight, wav2vec2.encoder.layers.14.layer_norm.bias, wav2vec2.encoder.layers.14.layer_norm.weight, wav2vec2.encoder.layers.14.attention.out_proj.bias, wav2vec2.encoder.layers.14.attention.out_proj.weight, wav2vec2.encoder.layers.14.attention.q_proj.bias, wav2vec2.encoder.layers.14.attention.q_proj.weight, wav2vec2.encoder.layers.14.attention.v_proj.bias, wav2vec2.encoder.layers.14.attention.v_proj.weight, wav2vec2.encoder.layers.14.attention.k_proj.bias, wav2vec2.encoder.layers.14.attention.k_proj.weight
  Parameter indices which did not receive grad for rank 0: 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309
  ```
  
  
  
  
  ### Who can help?
  
  @patrickvonplaten
  
  ### Information
  
  - [ ] The official example scripts
  - [X] My own modified scripts
  
  ### Tasks
  
  - [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
  - [ ] My own task or dataset (give details below)
  
  ### Reproduction
  
  Follow https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-pretraining#large
  
  ### Expected behavior
  
  ```shell
  Expect no such errors!
  ```
  