Wav2Vec2ForPreTraining doc example has None loss
Author: Sorrow321
Pull request: no
Labels: name: Good First Issue

name: Good First Documentation Issue

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: Linux-5.11.0-46-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): 2.7.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
Models:
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

Documentation: @sgugger


## Information

I'm trying to additionally pretrain Wav2Vec2.0 model on my dataset. [In the docs](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2ForPreTraining) you have an example for running the `Wav2Vec2ForPreTraining`:

```python
import torch
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2ForPreTraining
from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices
from datasets import load_dataset
import soundfile as sf

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("patrickvonplaten/wav2vec2-base")
model = Wav2Vec2ForPreTraining.from_pretrained("patrickvonplaten/wav2vec2-base")


def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch


ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

input_values = feature_extractor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1

# compute masked indices
batch_size, raw_sequence_length = input_values.shape
sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length)
mask_time_indices = _compute_mask_indices((batch_size, sequence_length), mask_prob=0.2, mask_length=2)

with torch.no_grad():
    outputs = model(input_values, mask_time_indices=mask_time_indices)

# compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
cosine_sim = torch.cosine_similarity(
    outputs.projected_states, outputs.projected_quantized_states, dim=-1
)

# show that cosine similarity is much higher than random
assert cosine_sim[mask_time_indices].mean() > 0.5

# for contrastive loss training model should be put into train mode
model.train()
loss = model(input_values, mask_time_indices=mask_time_indices).loss
```

If you print the `loss` that you get in the end, you will get `None`. This happens because [in the definition](https://github.com/huggingface/transformers/blob/f4b7420dfe419fe653908f091976517635a119e6/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1511) you have to pass `sampled_negative_indices` in order to get not `None` loss.


## To reproduce

Steps to reproduce the behavior:

1. Run the above code
2. `print(loss)` in the end

## Expected behavior

Expected to have some example on how to get the actual loss and train the model.


Comments:
Hi,

Thanks for spotting. Feel free to open a PR to fix this :) 

Hi As I am a beginner, I would love to contribute to this Good First Issue, can anyone guide me.
ThankYou

Hi,

The docs of `Wav2Vec2ForPretraining` can be found in `modeling_wav2vec2.py`, [here](https://github.com/huggingface/transformers/blob/c85547af2b69f9082bcd7bac97092b1d162f3fdc/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1469-L1502).

Hi, 

I am a beginner, and given that the issue has been inactive for some time, I'll like to take this and try fixing it.

The following changes to the code, mainly adding the `sampled_negative_indices` as mentioned by the original issue, work, and allow to print the loss ( run on google colab ).

The sampled_negative_indices can be calculated from the `_sample_negative_indices` function from `transformers.models.wav2vec2.modeling_wav2vec2`

Also I added a .item() at the end of 
`sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item()` 
because otherwise it would be a tensor, and would give an error in `_sample_negative_indices`


![image](https://user-images.githubusercontent.com/43698245/171943954-dceb4f2c-4e45-40b0-9803-6b7f20d000ee.png)


Does this change to the documentation seem fine? I am just a beginner so please let me know if I need to provide more information/context

Adding the code here

```python

import torch
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2ForPreTraining
from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices
from datasets import load_dataset
import soundfile as sf

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("patrickvonplaten/wav2vec2-base")
model = Wav2Vec2ForPreTraining.from_pretrained("patrickvonplaten/wav2vec2-base")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values  # Batch size 1

# compute masked indices
batch_size, raw_sequence_length = input_values.shape
sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item()
mask_time_indices = _compute_mask_indices((batch_size, sequence_length), mask_prob=0.2, mask_length=2)
sampled_negative_indices = _sample_negative_indices((batch_size, sequence_length), model.config.num_negatives, mask_time_indices)
mask_time_indices = torch.tensor(mask_time_indices, device=input_values.device, dtype=torch.long)
sampled_negative_indices = torch.tensor(sampled_negative_indices, device=input_values.device, dtype=torch.long)

with torch.no_grad():
  outputs = model(input_values, mask_time_indices=mask_time_indices)

# compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1)

# show that cosine similarity is much higher than random
print(cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5)
# tensor(True)

# for contrastive loss training model should be put into train mode
model = model.train()
loss = model(input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices).loss
print(loss)
```

Hey @Sorrow321 @ayushtues could you upgrade your transformers version to the newest version and try again? I cannot reproduce the error on `main`

Hi, @patrickvonplaten I am still getting None if we don't pass sampled_negative_indices to the model (as in the original code given by @Sorrow321) while passing it prints the actual loss.

I am using transformers 4.20.0 dev.0, built from source on the latest main branch

My code:

```python
import torch
import transformers
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2ForPreTraining
from transformers.models.wav2vec2.modeling_wav2vec2 import (
    _compute_mask_indices,
    _sample_negative_indices,
)
from datasets import load_dataset
import soundfile as sf

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(
    "patrickvonplaten/wav2vec2-base"
)
model = Wav2Vec2ForPreTraining.from_pretrained("patrickvonplaten/wav2vec2-base")

ds = load_dataset(
    "hf-internal-testing/librispeech_asr_dummy", "clean", split="validation"
)
input_values = feature_extractor(
    ds[0]["audio"]["array"], return_tensors="pt"
).input_values  # Batch size 1

# compute masked indices
batch_size, raw_sequence_length = input_values.shape
sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item()
mask_time_indices = _compute_mask_indices(
    (batch_size, sequence_length), mask_prob=0.2, mask_length=2
)
sampled_negative_indices = _sample_negative_indices(
    (batch_size, sequence_length), model.config.num_negatives, mask_time_indices
)
mask_time_indices = torch.tensor(
    mask_time_indices, device=input_values.device, dtype=torch.long
)
sampled_negative_indices = torch.tensor(
    sampled_negative_indices, device=input_values.device, dtype=torch.long
)

with torch.no_grad():
    outputs = model(input_values, mask_time_indices=mask_time_indices)

# compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states)
cosine_sim = torch.cosine_similarity(
    outputs.projected_states, outputs.projected_quantized_states, dim=-1
)

# show that cosine similarity is much higher than random
print(cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5)
# tensor(True)

# for contrastive loss training model should be put into train mode
model = model.train()
loss = model(
    input_values,
    mask_time_indices=mask_time_indices,
    # sampled_negative_indices=sampled_negative_indices,
).loss
print("transformers version")
print(transformers.__version__)
print("Loss without sampled_negative_indices")
print(loss)


loss = model(
    input_values,
    mask_time_indices=mask_time_indices,
    sampled_negative_indices=sampled_negative_indices,
).loss
print("Loss with sampled_negative_indices")
print(loss)
```

Output screenshot
![image](https://user-images.githubusercontent.com/43698245/173180102-ca1a5359-886c-4942-808b-aa5f56e7b047.png)


Hey @ayushtues,

We need to pass the `sampled_negative_indices` to get a loss - it's more or less equivalent to the "labels" in training

Hey @ayushtues are you still working on this? I would be interested in taking this issue up if not.

@pramodith by all means, feel free to look into this issue!

Hey @patrickvonplaten, since no one seems to be working on this issue at the moment I'd love to address it if that's okay!

@patrickvonplaten I just opened a PR to address this issue based on @ayushtues's suggestions.

On a side note I am fairly new to open source software contribution. Any feedback is very welcome and if there is any issue with this PR I'd be happy to address it.