DeepSpeed Stage 2, Gradient computed twice for this partition.
Author: lkm2835
Pull request: no
Labels: name: bug

### System Info

transformers == 4.20.1
python == 3.8.13
OS == ubuntu 20.4
DeepSpeed == 0.6.7

### Who can help?

@stas00 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**Error message**

```
File "../train_t5_encoder.py", line 218, in <module>
    main(args)
  File "../train_t5_encoder.py", line 186, in main
    train_result = trainer.train()
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/transformers/trainer.py", line 1409, in train
    return inner_training_loop(
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/transformers/trainer.py", line 1651, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/transformers/trainer.py", line 2361, in training_step
    loss = self.deepspeed.backward(loss)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 11, in wrapped_fn
    return func(*args, **kwargs)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1709, in backward
    self.optimizer.backward(loss)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1946, in backward
    self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 51, in backward
    scaled_loss.backward(retain_graph=retain_graph)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/torch/autograd/function.py", line 253, in apply
    return user_fn(self, *args)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 146, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 798, in reduce_partition_and_remove_grads
    self.reduce_ready_partitions_and_remove_grads(param, i)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1269, in reduce_ready_partitions_and_remove_grads
    self.reduce_independent_p_g_buckets_and_remove_grads(param, i)
  File "/home/kyungmin.lee/anaconda3/envs/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 833, in reduce_independent_p_g_buckets_and_remove_grads
    assert self.params_already_reduced[param_id] == False, \
AssertionError: The parameter 191 has already been reduced.             Gradient computed twice for this partition.             Multiple gradient reduction is currently not supported
```

**My model**
``` python3
class GPT2ForScoreModel(GPT2ForSequenceClassification):
    _keys_to_ignore_on_load_missing = [r"h\.\d+\.attn\.masked_bias"]
    
    def __init__(self, config):
        config.num_labels = 1
        self.main_input_name = 'input_ids_pos'

        super().__init__(config)

    def forward(self,
        input_ids=None,
        attention_mask=None,
        input_ids_pos=None,
        attention_mask_pos=None,
        input_ids_neg=None,
        attention_mask_neg=None,
        past_key_values=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        use_cache=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None):

        if input_ids is None:
            input_ids = input_ids_pos
            attention_mask = attention_mask_pos

        outputs_pos = super().forward(
            input_ids,
            attention_mask=attention_mask,
            past_key_values=past_key_values,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=True,
        )

        loss = None

        if input_ids_neg is not None:

            outputs_neg = super().forward(
                input_ids_neg,
                attention_mask=attention_mask_neg,
                past_key_values=past_key_values,
                token_type_ids=token_type_ids,
                position_ids=position_ids,
                head_mask=head_mask,
                inputs_embeds=inputs_embeds,
                use_cache=use_cache,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
                return_dict=True,
            )

            loss = - F.logsigmoid(outputs_pos.logits - outputs_neg.logits).mean()

        return SequenceClassifierOutputWithPast(
            loss=loss,
            logits=outputs_pos.logits,
            past_key_values=past_key_values,
            hidden_states=outputs_pos.hidden_states,
            attentions=outputs_pos.attentions,
        )
```
I used GPT2ForSequenceClassification and T5EncoderModel.

My training code works without Deepspeed, and with Deepspeed Stage 3 (but there is another problem in stage3).

But this error occurs on stage 2.

### Expected behavior

Both with and without Deepspeed work normally and the training is completed.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

the same

the same