`model.get_input_embeddings()` returns an embedding size of 0 with DS-ZeRO3
Author: regisss
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-5.15.0-1015-aws-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.17.1
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 1.12.0a0+bd13bc6 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: ?
- Using distributed or parallel set-up in script?: DeepSpeed ZeRO-3

### Who can help?

@pacman100 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the command suggested in the [summarization example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) with DeepSpeed:
```
deepspeed --num_gpus 8 examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate \
    --deepspeed ds_config.json
```
with `ds_config.json` being:
```json
{
  "fp16": {
    "enabled": true
  },
  "optimizer": {
    "type": "adam",
    "params": {
      "lr": "auto",
      "betas": "auto",
      "eps": "auto",
      "weight_decay": "auto",
      "torch_adam": "torch_impl",
      "adam_w_mode": false
    }
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto",
      "warmup_num_steps": "auto"
    }
  },
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": 1666777,
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e9,
    "stage3_max_reuse_distance": 1e9,
    "stage3_gather_16bit_weights_on_model_save": false
  },
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "steps_per_print": 2000,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "wall_clock_breakdown": false
}
```

### Expected behavior

The embedding size [here](https://github.com/huggingface/transformers/blob/6acc27eea853885270dba5313181443d43e31f2c/examples/pytorch/summarization/run_summarization.py#L466) should not be 0, I guess this happens because ZeRO-3 spreads the model params across devices. It doesn't happen with ZeRO-2.

Comments:
Hello, yes, that is the issue. Does the below code changes solve the issue?

```
embeddings = model.get_input_embeddings()
with deepspeed.zero.GatheredParameters(embeddings.weight, modifier_rank=None):
    embedding_size = embeddings.weight.shape[0]
```



That works, thanks!
Shall I open a PR to add this code snippet if DS ZeRO-3 is enabled in the summarization example?

Does something like `embedding_size = model.get_input_embeddings().num_embeddings` also work? Taken from #26264

Hmm where would that `num_embeddings` attribute come from? Would it be computed from the embedding shapes at some point?

<img width="525" alt="Screenshot 2023-10-18 at 3 01 56 PM" src="https://github.com/huggingface/transformers/assets/13534540/4f89d92f-cf33-461d-9fd7-803824f91677">

@ArthurZucker, `num_embeddings` only keeps the number when creating the embedding object but it would be incorrect if someone wants the vocab size post reshaping the embedding layers to add the special tokens. Example of this is given above 

Arf did not know this, thanks 😉