text: 
  
  TITLE
  [Deepspeed Inference] HF Integration
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  This PR is working on an integration of [Deepspeed Inference](https://www.deepspeed.ai/tutorials/inference-tutorial/) which implements Tensor Parallelism. This is different from Deepspeed ZeRO inference.
  
  This is a very early draft.
  
  To try:
  
  ```
  cd transformers
  export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0  \
  deepspeed --num_gpus=2 examples/pytorch/translation/run_translation.py \
  --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 \
  --evaluation_strategy=steps --do_eval --label_smoothing 0.1 --learning_rate \
  3e-5 --logging_first_step --logging_steps 500 --max_source_length 128 \
  --max_target_length 128 --overwrite_output_dir --per_device_eval_batch_size $BS \
  --predict_with_generate --sortish_sampler --source_lang en --target_lang ro \
  --dataset_name wmt16 --dataset_config ro-en --source_prefix \
  'translate English to Romanian: ' --val_max_target_length 128 --warmup_steps \
  50 --max_eval_samples 50 --deepspeed_inference --skip_memory_metrics 0
  ```
  
  and it currently hangs with `--num_gpus > 1`. One gpu finishes processing and the other is stuck in preparing inputs. So need to figure out the synchronization of the gpus.