text: 
  
  TITLE
  gpt2 and t5 parallel modeling
  
  COMMENTS
  6
  
  REACTIONS
  +1: 1
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 9
  eyes: 0
  
  BODY
  # Model Parallelism for T5 and GPT2
  
  Adds two new methods to `GPT2LMHead` and the `GPT2Model` classes to enable you to generate and fine-tune models using model parallelism. This feature is most applicable for `gpt2-large` and `gpt2-xl`. Minor modifications are made to the `TrainingArguments` and `Trainer` classes to avoid conflicting data parallelism behavior and related batch_size increases which would negate model parallelism. Note that nearly 64GB of GPU (4 Tesla v100s)  are needed to fine-tune `gpt2-xl` @ 1024 tokens.
  
  It is critically important to provide users the ability to specify where to put the blocks of a model because the GPU sizes and numbers are likely to be very diverse. This is done with a dictionary called `device_map`. I am planning on providing some examples and guidelines for the p3, p2 and g3 AWS instances. 
  
  Model parallelism has to be baked into the model class itself. Currently working on the T5 model. From my calculations the 11B model cannot fit on the largest p3 instance that I have access to (8 Tesla v100 GPUs). The 3B model can.
  
  The methods are:
  - `parallelize`, which will distribute the attention blocks of the model across several devices according to a device map
  - `deparallelize`, which will move the model back to cpu
  
  # Example
  
  ```
  model = GPT2LMHeadModel.from_pretrained('gpt2-xl')
  device_map = {0: [0, 1, 2, 3, 4, 5, 6, 7, 8],
                                1: [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21],
                                2: [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34],
                                3: [35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]}
  model.parallelize(device_map) # Distributes the model's attention blocks across several devices
  model.deparallelize() # Puts the model back on cpu and calls torch.cuda.empty_cache() to liberate GPU memory
  ```
  
  ## Reviewers
  @LysandreJik 