text: 
  
  TITLE
  LED Model returns AlgorithmError when using SageMaker SMP training
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ### System Info
  
  ```shell
  using sagemaker 
  mpi_options = {
      "enabled" : True,
      "processes_per_host" : 8
  }
  
  smp_options = {
      "enabled":True,
      "parameters": {
          "microbatches": 1,
          "placement_strategy": "spread",
          "pipeline": "interleaved",
          "optimize": "memory",
          "partitions": 2,
          "ddp": True,
      }
  }
  
  distribution={
      "smdistributed": {"modelparallel": smp_options},
      "mpi": mpi_options
  }
  hyperparameters={'epochs': 1,
                   'train_batch_size': 1,
                   'eval_batch_size': 1,
                   'model_name':HHousen/distil-led-large-cnn-16384,
                   'output_dir': 'bucket',
                   'warmup_steps': 25,
                   'checkpoint_s3_uri': 'bucket',
                   'logging_steps':100,
                   'evaluation_strategy':"steps",
                   'gradient_accumulation_steps':10
                   }
  huggingface_estimator = HuggingFace(entry_point='trainer.py',
                              source_dir='./scripts',
                              instance_type='ml.p3.16xlarge',
                              instance_count=1,
                              role=role,
                              volume=100,
                              transformers_version='4.6.1',
                              pytorch_version='1.8.1',
                              py_version='py36',
                              hyperparameters=hyperparameters,
                                     distribution=distribution)
  ```
  
  
  ### Who can help?
  
  @ydshieh @sgugger
  
  ### Information
  
  - [ ] The official example scripts
  - [ ] My own modified scripts
  
  ### Tasks
  
  - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
  - [ ] My own task or dataset (give details below)
  
  ### Reproduction
  
  1. Create huggingface estimator
  2.     training_args = Seq2SeqTrainingArguments(
          predict_with_generate=True,
          evaluation_strategy="steps",
          per_device_train_batch_size=1,
          per_device_eval_batch_size=1,
          fp16=True,
          fp16_backend="apex",
          output_dir=s3_bucket,
          logging_steps=50,
          warmup_steps=25,
          gradient_accumulation_steps=10,
      )
  
  Error I get:
  [1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/smdistributed/modelparallel/torch/patches/tracing.py", line 68, in trace_forward
  [1,0]<stderr>:    raise e
  [1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/smdistributed/modelparallel/torch/patches/tracing.py", line 51, in trace_forward
  [1,0]<stderr>:    output = original_forward(self, *args, **kwargs)
  [1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/transformers/models/led/modeling_led.py", line 125, in forward
  [1,0]<stderr>:    return super().forward(positions)
  [1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/smdistributed/modelparallel/torch/patches/tracing.py", line 68, in trace_forward
  [1,0]<stderr>:    raise e
  [1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/smdistributed/modelparallel/torch/patches/tracing.py", line 51, in trace_forward
  [1,0]<stderr>:    output = original_forward(self, *args, **kwargs)
  [1,0]<stderr>:  File "/opt/conda/lib/python3.6/site-packages/transformers/models/led/modeling_led.py", line 121, in forward
  [1,0]<stderr>:    bsz, seq_len = input_ids_shape[:2]
  [1,0]<stderr>:ValueError: not enough values to unpack (expected 2, got 1)
  --------------------------------------------------------------------------
  Primary job  terminated normally, but 1 process returned
  a non-zero exit code. Per user-direction, the job has been aborted.
  --------------------------------------------------------------------------
  --------------------------------------------------------------------------
  mpirun.real detected that one or more processes exited with non-zero status, thus causing
  the job to be terminated. The first process to do so was:
    Process name: [[41156,1],0]
    Exit code:    1
  --------------------------------------------------------------------------
  
  
  ### Expected behavior
  
  ```shell
  Training on a sagemaker notebook p3dn.24xlarge using fairscale `simple` and these versions
  transformers-4.16.2
  torch-1.10.2
  fairscale-0.4.5
  py37
  
  I can successfully train the LED model with my training data. Trying to get it to work with Huggingface estimator and sagemaker SMP I would assume the same outcome.
  ```
  