LayoutLMv2 training on sagemaker error: undefined value has_torch_function_variadic
Author: Natlem
Pull request: no
Labels: name: bug

### System Info

```shell
transformer: 4.17.0
torch: 1.10.2

Platform: Sagemaker Deep Learning Container
```


### Who can help?

@NielsRogge

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The error only comes when training on Sagemaker using Huggingface.

Scripts to start training on Sagemaker:

Folder organization:
```
./
----sg_training.py
----scripts
-------requirements.txt
-------train.py 
```

sg_training.py:
```
import boto3
import sagemaker
from sagemaker.huggingface import HuggingFace

if __name__ == "__main__":
    iam_client = boto3.client(...)

    role = iam_client.get_role(...)['Role']['Arn']
    sess = sagemaker.Session()

    sagemaker_session_bucket = 's3-sagemaker-session'

    hyperparameters = {'epochs': 20,
                       'train_batch_size': 1,
                       'model_name': "microsoft/layoutxlm-base",
                       'output_dir': '/opt/ml/model/',
                       'checkpoints': '/opt/ml/checkpoints/',
                       'combine_train_val': True,
                       'exp_tracker': "all",
                       'exp_name': 'Sagemaker Training'
                       }

    huggingface_estimator = HuggingFace(entry_point='train.py',
                                        source_dir='scripts',
                                        instance_type='ml.p3.2xlarge',
                                        instance_count=1,
                                        role=role,
                                        transformers_version='4.17.0',
                                        pytorch_version='1.10.2',
                                        py_version='py38',
                                        hyperparameters=hyperparameters,
                                        environment={'HF_TASK': 'text-classification'},
                                        code_location='s3://dummy_code_location')

    huggingface_estimator.fit()
```

Entrypoint scripts folder:


requirements.txt:
```
git+https://github.com/facebookresearch/detectron2.git
```

train.py:
```
import argparse
import logging
import os
import sys

from transformers import LayoutLMv2ForSequenceClassification


def run():
    model = LayoutLMv2ForSequenceClassification.from_pretrained('microsoft/layoutxlm-base',
                                                                num_labels=5)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--epochs", type=int, default=3)
    parser.add_argument("--exp_name", type=str, default="Sagemaker Training")
    parser.add_argument("--train-batch-size", type=int, default=2)
    parser.add_argument("--eval-batch-size", type=int, default=1)
    parser.add_argument("--warmup_steps", type=int, default=500)
    parser.add_argument("--model_name", type=str)
    parser.add_argument("--learning_rate", type=str, default=1e-5)
    parser.add_argument("--combine_train_val", type=bool, default=False)
    # Data, model, and output directories
    parser.add_argument("--output-data-dir", type=str, default=os.environ["SM_OUTPUT_DATA_DIR"])
    parser.add_argument("--checkpoints", type=str, default="/opt/ml/checkpoints")
    parser.add_argument("--model-dir", type=str, default='/opt/ml/code/model')
    parser.add_argument("--n_gpus", type=str, default=os.environ["SM_NUM_GPUS"])
    args, _ = parser.parse_known_args()

    logger = logging.getLogger(__name__)
    logging.basicConfig(
        level=logging.getLevelName("INFO"),
        handlers=[logging.StreamHandler(sys.stdout)],
        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    )

    run()

```


### Expected behavior

```shell
Here the log on the error from AWS Cloud Watch:

Invoking script with the following command:
/opt/conda/bin/python3.8 train.py --checkpoints /opt/ml/checkpoints/ --combine_train_val True --epochs 20 --exp_name Sagemaker_Training_doc_cls --exp_tracker all --model_name microsoft/layoutxlm-base --output_dir /opt/ml/model/ --train_batch_size 1
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/transformers/file_utils.py", line 2777, in _get_module
return importlib.import_module("." + module_name, self.__name__)
  File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/conda/lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 48, in <module>
from detectron2.modeling import META_ARCH_REGISTRY
  File "/opt/conda/lib/python3.8/site-packages/detectron2/modeling/__init__.py", line 2, in <module>
from detectron2.layers import ShapeSpec
  File "/opt/conda/lib/python3.8/site-packages/detectron2/layers/__init__.py", line 2, in <module>
from .batch_norm import FrozenBatchNorm2d, get_norm, NaiveSyncBatchNorm, CycleBatchNormList
  File "/opt/conda/lib/python3.8/site-packages/detectron2/layers/batch_norm.py", line 4, in <module>
    from fvcore.nn.distributed import differentiable_all_reduce
  File "/opt/conda/lib/python3.8/site-packages/fvcore/nn/__init__.py", line 4, in <module>
    from .focal_loss import (
  File "/opt/conda/lib/python3.8/site-packages/fvcore/nn/focal_loss.py", line 52, in <module>
    sigmoid_focal_loss_jit: "torch.jit.ScriptModule" = torch.jit.script(sigmoid_focal_loss)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_script.py", line 1310, in script
fn = torch._C._jit_script_compile(
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_recursive.py", line 838, in try_compile_fn
return torch.jit.script(fn, _rcb=rcb)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_script.py", line 1310, in script
fn = torch._C._jit_script_compile(
RuntimeError: 
undefined value has_torch_function_variadic:
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/smdebug.py", line 2962
         >>> loss.backward()
    """
    if has_torch_function_variadic(input, target, weight, pos_weight):
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        return handle_torch_function(
            binary_cross_entropy_with_logits,
'binary_cross_entropy_with_logits' is being compiled since it was called from 'sigmoid_focal_loss'
  File "/opt/conda/lib/python3.8/site-packages/fvcore/nn/focal_loss.py", line 36
    targets = targets.float()
    p = torch.sigmoid(inputs)
    ce_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction="none")
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    p_t = p * targets + (1 - p) * (1 - targets)
    loss = ce_loss * ((1 - p_t) ** gamma)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "train.py", line 6, in <module>
from transformers import LayoutLMv2ForSequenceClassification
  File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
  File "/opt/conda/lib/python3.8/site-packages/transformers/file_utils.py", line 2768, in __getattr__
value = getattr(module, name)
  File "/opt/conda/lib/python3.8/site-packages/transformers/file_utils.py", line 2767, in __getattr__
module = self._get_module(self._class_to_module[name])
  File "/opt/conda/lib/python3.8/site-packages/transformers/file_utils.py", line 2779, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.layoutlmv2.modeling_layoutlmv2 because of the following error (look up to see its traceback):
undefined value has_torch_function_variadic:
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/smdebug.py", line 2962
         >>> loss.backward()
    """
    if has_torch_function_variadic(input, target, weight, pos_weight):
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        return handle_torch_function(
            binary_cross_entropy_with_logits,
'binary_cross_entropy_with_logits' is being compiled since it was called from 'sigmoid_focal_loss'
  File "/opt/conda/lib/python3.8/site-packages/fvcore/nn/focal_loss.py", line 36
    targets = targets.float()
    p = torch.sigmoid(inputs)
    ce_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction="none")
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    p_t = p * targets + (1 - p) * (1 - targets)
    loss = ce_loss * ((1 - p_t) ** gamma)

```
```


Comments:
cc @philschmid (hope I am tagging correctly)

@Natlem could you try adding `debugger_hook_config=False` to the `HuggingFace` estimator? 

```python
    huggingface_estimator = HuggingFace(entry_point='train.py',
                                        source_dir='scripts',
                                        instance_type='ml.p3.2xlarge',
                                        instance_count=1,
                                        role=role,
                                        transformers_version='4.17.0',
                                        pytorch_version='1.10.2',
                                        py_version='py38',
                                        hyperparameters=hyperparameters,
                                        environment={'HF_TASK': 'text-classification'},
                                        code_location='s3://dummy_code_location',
                                        debugger_hook_config=False,
)
```

Hi @philschmid ,

Added the `debugger_hook_config=False`, the error is gone now. Thanks !

Awesome, closing the issue.  Feel free to reopen if you have more issues.

@Natlem i forwarded the error to the AWS team to be able use the debugger soon. 

@philschmid  Thanks !

@philschmid do you have any idea why this solves the problem? Is it documented by AWS anywhere?

Sagemaker Debugger has cost me multiple days of time in the mysterious problems it produces. Far more than anything else on Sagemaker. I posted an issue on awslabs about this awhile back and never got a reply. I would really like to know what is going on here

**For anyone encountering this while using a HyperparameterTuner**
Passing `debugger_hook_config=False` in the `Estimator` will not the solve the problem. Further, passing `environment={'USE_SMDEBUG':0}` also will not solve the problem. Somehow these settings never make it to a tuner's constituent training jobs.

The only way to solve it is to set `ENV USE_SMDEBUG="0"` in the docker container that will be running the constituent training jobs.

> Somehow these settings never make it to a tuner's constituent training jobs.

Are you using the `HuggingFace` estimator or the `HyperparameterTuner`