Unable to load LayoutLM from pretrained
Author: brian8128
Pull request: no

## Environment info

- `transformers` version: 3.3.0
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.10
- Python version: 3.8.2
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

### Who can help
@sgugger
@LysandreJik

## Information

When I try to load a LayoutLM model with the following script I hit an error. 

```
from transformers import LayoutLMForTokenClassification
model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased', from_tf=True)
```
```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
~/miniconda3/envs/ML38/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    905                 if resolved_archive_file is None:
--> 906                     raise EnvironmentError
    907             except EnvironmentError:

OSError: 

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-11-9f82aa1a161f> in <module>
----> 1 model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased', from_tf=True)

~/miniconda3/envs/ML38/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    911                     f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a file named one of {WEIGHTS_NAME}, {TF2_WEIGHTS_NAME}, {TF_WEIGHTS_NAME}.\n\n"
    912                 )
--> 913                 raise EnvironmentError(msg)
    914 
    915             if resolved_archive_file == archive_file:

OSError: Can't load weights for 'microsoft/layoutlm-base-uncased'. Make sure that:

- 'microsoft/layoutlm-base-uncased' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'microsoft/layoutlm-base-uncased' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.
```

Comments:
LayoutLM only has a PyTorch implementation available. If you remove the `from_tf=True` statement, it will work.


I am trying to load the model weights from [here](https://huggingface.co/microsoft/layoutlm-base-uncased/tree/main#) but `from_tf=False` doesn't work either.  Traceback is below.
```
file_share_pre_train_model_path = "layoutlm-base-uncased"
... config = LayoutLMConfig.from_pretrained(
...     os.path.join(file_share_pre_train_model_path, "config.json"), num_labels=len(tag_labels), cache_dir=None
... )
... model = LayoutLMForTokenClassification.from_pretrained(
...     file_share_pre_train_model_path,
...     from_tf=False,
...     config=config,
...     cache_dir=None,
... )
Traceback (most recent call last):
  File "/Users/hsk/Company/environments/ner_layoutlm/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1035, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/Users/hsk/Company/environments/ner_layoutlm/lib/python3.8/site-packages/torch/serialization.py", line 527, in load
    with _open_zipfile_reader(f) as opened_zipfile:
  File "/Users/hsk/Company/environments/ner_layoutlm/lib/python3.8/site-packages/torch/serialization.py", line 224, in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at ../caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at ../caffe2/serialize/inline_container.cc:132)
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 135 (0x111e90787 in libc10.dylib)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 2350 (0x119f5e14e in libtorch.dylib)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 143 (0x119f5d79f in libtorch.dylib)
frame #3: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::execute<pybind11::class_<caffe2::serialize::PyTorchStreamReader>, 0>(pybind11::class_<caffe2::serialize::PyTorchStreamReader>&)::'lambda'(pybind11::detail::value_and_holder&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >), void, pybind11::detail::value_and_holder&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::class_<caffe2::serialize::PyTorchStreamReader>&&,  (*)(0...), void pybind11::detail::initimpl::constructor<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::execute<pybind11::class_<caffe2::serialize::PyTorchStreamReader>, 0>(pybind11::class_<caffe2::serialize::PyTorchStreamReader>&)::'lambda'(pybind11::detail::value_and_holder&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) const&...)::'lambda'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const + 147 (0x1113f57c3 in libtorch_python.dylib)
frame #4: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 3382 (0x110dede66 in libtorch_python.dylib)
frame #5: cfunction_call_varargs + 120 (0x109b07518 in Python)
frame #6: _PyObject_MakeTpCall + 373 (0x109b06f85 in Python)
frame #7: method_vectorcall + 449 (0x109b0a0b1 in Python)
frame #8: PyVectorcall_Call + 109 (0x109b072ad in Python)
frame #9: slot_tp_init + 201 (0x109b5e619 in Python)
frame #10: type_call + 297 (0x109b59a29 in Python)
frame #11: _PyObject_MakeTpCall + 373 (0x109b06f85 in Python)
frame #12: call_function + 533 (0x109bd5945 in Python)
frame #13: _PyEval_EvalFrameDefault + 25678 (0x109bd274e in Python)
frame #14: _PyEval_EvalCodeWithName + 2804 (0x109bd6734 in Python)
frame #15: _PyFunction_Vectorcall + 270 (0x109b07a6e in Python)
frame #16: _PyObject_FastCallDict + 247 (0x109b06dd7 in Python)
frame #17: _PyObject_Call_Prepend + 143 (0x109b083df in Python)
frame #18: slot_tp_init + 145 (0x109b5e5e1 in Python)
frame #19: type_call + 297 (0x109b59a29 in Python)
frame #20: _PyObject_MakeTpCall + 373 (0x109b06f85 in Python)
frame #21: call_function + 533 (0x109bd5945 in Python)
frame #22: _PyEval_EvalFrameDefault + 25829 (0x109bd27e5 in Python)
frame #23: _PyEval_EvalCodeWithName + 2804 (0x109bd6734 in Python)
frame #24: _PyFunction_Vectorcall + 270 (0x109b07a6e in Python)
frame #25: call_function + 444 (0x109bd58ec in Python)
frame #26: _PyEval_EvalFrameDefault + 25976 (0x109bd2878 in Python)
frame #27: _PyEval_EvalCodeWithName + 2804 (0x109bd6734 in Python)
frame #28: _PyFunction_Vectorcall + 270 (0x109b07a6e in Python)
frame #29: method_vectorcall + 170 (0x109b09f9a in Python)
frame #30: call_function + 444 (0x109bd58ec in Python)
frame #31: _PyEval_EvalFrameDefault + 25976 (0x109bd2878 in Python)
frame #32: _PyEval_EvalCodeWithName + 2804 (0x109bd6734 in Python)
frame #33: PyEval_EvalCode + 100 (0x109bcc224 in Python)
frame #34: builtin_exec + 626 (0x109bc9612 in Python)
frame #35: cfunction_vectorcall_FASTCALL + 175 (0x109b438bf in Python)
frame #36: call_function + 444 (0x109bd58ec in Python)
frame #37: _PyEval_EvalFrameDefault + 25829 (0x109bd27e5 in Python)
frame #38: function_code_fastcall + 128 (0x109b078d0 in Python)
frame #39: call_function + 444 (0x109bd58ec in Python)
frame #40: _PyEval_EvalFrameDefault + 25642 (0x109bd272a in Python)
frame #41: _PyEval_EvalCodeWithName + 2804 (0x109bd6734 in Python)
frame #42: _PyFunction_Vectorcall + 270 (0x109b07a6e in Python)
frame #43: call_function + 444 (0x109bd58ec in Python)
frame #44: _PyEval_EvalFrameDefault + 25642 (0x109bd272a in Python)
frame #45: function_code_fastcall + 128 (0x109b078d0 in Python)
frame #46: call_function + 444 (0x109bd58ec in Python)
frame #47: _PyEval_EvalFrameDefault + 25642 (0x109bd272a in Python)
frame #48: function_code_fastcall + 128 (0x109b078d0 in Python)
frame #49: call_function + 444 (0x109bd58ec in Python)
frame #50: _PyEval_EvalFrameDefault + 25642 (0x109bd272a in Python)
frame #51: _PyEval_EvalCodeWithName + 2804 (0x109bd6734 in Python)
frame #52: _PyFunction_Vectorcall + 270 (0x109b07a6e in Python)
frame #53: call_function + 444 (0x109bd58ec in Python)
frame #54: _PyEval_EvalFrameDefault + 25642 (0x109bd272a in Python)
frame #55: function_code_fastcall + 128 (0x109b078d0 in Python)
frame #56: call_function + 444 (0x109bd58ec in Python)
frame #57: _PyEval_EvalFrameDefault + 25829 (0x109bd27e5 in Python)
frame #58: function_code_fastcall + 128 (0x109b078d0 in Python)
frame #59: call_function + 444 (0x109bd58ec in Python)
frame #60: _PyEval_EvalFrameDefault + 25678 (0x109bd274e in Python)
frame #61: _PyEval_EvalCodeWithName + 2804 (0x109bd6734 in Python)
frame #62: PyEval_EvalCode + 100 (0x109bcc224 in Python)
frame #63: PyRun_FileExFlags + 336 (0x109c1bed0 in Python)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 5, in <module>
  File "/Users/hsk/environments/ner_layoutlm/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1037, in from_pretrained
    raise OSError(
OSError: Unable to load weights from pytorch checkpoint file for 'layoutlm-base-uncased' at 'layoutlm-base-uncased/pytorch_model.bin'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. 

```

Maybe try updating to Transformers 4.1.1 (I just ran the following in a notebook and it works):

```
!pip install transformers

from transformers import LayoutLMForTokenClassification
model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased')
```

Thanks @NielsRogge It was a pytorch version issue. I solved it after seeing [this](https://github.com/huggingface/transformers/issues/7739#issuecomment-707214148)

Closing then, thanks for your help @NielsRogge !