[LayoutLMv3] Add TensorFlow implementation
Author: chrisohrstrom
Pull request: yes

Adds a TensorFlow implementation of LayoutLMv3.

We have TensorFlow weights available that we have tested are able to produce the same results as the PyTorch models at microsoft/layoutlmv3-base and microsoft/layoutlmv3-large. Can you help us upload the TF weights to those locations?

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik @NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

A fix was added for the `run_tests_layoutlmv2_and_v3` tests: #18680. Merging in `main` should resolve. 

Thank you for your comments :) I updated everything accordingly.

> * The reason for skipping `test_loss_computation` points to some funny behaviour somewhere. We should figure out what's happening and either update that comment or write a custom `test_loss_computation` for the model. This is the main thing I think is necessary to address before merging.

I know why test_loss_computation failed and have fixed it in an overriden method. I am not sure why it works on other models that accept both pixel_values and input_ids though.

> * Lots of great type hinting throughout. It would be great to add return types too for the full 💯  effect

No problem. Have added them 👍

> * We can help you with adding weights to the hub when the time comes. At the moment, the tests to compare the logits  of the pytorch and TF models are failing, and so the difference is too large for the weights to be added yet. Tests for ref:
>   
>   * `tests/models/layoutlmv3/test_modeling_layoutlmv3.py::LayoutLMv3ModelTest::test_pt_tf_model_equivalence`
>   * `tests/models/layoutlmv3/test_modeling_tf_layoutlmv3.py::TFLayoutLMv3ModelTest::test_pt_tf_model_equivalence`
> 
> To run these locally, you can do: `RUN_PT_TF_CROSS_TESTS=True pytest tests/models/layoutlmv3/test_modeling_layoutlmv3.py::LayoutLMv3ModelTest::test_pt_tf_model_equivalence`

Ah thank you, didn't know about those tests. Both of those tests should pass now. The problem was that we had implemented the relative position embeddings through an Embedding layer, but the PyTorch implementation uses a one-hot encoding followed by a dense layer. It "confused" the weight conversion function. The TensorFlow implementation does the same as the PyTorch implementation now.


@ChrisFugl Thank you for adding the TF version of LayoutLMv3 🧡 I'll be the third (and last) reviewer, so hopefully this will be quick 🔥 

While I do the actual review, there are two things we will need before we merge:
1. As you mentioned, we will need TF weights in the LayoutLMv3 repos. We have just finished the pieces to make the process open to all (as opposed to exclusive to HF members), so I'd like to ask you to be our first user 🎉 Assuming you are logged in to the Hub in your terminal (if not, run `huggingface-cli login`) and are on this branch, run `transformers-cli pt-to-tf --model-name foo/bar`, where `foo/bar` is the target model repo. Assuming it passes the CLI's checks, it will open a PR on the model hub. Please tag `@nielsr, @sgugger, @joaogante` in the opened Hub PRs :)
2. After the TF model weights are merged, re-run the test suite for the model locally and confirm that all tests pass (command: `NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 py.test -vv tests/models/layoutlmv3/modeling_tf_layoutlmv3.py`)

> @ChrisFugl Thank you for adding the TF version of LayoutLMv3 🧡 I'll be the third (and last) reviewer, so hopefully this will be quick 🔥
> 
> While I do the actual review, there are two things we will need before we merge:
> 
> 1. As you mentioned, we will need TF weights in the LayoutLMv3 repos. We have just finished the pieces to make the process open to all (as opposed to exclusive to HF members), so I'd like to ask you to be our first user 🎉 Assuming you are logged in to the Hub in your terminal (if not, run `huggingface-cli login`) and are on this branch, run `transformers-cli pt-to-tf --model-name foo/bar`, where `foo/bar` is the target model repo. Assuming it passes the CLI's checks, it will open a PR on the model hub. Please tag `@nielsr, @sgugger, @joaogante` in the opened Hub PRs :)
> 2. After the TF model weights are merged, re-run the test suite for the model locally and confirm that all tests pass (command: `NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 py.test -vv tests/models/layoutlmv3/modeling_tf_layoutlmv3.py`)

Thanks! Sure, I am happy to try it out :)

I get an error though when running `transformers-cli pt-to-tf --model-name microsoft/layoutlmv3-base`:

```
/tmp/microsoft/layoutlmv3-base is already a clone of https://huggingface.co/microsoft/layoutlmv3-base. Make sure you pull the latest changes with `repo.git_pull()`.
No detected architecture, using AutoModel/TFAutoModel
Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFLayoutLMv3Model: ['layoutlmv3.embeddings.position_ids']
- This IS expected if you are initializing TFLayoutLMv3Model from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFLayoutLMv3Model from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFLayoutLMv3Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
Reusing dataset cifar10 (/Users/x/y/z/.cache/huggingface/datasets/cifar10/plain_text/1.0.0/447d6ec4733dddd1ce3bb577c7166b986eaa4c538dcd9e805ba61f35674a9de4)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
/Users/x/y/z/transformers/.env/lib/python3.8/site-packages/transformers/modeling_utils.py:721: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFLayoutLMv3Model: ['layoutlmv3.embeddings.position_ids']
- This IS expected if you are initializing TFLayoutLMv3Model from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFLayoutLMv3Model from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFLayoutLMv3Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
Traceback (most recent call last):
  File "/Users/x/y/z/transformers/.env/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/Users/x/y/z/transformers/.env/lib/python3.8/site-packages/transformers/commands/transformers_cli.py", line 55, in main
    service.run()
  File "/Users/x/y/z/transformers/.env/lib/python3.8/site-packages/transformers/commands/pt_to_tf.py", line 289, in run
    max_crossload_output_diff = max(output_differences.values())
ValueError: max() arg is an empty sequence
```

I debugged line 289 of pt_to_tf.py and found that `output_differences` is empty because it excludes model outputs with "hidden" in the output name, and both the PyTorch and TensorFlow model only outputs these keys:

* last_hidden_state
* hidden_states

That seems like correct behaviour on the model side - is the check buggy?


@ChrisFugl probably the `pt-to-tf` CLI is missing the correct behavior for this type of models -- will take a look :)

> Thank you for this high-quality implementation -- I appreciated the little details like type hints and better variable names (wrt PT implementation) ❤️
> 
> I've added a few minor comments. Some model classes are also missing a few `_keys_to_ignore` members (they don't impact correctness, but prevent some useless warnings from popping up)

That is great! And thank you :)

I was not entirely sure of which keys would be relevant to ignore, but I think it is only "position_ids"?

Let me know when the CLI is ready - then I will try and push the model weights again.

@ChrisFugl copying the keys to ignore from PT should be good enough, we can tweak after merging if we are still missing some.

As for the CLI, I'll be doing a fix now -- I will tag you in the PR once it is open 👍 

@ChrisFugl if you rebase this PR with `main` you should be able to run the CLI now. 

Note: I've tried converting `microsoft/layoutlmv3-base` (w/o opening a PR) and it failed a check in one of its hidden layers. The maximum error is slightly above the constant we defined. This constant is a "guesstimate", it's okay to override it with `--max-hidden-error` if the problem consists of a minor difference in one layer.

@gante I just tried again after rebasing and increasing the max error, but I get a new error now. I think it is because it recognises the architecture from the config now, which it didn't before (see detected architecture). I don't know why that part changed. What do you think?

```
/tmp/microsoft/layoutlmv3-base is already a clone of https://huggingface.co/microsoft/layoutlmv3-base. Make sure you pull the latest changes with `repo.git_pull()`.
Detected architecture: LayoutLMv3Model
All PyTorch model weights were used when initializing TFLayoutLMv3Model.

All the weights of TFLayoutLMv3Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
Reusing dataset cifar10 (/Users/x/y/z/.cache/huggingface/datasets/cifar10/plain_text/1.0.0/447d6ec4733dddd1ce3bb577c7166b986eaa4c538dcd9e805ba61f35674a9de4)
/Users/x/y/z/transformers/src/transformers/modeling_utils.py:721: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
All PyTorch model weights were used when initializing TFLayoutLMv3Model.

All the weights of TFLayoutLMv3Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
Traceback (most recent call last):
  File "/Users/x/y/z/transformers/.env/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/Users/x/y/z/transformers/src/transformers/commands/transformers_cli.py", line 55, in main
    service.run()
  File "/Users/x/y/z/transformers/src/transformers/commands/pt_to_tf.py", line 290, in run
    raise ValueError(
ValueError: Something went wrong -- the config file has architectures (['LayoutLMv3Model']), but no model head output was found. All outputs start with 'hidden'

```

@ChrisFugl Interesting -- that model has no `architectures` field in [its config](https://huggingface.co/microsoft/layoutlmv3-base/blob/main/config.json). From your error message, it means that it is being initialized somewhere, implying that the logic for the conversion script is wrong in these cases or that is something wrong with the model's configuration class.

Going to dig deeper!

[Apologies for all these hiccups, we are still tuning the conversion script to work in all cases]

@ChrisFugl I've rerun the CLI from this branch, and it works fine on my end -- i.e. `architectures` is set to `None`, as expected, and the error above doesn't get triggered 🤔 

Is there any chance the files in `/tmp/microsoft/layoutlmv3-base` were touched? Can you try deleting the folder and running the CLI again? 

@gante No worries about the hiccups :)

I don't know why it worked since I haven't touched the files in /tmp/microsoft/layoutlmv3-base myself, but it worked to delete the folder.

I now have permission issues though. I am logged in with `huggingface-cli login` and have saved the token in git credentials store.

```
Cloning https://huggingface.co/microsoft/layoutlmv3-base into local empty directory.
Download file pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 478M/478M [10:09<00:00, 823kB/s]
Clean file pytorch_model.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 478M/478M [05:04<00:00, 1.65MB/s]
No detected architecture, using AutoModel/TFAutoModel█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 478M/478M [05:04<00:00, 1.93MB/s]
All PyTorch model weights were used when initializing TFLayoutLMv3Model.

All the weights of TFLayoutLMv3Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
Reusing dataset cifar10 (/Users/x/y/.cache/huggingface/datasets/cifar10/plain_text/1.0.0/447d6ec4733dddd1ce3bb577c7166b986eaa4c538dcd9e805ba61f35674a9de4)
/Users/x/y/z/transformers/src/transformers/modeling_utils.py:721: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
All PyTorch model weights were used when initializing TFLayoutLMv3Model.

All the weights of TFLayoutLMv3Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
All model checkpoint layers were used when initializing TFLayoutLMv3Model.

All the layers of TFLayoutLMv3Model were initialized from the model checkpoint at /tmp/microsoft/layoutlmv3-base.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
Uploading the weights into a new PR...
Traceback (most recent call last):
  File "/Users/x/y/z/transformers/.env/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/Users/x/y/z/transformers/src/transformers/commands/transformers_cli.py", line 55, in main
    service.run()
  File "/Users/x/y/z/transformers/src/transformers/commands/pt_to_tf.py", line 366, in run
    hub_pr_url = create_commit(
  File "/Users/x/y/z/transformers/.env/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1817, in create_commit
    upload_lfs_files(
  File "/Users/x/y/z/transformers/.env/lib/python3.8/site-packages/huggingface_hub/_commit_api.py", line 211, in upload_lfs_files
    batch_actions, batch_errors = post_lfs_batch_info(
  File "/Users/x/y/z/transformers/.env/lib/python3.8/site-packages/huggingface_hub/lfs.py", line 217, in post_lfs_batch_info
    resp.raise_for_status()
  File "/Users/x/y/z/transformers/.env/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/microsoft/layoutlmv3-base.git/info/lfs/objects/batch
```

@ChrisFugl permission to programmatically open a Hub PR as a non-admin was just released -- run `pip install huggingface_hub -U` and then you should be able to open the PRs :D 

@gante Ah thank you - got it working now :D 

https://huggingface.co/microsoft/layoutlmv3-base/discussions/3
https://huggingface.co/microsoft/layoutlmv3-large/discussions/1

Awesome 🚀 

Next steps:
1. Wait for the `base` model weights to be merged, as tests depend on it;
2. [can be done now] Rebase the PR with `main`, as the diff contains changes that are already in `main` (pull main -> rebase branch with main -> force push);
3. Run the slow tests locally and confirm that they pass, after 1. and 2. (`NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 py.test -vv tests/models/layoutlmv3/test_modeling_tf_layoutlmv3.py`);
4. After the three points above are complete, I will merge the PR 🙌 

@ChrisFugl BTW, this will be the first TF PR where an external contributor was able to do the whole process on their own 🎉  (previously only HF admins could add weights)

@ChrisFugl if it is not asking too much, can you open a PR for [this](https://huggingface.co/microsoft/layoutlmv3-base-chinese) layoutlmv3 as well? :D

> @ChrisFugl BTW, this will be the first TF PR where an external contributor was able to do the whole process on their own 🎉 (previously only HF admins could add weights)

@gante Well I feel honoured :)

> @ChrisFugl if it is not asking too much, can you open a PR for [this](https://huggingface.co/microsoft/layoutlmv3-base-chinese) layoutlmv3 as well? :D

Not at all. There seems to be some problem though. I have tried to remove /tmp/microsoft and run it again, but that didn't work.

```
/tmp/microsoft/layoutlmv3-base-chinese is already a clone of https://huggingface.co/microsoft/layoutlmv3-base-chinese. Make sure you pull the latest changes with `repo.git_pull()`.
No detected architecture, using AutoModel/TFAutoModel
All PyTorch model weights were used when initializing TFLayoutLMv3Model.

All the weights of TFLayoutLMv3Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFLayoutLMv3Model for predictions without further training.
Reusing dataset cifar10 (/Users/x/y/z/.cache/huggingface/datasets/cifar10/plain_text/1.0.0/447d6ec4733dddd1ce3bb577c7166b986eaa4c538dcd9e805ba61f35674a9de4)
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'XLMRobertaTokenizer'. 
The class this function is called from is 'LayoutLMv3Tokenizer'.
Traceback (most recent call last):
  File "/Users/x/y/z/transformers/.env/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/Users/x/y/z/transformers/src/transformers/commands/transformers_cli.py", line 55, in main
    service.run()
  File "/Users/x/y/z/transformers/src/transformers/commands/pt_to_tf.py", line 276, in run
    pt_input, tf_input = self.get_inputs(pt_model, config)
  File "/Users/x/y/z/transformers/src/transformers/commands/pt_to_tf.py", line 218, in get_inputs
    processor = AutoProcessor.from_pretrained(self._local_dir)
  File "/Users/x/y/z/transformers/src/transformers/models/auto/processing_auto.py", line 250, in from_pretrained
    return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/Users/x/y/z/transformers/src/transformers/processing_utils.py", line 182, in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/Users/x/y/z/transformers/src/transformers/processing_utils.py", line 226, in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
  File "/Users/x/y/z/transformers/src/transformers/tokenization_utils_base.py", line 1768, in from_pretrained
    return cls._from_pretrained(
  File "/Users/x/y/z/transformers/src/transformers/tokenization_utils_base.py", line 1798, in _from_pretrained
    slow_tokenizer = (cls.slow_tokenizer_class)._from_pretrained(
  File "/Users/x/y/z/transformers/src/transformers/tokenization_utils_base.py", line 1923, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/Users/x/y/z/transformers/src/transformers/models/layoutlmv3/tokenization_layoutlmv3.py", line 324, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

> 3. Run the slow tests locally and confirm that they pass, after 1. and 2. (`NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 py.test -vv tests/models/layoutlmv3/test_modeling_tf_layoutlmv3.py`);

@gante Oh and I forgot to mention that the tests pass now.

@ChrisFugl no worries about the last conversion, we'll see what's wrong afterward. Regarding failing tests, it seems like it is a problem on our end. 

No further input from you should be needed to merge this PR 🎉 

@gante That is awesome! Of course still happy to help if it is needed :)

@ChrisFugl the underlying issue is fixed -- can you rebase the PR please? 🙏 

We should be able to merge the PR after that 🎉 

> @ChrisFugl the underlying issue is fixed -- can you rebase the PR please? 🙏
> 
> We should be able to merge the PR after that 🎉

@gante I followed your instructions on how to rebase, but I am afraid that it messed things up a bit. There are now 45 files changed in this PR - most of which this PR haven't even touched upon.

I resolved all conflicts while rebasing by consistently accepting the incoming changes. The tests still pass.

I think the problems with rebasing must be because I merged upstream/main into this PR prior to the first rebase.

Sorry about the rebasing issues. I am not sure what the right course of action is now?

@ChrisFugl can you try the solutions from [this StackOverflow page](https://stackoverflow.com/questions/16306012/github-pull-request-showing-commits-that-are-already-in-target-branch)?

The commits will be squashed at merge time, so everything will be fine as long as the diff we see here is correct :) 

@gante Ah that was simple :) Thanks! Should be all good now.

@ChrisFugl Fantastic, I'll merge the PR now! 🧡 

Thank you so much for participating through the whole process, even with all the hiccups. LayoutLMv3 is a very useful model, I'm confident TF users will deeply appreciate your contribution 🤗 

There are two post-merge action points. As always, a helping hand is appreciated (but we will do them otherwise):
- Find and fix the tokenizer problem that is preventing the conversion of `microsoft/layoutlmv3-base-chinese` (and convert the model)
- [EDIT: we are going to write about it today, ~6pm CEST] Announce the TF model on social media (twitter and LinkedIn)! Unannounced changes are changes that most users will miss. [EDIT2: shared here -- https://twitter.com/joao_gante/status/1564650841688006656 ; https://www.linkedin.com/posts/gante_tensorflow-ai-microsoft-activity-6970415100406951936-dywm?utm_source=share&utm_medium=member_desktop]

Thank you @gante - and for your help with getting the PR merged in 😁 Super nice to get it merged in.

I can give a shot at the tokenizer problem 👌 

Happy to help with sharing as well. Already shared the LinkedIn post. We will probably also do some more spreading of the news in my team :)