A few CI fixes for DocumentQuestionAnsweringPipeline
Author: ankrgyl
Pull request: yes

# What does this PR do?

Fixes a few issues caught by CI (see [comment](https://github.com/huggingface/transformers/pull/19204#issuecomment-1277106187)).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x ] Did you write any new necessary tests?


## Who can review?

@ydshieh @Narsil @sgugger 


Comments:
I have not yet updated these tests:

```
FAILED tests/pipelines/test_pipelines_document_question_answering.py::DocumentQuestionAnsweringPipelineTests::test_large_model_pt_chunk - AssertionError: Lists differ: [{'score': 0.9974, 'answer': '1110212019', 'start': 23, [69 chars] 16}] != [{'score': 0.9967, 'answer': '1102/2019', 'start': 22, '[67 chars] 15}]
FAILED tests/pipelines/test_pipelines_document_question_answering.py::DocumentQuestionAnsweringPipelineTests::test_large_model_pt_layoutlm_chunk - AssertionError: Lists differ: [{'sc[39 chars]t': 16, 'end': 16}, {'score': 0.9998, 'answer'[31 chars] 16}] != [{'sc[39 chars]t': 15, 'end': 15}, {'score': 0.9924, 'answer'[3...
```

as I want to inspect the CI failures first. I think both of these tests are caused by tesseract OCR errors (specifically I think the CI is running a diff. version of tesseract than my local machine).

This is what we have `tesseract-ocr` on our CI runners

```bash
root@6aec4d26d7ac:/transformers# apt-show-versions tesseract-ocr
bash: apt-show-versions: command not found
root@6aec4d26d7ac:/transformers# apt show tesseract-ocr
Package: tesseract-ocr
Version: 4.1.1-2build2
Priority: optional
Section: universe/graphics
Source: tesseract
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Alexander Pozdnyakov <almipo@mail.ru>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 1573 kB
Depends: libarchive13 (>= 3.2.1), libc6 (>= 2.29), libcairo2 (>= 1.2.4), libfontconfig1 (>= 2.12.6), libgcc-s1 (>= 3.0), libglib2.0-0 (>= 2.12.0), libicu66 (>= 66.1~rc-1~), liblept5 (>= 1.75.3), libpango-1.0-0 (>= 1.37.2), libpangocairo-1.0-0 (>= 1.22.0), libpangoft2-1.0-0 (>= 1.14.0), libstdc++6 (>= 5.2), libtesseract4 (= 4.1.1-2build2), tesseract-ocr-eng (>= 4.00~), tesseract-ocr-osd (>= 4.00~)
Replaces: tesseract-ocr-data
Homepage: https://github.com/tesseract-ocr/
Download-Size: 262 kB
APT-Manual-Installed: yes
APT-Sources: http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
```

_The documentation is not available anymore as the PR was closed or merged._

Thank you! Yes, I filed #347 at some point about this. Unfortunately it looks like a different (potentially flaky) test failure occurred in this run:

```
        run_command(self._launch_args + testargs)
        result = get_results(tmp_dir)
        # Because we use --version_2_with_negative the testing script uses SQuAD v2 metrics.
>       self.assertGreaterEqual(result["eval_f1"], 28)
E       AssertionError: 21.428571428571427 not greater than or equal to 28

examples/pytorch/test_accelerate_examples.py:201: AssertionError
```

and so it did not repro the error. I'm going to spin up a VM or docker container that has tesseract 4, and then update the remaining tests there.

Thank you a lot @ankrgyl! If it's easier, we can get the new expected values from our runners, and see if it will pass with that in multiple runs. Let me know what you prefer :-)

Oh that is definitely easier. Could you help with that, or show me how to get those values?

I usually get values from report, say [here](https://github.com/huggingface/transformers/actions/runs/3231701081/jobs/5291537526) or its raw log version. Sometimes I need to run the tests inside the CI runner.

I will do that tomorrow and check if I can get those updated tests pass in a consistent way.

Okay sounds great!

Hi @ankrgyl I am not able to push to your PR branch (you don't give us the permission I think 😢 )

Could you check [this branch](https://github.com/ydshieh/transformers/commit/33fff18d421187045197f1bbfcc6a4ed72cebe3c) and see if the new values work well on your side too 🙏 ?

(I don't pay attention to the style, so you will need to re-style it before we can merge)

Hi @ydshieh the changes look good. I _think_ I just gave you write access to our fork (which should give you write permissions to the branch?). Would you mind checking if that worked?

Yes, I pushed, with the correct style.



With the latest commit, all `DocumentQuestionAnsweringPipelineTests` pass now. I can have a super happy weekend now.

Excellent! Let me know if there is anything else I can do to help.

@Narsil The tests with updated expected values in this PR are recently added in #19204. I would say this is just the environment difference (which gave the different values when @ankrgyl worked on #19204)

> I would say this is just the environment difference 

This is what we should be careful about :) If the environment provides such different results, which should either fix something so that the values are more consistent, or workaround the flaky dependency :) (If the test becomes bothering to maintain)

@Narsil the precise reason for the difference is that locally, I have tesseract 5 (the latest stable release), and the test runners have version 4, which produces slightly different OCR results. I filed https://github.com/huggingface/transformers/pull/347 some time ago about installing tesseract 5 in the docker containers used for spaces, which could help resolve the issue.

In the meantime, I can think of a few ways to ensure the tests are more consistent/robust:

- We can write up some instructions about how to update the tests within a docker container that has the same version of tesseract
- We can freeze the tesseract/OCR results into the test so that tesseract is not actually run while evaluating them (small added benefit that tests will run a bit faster)
- We can attempt to write a question and/or use a document where the results are the same b/w tesseract 4 and 5.

@ankrgyl thanks for the explanation.

Since this is `transformers` not `tesseract` fixing the version being tested to `4` is OK.
But if it becomes to hard to make stable, it's always possible to mock it in the tests so that we don't need to wipe up the whole program, and depend on its own instabilities/changes.