run_ner.py --do_predict inference mode errors. Right data format?
Author: zampierimatteo91
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello again,

I'm here to bother you one more time.

I fine-tuned preloaded BioBERT weights on a custom dataset to run biomedical NER.

Now I want to use the model for inference mode on a 'raw' set of documents. I renamed this set 'test.txt' and formatted it the following way (documents are separated by '-DOCSTART- (num_doc)' lines):

```
to O
be O
referred O
to O
the O
location O
of O
the O
disease O
in O
the O
skeletal O
structures O
examined O
; O
           
unchanged O
the O
areas O
of O
bone O
rarefaction O
reported O
to O
the O
sternum O
as O
a O
result O
of O
median O
sternotomy O
. O
```

I had to add the 'fake' labels on the right and place a space " " between col1 and col2.

The error I now get is:
```
Traceback (most recent call last):
  File "run_ner.py", line 531, in <module>
    main()
  File "run_ner.py", line 522, in main
    output_line = line.split()[0] + " " + predictions[example_id].pop(0) + "\n"
IndexError: list index out of range
```
Many thanks again.

Comments:
Three questions:
- the error occurs at line 522, so at line 507 you've saved the file called `test_results.txt`. Do you see the content of this file and whether is it correct?
- the input file has been formatted as CoNLL-2003?
- **N.B**: moreover, the code block from line 512 to line 525 save to .txt file the predictions obtained from the NER. But already at line 505 you have the `predictions` variable. Have you seen the content of this variable? Maybe, it is only a saving problem.

> ##  Questions & Help
> Hello again,
> 
> I'm here to bother you one more time.
> 
> I fine-tuned preloaded BioBERT weights on a custom dataset to run biomedical NER.
> 
> Now I want to use the model for inference mode on a 'raw' set of documents. I renamed this set 'test.txt' and formatted it the following way (documents are separated by '-DOCSTART- (num_doc)' lines):
> 
> ```
> to O
> be O
> referred O
> to O
> the O
> location O
> of O
> the O
> disease O
> in O
> the O
> skeletal O
> structures O
> examined O
> ; O
>            
> unchanged O
> the O
> areas O
> of O
> bone O
> rarefaction O
> reported O
> to O
> the O
> sternum O
> as O
> a O
> result O
> of O
> median O
> sternotomy O
> . O
> ```
> 
> I had to add the 'fake' labels on the right and place a space " " between col1 and col2.
> 
> The error I now get is:
> 
> ```
> Traceback (most recent call last):
>   File "run_ner.py", line 531, in <module>
>     main()
>   File "run_ner.py", line 522, in main
>     output_line = line.split()[0] + " " + predictions[example_id].pop(0) + "\n"
> IndexError: list index out of range
> ```
> 
> Many thanks again.

Ciao @TheEdoardo93 ,
Thanks for your support!

 - I formatted the test set trying to follow the indications from the tutorial on the german-eval, with the first column being the token and the second being the B-I-O tags (in this set it's just a pile of Os to fill the column). They are space-separated.
- `test_results.txt` is saved and shows precision, recall, f-1, and loss. All are terrible of course, as the test set was actually filled with the dummy BIO tags.
- `test_predictions.txt` is truncated after about 50 lines of token+BIO prediction.

I'm now trying to print the content of `predictions`, I'll let you know.

I wait your `predictions` variable content :D
We can implement a saving method that works as we expect (and not using the code lines in the `run_ner.py` script and see what happens!)

> Ciao @TheEdoardo93 ,
> Thanks for your support!
> 
> * I formatted the test set trying to follow the indications from the tutorial on the german-eval, with the first column being the token and the second being the B-I-O tags (in this set it's just a pile of Os to fill the column). They are space-separated.
> * `test_results.txt` is saved and shows precision, recall, f-1, and loss. All are terrible of course, as the test set was actually filled with the dummy BIO tags.
> * `test_predictions.txt` is truncated after about 50 lines of token+BIO prediction.
> 
> I'm now trying to print the content of `predictions`, I'll let you know.

I'm back.

What I did was: changed columns' separation from tab to space (I was wrong in the previous comment, I thought I already changed it).

Now the code runs properly and `test_predictions.txt` is complete.
This is a snapshot of `print(predictions)`:
```
[['O', 'O', 'B-Organism_subdivision', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Organ', 'O', 'B-Organ', 'O', 'B-Multi-tissue_structure', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Organism_subdivision', 'I-Organism_subdivision', 'O', 'O', 'O', 'O', 'O', 'B-Cancer', 'I-Cancer', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'B-Immaterial_anatomical_entity', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'B-Multi-tissue_structure', 'B-Multi-tissue_structure', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Immaterial_anatomical_entity', 'O', 'O'], ..., ['O', 'O', 'O', 'O', 'B-Organ', 'O', 'B-Organ', 'O', 'B-Organ', 'O', 'B-Organ', 'O'],  ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Tissue', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Tissue', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Tissue', 'O', 'O', 'O', 'O', 'B-Multi-tissue_structure', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]
```
There is another minor issue I guess, i.e. a long series of warnings about no predictions because of exceeded maximum sequence length. The non-predicted tokens appear to be not really relevant for my final interest, but I'd like to have a complete output nonetheless.
I will try to place a newline not only after usual end-of-sentence punctuation (.!?), but also after semi-colons and colons, in order to split each document in more pieces.
Is it a strategy that makes sense or have I misinterpreted the meaning of the maximum sequence length?


Do you have sentences with length greater than **512 tokens**? BioBERT allows to have sentences with 512 tokens length maximum, as stated in the [paper](https://watermark.silverchair.com/btz682.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAApcwggKTBgkqhkiG9w0BBwagggKEMIICgAIBADCCAnkGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMYCT-oIlIkXngiEprAgEQgIICSsx4GD3tbVdWpu1VPO7avpF_9v-YeT2MtTj6ysC7p_RRxtqG74n5C_tpst2LvM8UKCDfiuvU4bWi8PfJiJfKoQiWwR4AH4K0JC2m-Q8YC2V8_Jfuk-AL_CR8oWJc2U40FdB2fyV2tdoYxV0v1A35Qjg6PEujCkc3ztZqcGctW1_awddqskqkGF8Oz02kiwFQyHHlMPuRAewMopnxii_Pqo-nSNynCl03WCCGUKPbkC-vbPwIjo7vjz-opJQaNcTrNOLI8xPzQm3qT5_R85w3mm-CpHbo2rj4LW7YkJrswc7Z4KOlEfdq7AC5WkiIYhYqyauVLTDNzVYwSYJ_L6RsPeNlfxv3rm71J7fppWO_fu4Mbn8vnzmjKS0nqxdEbRcI4bGkpkjCvW-sVa3FIcRbNlOp_fH_PTeMf3VwIR_wGR0Nrw_80_BMzqy774SB1LitxarWsA7h3dU7Gp1f162TloTdqISAsTzfJJSTa4YVU2qHDp2iRzghvsBlXGhtuuiNkLQ_TblRFq3hdMpLtpHH5KlfahZ0tMvfBvbc_YGLi-9U5NmQbUnM0unhb73mQ5SneLAAD9JlLQv-4pXwYDIGi9ekn5G2RwueTOKSiKji8dm1rCtmUFXVL56WsPUdNkgJROoqGCC87_iVdV95TjpL7jVvNfOX8Bvh1eF_iCGyfrsKyK1aDpvY8B4vt3uUJowPlFjDo21AXOe53aAgnb9yay-t53WzmTNw-Q6lfZNiWsSQn9H1cUi7g8P5bRruZkmL8HaYlZje8TVNIn4).
> The maximum sequence length was fixed to 512

 If you have sentences with more than 512 tokens, you have to apply different workaround, e.g. splitting a sentence length 1024 in two different sentences of 512 length and combine in some manner their output.

However, the strategy you've proposed (e.g. split by comma, dot, semi-column, etc.) works! Try to follow this approach and share the results with us! I suggest you to do a visual evaluation/comparison between the current output and the output you'll obtain with the strategy highlight by you.

> I'm back.
> 
> What I did was: changed columns' separation from tab to space (I was wrong in the previous comment, I thought I already changed it).
> 
> Now the code runs properly and `test_predictions.txt` is complete.
> This is a snapshot of `print(predictions)`:
> 
> ```
> [['O', 'O', 'B-Organism_subdivision', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Organ', 'O', 'B-Organ', 'O', 'B-Multi-tissue_structure', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Organism_subdivision', 'I-Organism_subdivision', 'O', 'O', 'O', 'O', 'O', 'B-Cancer', 'I-Cancer', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'B-Immaterial_anatomical_entity', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'B-Multi-tissue_structure', 'B-Multi-tissue_structure', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'], ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Immaterial_anatomical_entity', 'O', 'O'], ..., ['O', 'O', 'O', 'O', 'B-Organ', 'O', 'B-Organ', 'O', 'B-Organ', 'O', 'B-Organ', 'O'],  ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Tissue', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Tissue', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-Tissue', 'O', 'O', 'O', 'O', 'B-Multi-tissue_structure', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]
> ```
> 
> There is another minor issue I guess, i.e. a long series of warnings about no predictions because of exceeded maximum sequence length. The non-predicted tokens appear to be not really relevant for my final interest, but I'd like to have a complete output nonetheless.
> I will try to place a newline not only after usual end-of-sentence punctuation (.!?), but also after semi-colons and colons, in order to split each document in more pieces.
> Is it a strategy that makes sense or have I misinterpreted the meaning of the maximum sequence length?

Quite funnily, now a lot more tokens are without predictions.
What I did was just adding a newline after each semicolon with `sed`.

A question that I thought was easy to answer to: what constitutes a sequences in BERT relative to this task? Is it a sequence of tokens between empty lines? Or between defined punctuation?

Taken from the official BERT [paper](https://arxiv.org/pdf/1810.04805.pdf):

> Throughout this work, a “sentence” can be an arbitrary span of contiguous text, rather than an actual linguistic sentence. A “sequence” refers to the input token sequence to BERT, which may be a single sentence or two sentences packed together.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.