BERT QnA is not matching correct answer when document is in QnA format
Author: ghost
Pull request: no
Labels: name: wontfix

I have BERT trained on Squad (and without trained as well) . My documents contains question and big answers. When we try to ask question as it is and BERT to find that question within document, then it gives some arbitrary answer from some other page of document. 
What can be wrong?

Comments:
have you tried using your own data to train the model rather than using the squad1.1 or squad 2.0 data?
I am doing QnA system as well, I have my own data and I split them into train, dev and test data, then use the train and dev data to train the model, eventually it works ok on the test data.
because I am building the QnA system for my own task, not squad task, so I trained the model by using own data.


Thanks for reply and confirming that it works. @mushro00om  May I know how much is your training data? (Building a large QnA training corpus is a challenge). 

@SandeepBhutani 
My training data contain 140k questions, my dev data contain 5k questions, I haven't tried using the whole 140k to train yet because its a bit too large, I took 50k out of 140k to train, and it took roughly 3 hours.

@mushro00om : Thats a huge training data set. Unfortunately creating this corpus takes a lot of effort. So we are trying to ask same question that is there as it is in the document, based on vanilla bert-uncased (or squad trained too). 

@SandeepBhutani  Wish you good luck !

Hey @SandeepBhutani, I am facing the exact same issue. Did you solve it? I would like to talk to you about the same. 

Hi.. Not yet

On Tue, 18 Jun, 2019, 4:21 PM dhruvkp090, <notifications@github.com> wrote:

> Hey @SandeepBhutani <https://github.com/SandeepBhutani>, I am facing the
> exact same issue. Did you solve it? I would like to talk to you about the
> same.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/627?email_source=notifications&email_token=AHRBKIBNACCIYZFQWCOTNKLP3C43FA5CNFSM4HOJYYQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX57T3Q#issuecomment-503052782>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHRBKIEHHKK3S3IFTCVWWNDP3C43FANCNFSM4HOJYYQA>
> .
>


@SandeepBhutani is there any way we can connect? I would like to talk to you about this..

Please email me your contact details at sandtemp01@gmail.com

On Tue, 18 Jun, 2019, 4:28 PM dhruvkp090, <notifications@github.com> wrote:

> @SandeepBhutani <https://github.com/SandeepBhutani> is there any way we
> can connect? I would like to talk to you about this..
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/627?email_source=notifications&email_token=AHRBKIAJFWSHZTLRDSGZNSDP3C5UVA5CNFSM4HOJYYQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX6ADIY#issuecomment-503054755>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHRBKIDPOVXVESF72DCFYWLP3C5UVANCNFSM4HOJYYQA>
> .
>


@SandeepBhutani, I sent my details.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.