Question: What will they do in future experiments?
Evidence: In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT. After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations. The latter we tried to address with mixed results using feature engineering. Overall these experiments allowed us to secure a best and a second best score in different test batches. Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach.
For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance.
All our data and software is available at Github, in the previously referenced URL (end of Section 2).
In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30.
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy.
We also see potential for incorporating domain specific inference into the task e.g. using the MedNLI dataset BIBREF14. For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e.g. BIBREF16) and possibly deeper discourse representations (e.g. BIBREF17).
In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30.
In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network. In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’. Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus. To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases.
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy.
In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT. After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations. The latter we tried to address with mixed results using feature engineering. Overall these experiments allowed us to secure a best and a second best score in different test batches. Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach.
For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance.
All our data and software is available at Github, in the previously referenced URL (end of Section 2).
In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30.
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy.
We also see potential for incorporating domain specific inference into the task e.g. using the MedNLI dataset BIBREF14. For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e.g. BIBREF16) and possibly deeper discourse representations (e.g. BIBREF17).
In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30.
In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network. In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’. Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus. To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases.
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy.
In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span.****In one more experiment, we would like to add a better version of LAT contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network.
Answer: They want to create dense question answering neural network with a softmax layer for predicting answer span and to add a better version of LAT contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network.