Question: Why does the bidirectional model perform better than the forward-only model?
Evidence: In this work we find that all six of the ERP components from BIBREF0 can be predicted above chance by a model which has been pretrained using a language modeling objective and then directly trained to predict the components. This is in contrast to prior work which has successfully linked language models to the N400 BIBREF0 and P600 BIBREF7 but not the other ERP components. We also note that contrary to BIBREF7 , we find that an LSTM does contain information that can be used to predict EEG data, and in particular that it can predict the P600. We speculate that the analysis used in BIBREF7 did not find reliable effects because the language models were related to the EEG data through functions chosen a priori (the surprisal, and the `distance' metric). These functions, though interpretable, might be interpretable at the cost of losing much of the information in the representations learned by the network.
In addition, we show through our multitask learning analysis that information is shared between ERP components, and between ERP components and behavioral data. Although these relationships must be viewed with caution until they can be verified across multiple datasets and with more variation in neural network architectures, here we consider some potential reasons for our findings. The broad point we wish to make is that by better understanding which ERP components share information with each other and with behavioral data through the type of analysis we present here (multitask learning) or other means, we can better understand what drives each ERP component and in turn the processes involved in human language comprehension.
There are at least three possible explanations for why the bidirectional model performs better than the forward-only model. First, it is possible that when a human reads a sentence, he or she predicts the upcoming language input. Under this hypothesis, a model with access to the future language input can do a better job of predicting the current brain activity because the future language is reflected in that brain activity. Second, it is possible that a bidirectional model is simply able to produce better embeddings for each word in the input because it has more context than a forwardonly model. For example, the bidirectional model might be (implicitly) better at anaphora resolution given more context. Under this hypothesis, the additional context given to the model partially compensates for its relative deficit of real-world knowledge compared to a human. Where a human can in many cases solve the anaphora resolution problem by using background knowledge and does not need to see the future language input, a model benefits from additional context. Finally, in our setup, the bidirectional model has more parameters than the forward-only model, and the additional degrees of freedom might give the model an advantage in predicting brain activity.
Answer: People may predict the upcoming language input when reading, it is possible that a bidirectional model is simply able to produce better embeddings for each word in the input and the bidirectional model has more parameters than the forward-only model in authors' setup.