SIGNAL: Dataset for Semantic and Inferred Grammar Neurological Analysis of Language
Authors: Anna Zhuravleva, Ekaterina Voloshina, Anastasia Cheveleva, Ilia Semenkov, Oleg Serikov, Alex Ossadtchi


Reviewer paper summaries: This paper presents SIGNAL, an EEG dataset focused on semantic/grammatical incongruities in text. The sentences presented to participants are simplified versions of sentences from RuSentEval, a Russian-language probing dataset. They construct sentences that are grammatically incongruent, semantically incongruent, or both, in contrast to congruent sentences. They use this dataset to probe differences between processing of the normal sentences and the incongruent sentences, and find that there are significant differences in channels over time to distinguish these conditions. They also conduct a probing experiment with a Russian BERT model.
Summary of strengths: - The new dataset could be useful to researchers studying psycholinguistics or working on the connection between language models and brain responses. As it takes significant effort to gather EEG recordings for participants, the data contribution is thus valuable (moreso than datasets in other areas of NLP on average). 

- The dataset is also non-English, which is a benefit since most datasets I’m aware of in this area are English-based, biasing current findings toward the English language.
Summary of weaknesses: - I am not sure about the novelty of this dataset and research questions. Although I haven’t seen resources related to brain imaging of semantic/grammatical incongruencies in the Russian language specifically, the study of grammatical and semantic anomalies has been thoroughly investigated back to the 1980s (e.g. Kutas and Hilyard 1983), and the entire line of N400/P600 research touches on this as well. Although it may be a new contribution to scale this up and collect a non-English based dataset, I am not convinced that the (type of) dataset itself provides a novel contribution. If there are features unique to this dataset that have not been collected in previous studies, the authors should make this more clear in the related work. A dataset being “aimed for comparison of human signals and language model responses” is not a good argument as if we’re just probing the response of the language model or checking how well a language model can predict brain signals, as we can do this with any existing neuroimaging dataset, as we just need to feed in the text/brain responses to the language model. 

- The probing experiments don’t seem particularly well designed. I may be misreading this section since it’s not very clear, but is the classification task simply based on categorizing normal/S anomalous/G anomalous/both anomalous based on hidden representations of the text? This doesn’t seem to involve the EEG recordings at all, so it’s unclear what connection this is meant to draw between the LM and human brain responses. The authors should clarify the intentions of this experiment. If it’s just for validation of the dataset, I’m not sure this is very important, as they already did human validation. This relates to the claim of the dataset being “aimed for comparison of human signals and language model responses”, as this is not really supported by the experiments. The authors should try to draw a more explicit comparison, perhaps by using the LM representation + regression to actually try to predict the EEG response, and seeing if this is harder to predict for the anomalous words/sentences.
Comments, suggestions and typos: - Figure 1: The text is too small to be read. The authors could consider increasing text size and making the diagram more readable
Overall assessment scores: 2.5