<figure><img src="image_1.png" /></figure>

HAL
open science

# Past Word Context Enables Better MEG Encoding Predictions than Current Word in Listening Stories

Subba Reddy Oota, Nathan Trouvain, Frédéric Alexandre, Xavier Hinaut

► To cite this version:

Subba Reddy Oota, Nathan Trouvain, Frédéric Alexandre, Xavier Hinaut. Past Word Context Enables Better MEG Encoding Predictions than Current Word in Listening Stories. NeuroFrance 2023, May 2023, Lyon, France. . [https://hal-04154794](https://hal-04154794)

HAL Id: hal-04154794

https://inria.hal.science/hal-04154794

Submitted on 7 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Past Word Context Enables Better MEG Encoding Predictions
than Current Word in Listening Stories

S. Reddy Oota¹,²,³*, N. Trouvain¹,²,³†, F. Alexandre¹,²,³, X. Hinaut¹,²,³‡

¹Inria, Bordeaux, France

²IMN, UMR 5293, Bordeaux, France

³LaBRI, UMR 5800, Talence, France

Contact:

*subba-reddy.oota@inria.fr
†nathan.trouvain@inria.fr
‡xavier.hinaut@inria.fr

Abstract

> Brain encoding is the process of mapping stimuli to brain activity. In this study, we encode brain activity recorded through magnetoencephalography (MEG) to explore the nature of language processing in the brain.

> MEG enables us to precisely examine the timing of language processing during a naturalistic task: 8 subjects listened to 4 independent stories (Gwilliams et al. 2022).

> We parsed or encoded words using several semantic embeddings or features, syntactic parsers, or language models, and predicted MEG activity using these word representations.

> We find that only contextual representations of words significantly predict brain activity at word onset. Moreover, the strong dependence of predictivity on past word context suggests that the brain encodes words in transient representations of their past context.

> Those findings are aligned with previous studies on MEG encoding showing past information retention at phonemic scale (Gwilliams et al. 2022), and echo research on bird song production showing a preference for past actions encoding at ambiguous transition points in song in HVC (a song-related brain area in songbirds; Cohen et al. 2020).

Brain encoding using word representations

Dataset: MASC-MEG (8 subjects listening to 4 stories)

<figure><img src="image_3.png" /></figure>

Word representations

<figure><img src="image_4.png" /></figure>

> Complexity Metrics (CM)

> Part-of-speech (POS)

> Dependency tags (DEP)

> GloVe (one 300-dimensions vector per word, irrespective of word context)

> BERT (one 768-dimensions vector from the intermediate encoder layer (layer-7) to obtain the embeddings.)

Static: Word is represented out of context

Semantic:
Word is represented by its meaning

<figure><img src="image_5.png" /></figure>

Syntaxic: Word is represented by its role in the sentence

Contextual: Word is represented with its surrounding context

Context is all you need

Among all word representations, only contextual representations (BERT) are able to significantly predict MEG activity at word onset.

(One sample t-test, p < 0.05, FDR correction)

<figure><img src="image_6.png" /></figure>

Predictivity is function of context size and direction

<figure><img src="image_7.png" /></figure>

< For each word, BERT representations are built from varying context sizes, e.g. from 1 to 5 words in the past or future, including current word.

<figure><img src="image_8.png" /></figure>

< Predictivity is function of past context size. Peak predictivity of MEG activity is obtained 200 ms after word onset. Representations built using long contexts correlate more to brain activity than using short contexts.

Predictivity is dependent on context direction: Past context is more correlated to brain activity than future context. >

<figure><img src="image_9.png" /></figure>

MEG activity is the product of current word information and short-term context

<figure><img src="image_10.png" /></figure>

Predictivity is mostly significative in temporal and inferior frontal lobes

<figure><img src="image_11.png" /></figure>

Method

> Preprocessing: MEG activity is filtered (0.5Hz-30Hz) and epoched in 800ms windows around word onset (-0.2s before onset, +0.6s after word onset). First 200ms serve as baseline for detrending.

> Encoding models are fitted using linear regression with L2 regularization (regularization parameter found through 10-fold cross validation, for each story and each sensors and timepoints).

> R² score significance is assessed using a permutation test (5000 random permutations within each story, FDR corrected using Benjamini-Hochberg method with α=0.05)

Conclusion

> Brain activity seems to reflect contextual information. MEG activity is only significantly predicted by BERT contextual embeddings, and predictivity is function of past context size.

> Words encoded with future context show very low predictivity: If predictions are made by the brain, they can not be retrieved using BERT encodings.

> High predictivity with lagged encodings suggests that past context plays an important role in encoding current word information in the brain

> In the speaker's brain, there appears to be an "encoding center of mass" that is responsible for processing long-time dependencies and encoding past actions rather than future actions. This is supported by studies on songbirds, where the brain area involved in managing these dependencies preferentially encodes past actions, particularly during phrases with history-dependent transitions in song. Similarly, research on phoneme representations suggests that past events or actions are held in memory until they are used to disambiguate future events or actions (Gwilliams et al. 2022)

References

L. Gwilliams, G. Flick, A. Marantz, L. Pylkkanen, D. Poeppel, and J.-R. King, "Meg-masc: a high-quality magneto-encephalography dataset for evaluating natural speech processing," *arXiv preprint arXiv:2208.11488*, 2022.

L. Gwilliams, J.-R. King, A. Marantz, and D. Poeppel, "Neural dynamics of phoneme sequences reveal position-invariant code for content and order," *Nature communications*, vol. 13, no. 1, p. 6606, 2022.

Y. Cohen, J. Shen, D. Semu, D. P. Leman, W. A. Liberti III, L. N. Perkins, D. C. Liberti, D. N. Kotton, and T. J. Gardner, "Hidden neural states underlie canary song syntax," *Nature*, vol. 582, no. 7813, pp. 539–544, 2020.