Interpreting mental state decoding with deep learning models

Deep learning (DL) models have transformed many areas of research and industry through their unparalleled ability to autonomously learn versatile representations of complex data. Because of this empirical success, neuroimaging researchers have started applying DL models to mental state decoding analyses, hoping that they can provide novel insights into the association between mental states (e.g., accepting/rejecting a gamble) and brain activity, beyond the capabilities of conventional machine learning approaches. Several challenges at the intersection of functional neuroimaging and DL research hinder the broad application of DL models in mental state decoding. We review recent advances in both fields to provide a set of solutions to these challenges and enable researchers to fully leverage the potential of DL models in mental state decoding. In mental state decoding, researchers aim to identify the set of mental states (e.g., experiencing happiness or fear) that can be reliably identified from the activity patterns of a brain region (or network). Deep learning (DL) models are highly promising for mental state decoding because of their unmatched ability to learn versatile representations of complex data. However, their widespread application in mental state decoding is hindered by their lack of interpretability, difficulties in applying them to small datasets, and in ensuring their reproducibility and robustness. We recommend approaching these challenges by leveraging recent advances in explainable artificial intelligence (XAI) and transfer learning, and also provide recommendations on how to improve the reproducibility and robustness of DL models in mental state decoding. In mental state decoding, researchers aim to identify the set of mental states (e.g., experiencing happiness or fear) that can be reliably identified from the activity patterns of a brain region (or network). Deep learning (DL) models are highly promising for mental state decoding because of their unmatched ability to learn versatile representations of complex data. However, their widespread application in mental state decoding is hindered by their lack of interpretability, difficulties in applying them to small datasets, and in ensuring their reproducibility and robustness. We recommend approaching these challenges by leveraging recent advances in explainable artificial intelligence (XAI) and transfer learning, and also provide recommendations on how to improve the reproducibility and robustness of DL models in mental state decoding. an area of artificial intelligence research which aims to enable computers to derive meaningful information from the visual world and to take actions based on that information. a class of representation learning methods which transform the input data in multiple sequential steps (or layers), each applying stacks of simple, but nonlinear, functions. a class of methods which aim to make the behavior of DL models understandable to human observers, for example, by relating the features of the input data to the respective outputs of the model. measures brain activity by detecting changes in activity associated with changes in local blood flow. an unobservable construct of psychological theory that refers to a particular mental operation or content and is often associated with specific observable behaviors. an area of artificial intelligence research which aims to enable computers to derive meaningful information from human language and to take actions based on that information. as used in computer science, a transform of some data in terms of a different set of features. Note that this definition stands in contrast to the understanding of representations in cognitive neuroscience, where they indicate the set of mental states that are encoded in (or represented by) the patterns of neural activity of a brain region (or network).