Inducing Human-like Biases in Moral Reasoning Language Models

Authors: austin  meek
artem  karpov
independent  seong
hah  cho
lucy  farnik
bogdan-ionut  cirstea
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 11:56:02

Abstract: In this work, we study the alignment (BrainScore) of large language models (LLMs)
fine-tuned for moral reasoning on behavioral data and/or brain data of humans
performing the same task. We also explore if fine-tuning several LLMs on the
fMRI data of humans performing moral reasoning can improve the BrainScore.
We fine-tune several LLMs (BERT, RoBERTa, DeBERTa) on moral reasoning
behavioral data from the ETHICS benchmark Hendrycks et al. [2020], on the
moral reasoning fMRI data from Koster-Hale et al. [2013], or on both. We study
both the accuracy on the ETHICS benchmark and the BrainScores between model
activations and fMRI data. While larger models generally performed better on both
metrics, BrainScores did not significantly improve after fine-tuning.

Main paper: 
 introduction: 
 Recently, multiple papers have shown surprising similarities between the internal representations in biological brains and those in artificial neural networks, in multiple domains and for multiple tasks; see e.g. Huh et al. [2024] for a review and for some potential theoretical explanations of why one might expect this, including with increasingly powerful machine learning models. However, to the best of our knowledge, no previous work has analyzed potential analogous representational similarities in the domain of moral reasoning, nor whether the degree of similarity might be increased by using human neural data (e.g. fMRI).In this work, we undertake the first such measurement (BrainScore) of the similarity of the internal representations of biological brains (measured through fMRI) and large language models (LLMs), in the domain of moral reasoning (in a task which is also partially relevant to Theory of Mind). We also study whether fine-tuning the LLMs on a train set of the corresponding neural data helps with improving the BrainScore on a separate test set. While our attempts here proved unsuccessful, we think this is an important problem to study and that increasing said alignment might be important for the AI alignment problem Christian [2020] .We next discuss related works in section 2, our methodology in section 3, the results we obtained in section 4, and finally in section 5 conclude and discuss some potential future work. 
related works: 
 There is a growing interest in brain-model alignment work, here we only provide a brief overview. For a more detailed survey, see Sucholutsky et al. [2023] , especially section 4. 3.2, and Schrimpf et al. [2020] for a systematic approach to collecting and scoring many models. Earlier work on fine-tuning transformers to predict fMRI data has found that adding MEG data also helps Schwartz et al. [2019] . Other work Aw and Toneva [2023] focused on fine-tuning models on the much larger Booksum dataset Kryściński et al. [2022] , which they found increased alignment. Dapello et al. [2022] used rhesus macaque neural data and showed improved alignment with human neural data and greater adversarial robustness.To the best of our knowledge, we are the first to attempt increasing brain-model alignment on moral reasoning neuroimaging data. 
benchmark and dataset: 
 To quantitatively measure the moral reasoning performance of the fine-tuned LLMs, we used the common sense category of the ETHICS benchmark Hendrycks et al. [2020] , which consists of multiple choice questions rather than free form responses. To predict an answer, we use a linear transformation layer (a CLS head or just a head) attached to predictions (logits) for a classification token, "[CLS]", of a base model. We used the fMRI dataset, 'Moral judgments of intentional and accidental moral violations across Harm and Purity domains', from Koster-Hale et al. [2013] . Human subjects were given a series of scenarios describing moral, immoral, and neutral actions across a wide variety of scenarios, and then answered on a 1-4 scale how moral or immoral each action was. Koster-Hale et al. [2013] was approved by an IRB and subjects were paid and gave written, informed consent.We used a pre-processed version of the dataset from Thomas et al. [2023] , specifically the version fit with the DiFuMo atlas Dadi et al. [2020] with a dimensionality of 1,024 regions of interest (the maximum number of dimensions DiFuMo provides).Because of the high granularity of our chosen atlas, we used NeuroSynth Yarkoni et al. [2011] , a tool for meta-analysis conducted over thousands of fMRI studies to isolate regions consistently activated during experiments, to map activations to specific themes. We conducted our analyses on four regions related to Theory of Mind, moral reasoning, language, and vision. We used vision as the control group as we expected scores there not to increase (see Appendix A). We visualized the relationship between the fMRI and LLM activations on the cortical surface (see Appendix A) using the Coefficient of Determination (CoD). The CoD scores were then negative log transformed and the weighted average of the parcel scores were plotted at each vertex, since the DiFuMo atlas is probabilistic with overlapping boundaries.We focused on encoder models (BERT-based) due to computational constraints and since this was a classification task. Additionally, encoder models originally showed better results on the ETHICS dataset Hendrycks et al. [2020] . Overall we used four models, BERT-base-cased and BERT-large cased (108 and 333 million parameters) Devlin et al. [2019] , RoBERTa-large (355 million parameters) Liu et al. [2019] , and DeBERTa-v2-xlarge (864 million parameters) He et al. [2021] .For fine-tuning, we used the HuggingFace library Wolf et al. [2019] to train additional heads of dimensionality 1,024 (to match the DiFuMo atlas) on top of the classification token, "[CLS]". We also train heads to predict the ETHICS benchmark Hendrycks et al. [2020] . We report fine-tuning on ETHICS only and with the addition of fMRI data in Table 1 . In total, we ran 450 fine-tuning runs, totaling 292 hours of training for 1,082 different models (not all shown in the results section). Our 'brain-score' metric is based on similar metrics found within the broader NeuroAI literature, such as in Schrimpf et al. [2020] , Aw and Toneva [2023] , inter alia. We use the Pearson's correlation coefficient (PCC) to measure the correlation between predicted brain activity and actual brain activity. For some moral scenario given to a subject, we sample the fMRI data at several time points, taking the hemodynamic lag into account. This data has been fit to the DiFuMo atlas, resulting in 1,024 ROIs at the time points sampled. We do this over a collection of similar examples, and then fit a regression model to predict this brain activity. The PCC between the predicted response and the actual held out brain activity gives us the brain-score metric. We can also do this on a layer-by-layer basis and aggregrate over all layers to provide a single brain-score for the whole model, which we provide in Table 2 . 
results: 
 Our results indicate that improving brain-model alignment on moral reasoning by fine-tuning on relevant fMRI data does not consistently improve accuracy on ETHICS Hendrycks et al. [2020] . While our fine-tuning procedures do improve accuracy on the Commonsense split of ETHICS (bolded values in Table 1 are higher than those reported by Hendrycks et al. [2020] ), we could not improve accuracy by fine-tuning on the fMRI data only or on a combination of fMRI and ETHICS, compared to fine-tuning purely on ETHICS.To thoroughly test these results, we also used a variety of sampling methods to pull from the fMRI data, as shown in Table 3 and Figure 2 in Appendix B. AVG indicates an average of all time points, LAST indicates the time point at the hemodynamic lag before the last time point, MIDDLE indicates the middle point, and SENTENCES indicates four points in a scenario, which match the end of the four sentences read by the subject. We find that LAST tends to produce the best accuracy.We generally find that larger models are more performant overall, a finding also reported by Hendrycks et al. [2020] . This additionally holds with the brain-score metric, which measures brain-model alignment. However, we were also unable to significantly improve our brain-score metric beyond the pre-trained models, as shown in Table 2 . This finding is also consistent in layer-wise scores across each of the three models; see Appendix A for further brain-score details (including region specific information, such as Theory of Mind ROIs) and cortical mappings. 
conclusion and future work: 
 While we were unable to significantly increase brain alignment on moral reasoning through finetuning methods, we do believe that our results can be of use for downstream work. Firstly, we believe that our work is ample evidence for the importance of gathering more data on moral reasoning and for more niche tasks in general if future researchers want to increase brain-model alignment See Table 1 for a tabular depiction of the data. 2 : Brain scores across models and different fine-tuning methods. We were unable to significantly increase brain-model correlation using any of the fine-tuning methods. in specific domains. Secondly, we make our code available. 3 We believe that further work along the neuroconnectionist research agenda Doerig et al. [2023] will be useful generally, and hope that the preliminary evidence we provide here will help update others' research models on the ability to increase alignment in specific domains.  
 
 Appendix: Code available at: https://github.com/ajmeek/Inducing-human-like-biases-in-moral-reasoning-LLMs 


Decision: In light of the reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Review for: Inducing Human-like Biases in Moral Reasoning Language Models
main_review: main_review: This paper studies the alignment of BERT representations on moral reasoning tasks with corresponding fMRI data recording the brain activations of humans passing moral judgements. The results seem to indicate that for the task of moral reasoning, aligning the representations of the classifier to brain activity does not help to improve the performance on the moral reasoning dataset.

The paper is well written and I agree that the research is interesting and worthwhile pursuing. To make it accessible to a wider audience, it would probably help to combine Sections 1 and 2 and to better motivate and explain some of the claims, e.g. how aligning representations for moral reasoning tasks would be relevant for ToM or why it would be relevant for AI alignment.

I also think the finding that "improving brain-model alignment on moral reasoning by fine-tuning on relevant fMRI data does not consistently improve accuracy on [the textual moral judgement dataset]" might be due to the fact that according to the measured BrainScores none of the training/fine-tuning actually improved the brain-model alignment, according to Table 2 and Figure 3.

Being outside the field, I cannot pass judgement whether the observed "unlearnability" and/or failure to align the representation are inherently due to the difference in tasks/modalities/representations or due to the chosen method of representing the fMRI data in such a way that it is learnable by BERT.

Nonetheless I think this work in progress deserves to be presented as an extended abstract if there is space to do so in this workshop.
title: title: Offical Review
main_review: main_review: **Summary**

This paper investigates how fine-tuning large language models (LLMs) for moral reasoning tasks can improve their alignment with human brain activity (BrainScore).

**Strengths**

This study is the first to attempt aligning LLMs with human brain activity during moral reasoning by using fMRI data. This cross-disciplinary research offers a new approach for addressing the AI alignment problem, particularly in complex cognitive tasks.

**Weaknesses**

Although the models showed improved performance on the ETHICS benchmark, BrainScore did not see significant gains. Whether fine-tuning with only fMRI data or combining ETHICS and fMRI data, there was little improvement in the alignment between the model and human brain activity.
title: title: Paper proves that current fMRI data is not useful for aligning LLM models in the moral domain.
main_review: main_review: The reviewer is not familiar with neuroscience literature and thus trusts that the authors are using the best data available for such a task. Given the data, the setting seems reasonable, and the experiments are thorough. 

The expectation that any representation of neural data would be functionally useful to steer LLM representations in a desired direction seems a little absurd. While the authors successfully show that such beliefs are unfounded, having such an expectation is questionable to begin with.
title: title: Review of Inducing Human-like Biases in Moral Reasoning Language Models
main_review: main_review: Quality:
The paper focuses on alignment of large language models fine-tuned for moral reasoning on behavioral data and/or brain data of humans performing the same task. It conducts solid experiments, and gives analysis of the use of fMRI data to fine-tune large language models (LLMs) for moral reasoning.

Clarity:
The paper is generally well-structured, with clear sections and explanations. The authors provide justifications for their answers to ethical considerations, ensuring transparency in their research process. Figures help readers to compare the results of experiments easily.

Originality:
The study's approach of aligning LLMs with human brain data using fMRI is original and contributes to the emerging field of ethical AI and natural language understanding.

Significance:
The significance of this research is less pronounced since authors are unable to significantly increase brain alignment on moral reasoning through fine-tuning methods. While they suggest their results could benefit future downstream work, this claim is unsupported. Additionally, their assertion that the study highlights the need for more data on moral reasoning and niche tasks feels somewhat self-evident and offers little in the way of new insights.

Pros:

Comprehensive and diverse experiments into brain alignment on moral reasoning.

Undertake the first measurement (BrainScore) of the similarity of the internal representations of biological brains (measured through fMRI) and large language models (LLMs).

Cons:

Experiment result does not bring consistent improve of accuracy on ETHICS.

Need more analysis into how the insights gained from the experiments could be applied to future study or real world applications.
Some typos, for example line 101 misses "Table" in "Table 2"