# Multi-Agent LLMs for Personalizing STEM Texts
## 4 Results  
All analyses were performed using IBM SPSS Statistics v. 29. All participants completed the study activities; thus, the dataset contains no missing or incomplete entries. A choice score was calculated for each participant based on data collected during the test session. This score was equal to the number of times a participant selected the personalized paragraph over the generic text. Recall that participants in the experimental group were expected to select the personalized texts, while those in the control group were expected to select the generic texts. Because the test session consisted of two paragraph pairs, each participant's final choice score could be either 0, 1, or 2. Table 3 summarizes the results of the test session for the experimental and control groups. Figure 4: Profile evaluation component of this study. True and opposite profiles generated for one participant by GPT-4 are shown. Figure 3: Screenshot of graphical user interface (GUI). One paragraph pair used in the training session is shown (Topic: water cycle). Given the count nature of the dependent variable, we considered the use of a negative binomial regression model. However, we find that our data are not over-dispersed (grand mean and variance are 1.0 and 0.545, respectively), which is a core assumption of a negative binomial regression. In addition, we recognize that our dependent variable is strictly bound to integers between 0 and 2, which precludes the use of count models that assume an unbounded range of non-negative integers. With this in mind, ordinal logistic regression was selected to model the effect of group membership (i.e., experimental or control) on the ability of the two-agent GPT-4 MAS to identify and adapt text to the participant's learning preferences. In this model, the dependent variable is treated as an ordinal variable. The test of parallel lines was used to check the assumption of proportional odds (\(\chi^{2}(1)=1.193\), \(p=0.275>0.05\)). Since \(p>0.05\), we fail to reject the null hypothesis assumption of proportional odds. Other assumptions, such as no multicollinearity between independent variables and the independence of observations, are satisfied by the study design (i.e., only one independent variable--study group--is tested, and all participants are only observed once). Thus, an ordinal logistic regression model was determined to be appropriate for these data. The final model trended towards significance at the 0.05-level when compared to the intercept-only model (\(\chi^{2}(1)=3.183\), \(p=0.074\)). Students in the experimental group (i.e., students who had texts personalized to their preferences) were estimated to be 4.373 (95% CI: (0.815, 23.454)) times more likely (Wald \(\chi^{2}(1)=2.965\), \(p=0.085\)) than students in the control group to prefer the personalized text. This odds ratio trended towards significance at the 0.05-level, suggesting that participants in the experimental group may have been more likely to select the personalized paragraphs during the test session. We obtain a pseudo-R\({}^{2}\) (Nagelkerke) of 0.147, meaning that our model explains approximately 14.7% of the variance observed in our dependent variable. Finally, we note that the confidence interval of the odds ratio is quite wide due to the limited sample size and exploratory nature of the study, so we suggest that the results of this pilot RCT be interpreted with extreme caution. While we find the results to be suggestive of a potential relationship within the context of this feasibility study, we caution readers against generalizing beyond this work. Of the 23 participants enrolled in this study, 14 selected their true profile in the profile evaluation phase (Table 4). This value is not significantly different (\(p=0.405\)) from the expected value of the binomial distribution, suggesting that, on average, participants performed no better than random when tasked with selecting their true profile.