# Modelling the effect of courses over soft skills: A simulation study

Luis Alberto Pinos Ullauri, A. Fleury, Wim van den Noortgate, Alexis Lebis, Mathieu Vermeulen, Abir-Beatrice Karami

► To cite this version:

Luis Alberto Pinos Ullauri, A. Fleury, Wim van den Noortgate, Alexis Lebis, Mathieu Vermeulen, et al.. Modelling the effect of courses over soft skills: A simulation study. 2023 AERA Annual Meeting, American Educational Research Association, Apr 2023, Chicago, United States. 10.3102/2008368 hal-04202625

HAL Id: hal-04202625

https://hal.science/hal-04202625v1

Submitted on 11 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Modelling the effect of courses over soft skills: A simulation study

Pinos Ullauri, Luis Alberto<sup>a,b,c</sup>, Fleury, Anthony<sup>a</sup>, Van Den Noortgate, Wim<sup>b,c</sup>, Lebis, Alexis<sup>a</sup>, Vermeulen, Mathieu<sup>a</sup>, Karami, Abir<sup>d</sup>

<sup>a</sup>CERI SN, IMT Nord Europe, France

<sup>b</sup>Faculty of Psychological and Educational Sciences, Belgium

<sup>c</sup>imec ITEC KU Leuven, Belgium

<sup>d</sup>Université Catholique de Lille, France

## Abstract

Over the last few years, the educational interest of soft skills has steadily increased. Nonetheless, there is no considerable literature concerning the impact of followed courses on them. This simulation explores the modelling of the effects of courses over the students' soft skill proficiency, considering interaction effects and confounding variables through nine different scenarios. Moreover, the study compares the model with and without propensity scores as predictors, which are proposed as alternatives to handle the selection bias issue of the students' course choices. The simulation results show that in general models without propensity scores produce less bias, nonetheless the higher the effect of interactions between courses the better propensity scores become in reducing the bias.

Keywords: course effect, modelling, soft skills, propensity score, selection bias

## 1. Introduction

In the last decade, soft skills have gathered attention in both industry and academia. Soft skills is an evolving term that is difficult to define in a cohesive and clear perspective. Nonetheless, its general conception, as pointed out by Almonte (2021), consists of a group of inter- and intrapersonal skills that can help manage situations that are normally faced in real-life. Furthermore, Jardim et al. (2022) argues the soft skills definition becomes closely related with that of transferable skills in the sense that these skills could be applied in various situations.

Higher Education (HE) has since faced the challenge of including soft skills in their pedagogical design to further improve the students' professional formation (Almeida and Morais, 2021). Part of this challenge is the evaluation of soft skills throughout academic programs, which would allow to track their development across time and study their relation with followed courses. This would allow the estimation of course effects over soft skills and analyse whether it is
appropriate to adjust the curriculum (e.g. remove or adapt courses with soft skills centred pedagogical activities) if it does not optimally foster soft skills.

The students' academic records, specifically their previously followed courses, are essential to model the relation between courses and soft skills. Considering that most HE programs allow students a pseudo-free choice of courses, a naive approach can be to utilise a linear mixed model to explain the soft skills proficiency with courses as predictors. Furthermore, interaction effects between courses (e.g. courses with similar soft skill centred pedagogical design) can also be considered as well as additional covariates in order to account for possible confounding (e.g. interest for specific courses).

Yet, there can be many potential confounding variables, hampering causal inference. A randomised controlled trial (RCT) is the most popular method to infer causal effects over treatments (Austin, 2011; Williamson and Forbes, 2014; Elze et al., 2017). In RCTs, randomisation is a core part of the experiment design, where subjects are randomly placed in either control or treatment groups. This type of randomisation eliminates the bias and confounding variables from the subjects. Nonetheless, these types of studies bear a high cost in terms of planning and logistics, which is why natural experiments (NE) are increasingly gathering attention. NEs in contrast to RCTs are less controlled and occur "naturally", where subjects choose either the control or treatment groups (e.g. students choosing their courses instead of being placed). This change may largely reduce the cost of the studies, but it also brings up issues such as selection bias (Craig et al., 2017).

In the naive approach, the course effect estimates can be severely affected by selection bias. For instance, students may choose to follow a particular course because they already have some expertise or are quite interested. This means that students who follow a course may not necessarily be comparable to those not following that course. Moreover, selection bias can also present itself as sampling bias (Deschacht and Goeman, 2015). For example, certain courses may be more followed or have more students than other courses, producing a biased sample that may not be representative.

A possible strategy to handle selection bias is to use propensity scores. Propensity scores allow NEs to resemble RCTs by taking into account selection bias (Lecocq et al., 2014), and they represent in this case the probabilities of students choosing a specific course. For instance, if the profiles (e.g. previously followed courses) of student A and B are similar, the probabilities of A and B choosing a course C may also be similar, whereas if their profiles differ considerably, the propensity of A choosing C may vary from B. Furthermore, propensity scores can be included as additional predictors.

The general aim of this study is to explore how propensity scores could mitigate course selection bias. To the best of our knowledge, there are no publicly available datasets with the required information. Therefore, data is generated to analyse this problem in different scenarios. As a proof of concept and in favour of simplicity, a single soft skill is modelled instead of multiple skills.
## 2. Methodology

### 2.1. Data Generation

To further illustrate this issue, let us present a hypothetical example of an academic program of two years (e.g Master program) defining three stages (e.g. academic terms). Stage 0 at the start, Stage 1 (end of year 1) and Stage 2 (end of the program). Students are assumed to have freedom of choosing their courses and there is no course failure. Table 1 shows the number of courses per stage. Please note that not all offered courses would be chosen, hence the students' course-backgrounds would not necessarily be the same.

<table><caption>Table 1: Number of courses per stage</caption><thead><tr><td>Stage</td><td>Followed courses</td><td>Number of courses to follow</td><td>Offered courses</td></tr></thead><tbody><tr><td>Initial (0)</td><td>0</td><td>3</td><td>8</td></tr><tr><td>Middle (1)</td><td>3</td><td>2</td><td>5</td></tr><tr><td>Final (2)</td><td>5</td><td>0</td><td>0</td></tr></tbody></table>

A total of 800 students are considered. The courses are assumed to be grouped by pedagogical similarities in three groups. $G_1$ is composed by courses $(c_1, c_2, c_3)$, $G_2$ by $(c_4, c_5)$, and $G_3$ by the last three $(c_6, c_7, c_8)$.

Moreover, different scenarios are simulated on the combinations of two conditions. The first is the modelling of interaction effects. This means that if a student follows $c_1$ and $c_2$, which are assumed to be related, the students' soft skills may be affected not only from their individual effects, but also by the interaction of both courses. The interaction effects are included in the model as fixed effects (in three categories: no effect, small or large effect).

The second is the interest for specific courses as a confounding variable. An intuitive assumption is that interested students are more likely to better develop skills compared to students with little or no interest. For instance, interest may increase the time that students work on the course or it may trigger other related activities outside the course.

Furthermore, this interest can not only affect skill growth, but also influence the student's choice of courses. The interest $I$ is sampled from a normal distribution with zero mean. This interest would influence the probability of course selection from groups $G_1$ and $G_3$, while maintaining the same probability for $G_2$. This mitigates sampling bias issues, as all courses would approximately be followed by the same amount of students at every stage of the program. Figure 1 shows an example of how the probability of choosing courses from $G_1$ or $G_3$ depend on this interest. Similarly to the interactions, three levels of interest effect are simulated (no effect, small or large effect).

A total of 9 scenarios are presented in Table 2 based on the combinations of the conditions and their levels.

### 2.2. Skill generation model

The skill generation model is a generalised linear mixed model, with fixed course effects and random student effects as shown in Equation 1. $\beta_0$ would
<figure><img src="image_2.png" /><figcaption>Figure 1: Interest distribution and its confounding nature towards course selection. This example shows how the student’s interest for specific courses affects their probability of choosing courses from G₁ or G₃. In the left plot, there is less probability for G₁. In the middle, the probability is equal for both groups. In the last plot, courses from G₁ have more probability of being chosen</figcaption></figure>

<table><caption>Table 2: Scenario by interest for specific courses and interaction condition combinations</caption><thead><tr><th>Condition</th><th>No Interaction</th><th>S. Effect Interaction</th><th>L. Effect Interaction</th></tr></thead><tbody><tr><td>No Interest</td><td>Scenario 1</td><td>Scenario 2</td><td>Scenario 3</td></tr><tr><td>S. Effect Interest</td><td>Scenario 4</td><td>Scenario 5</td><td>Scenario 6</td></tr><tr><td>L. Effect Interest</td><td>Scenario 7</td><td>Scenario 8</td><td>Scenario 9</td></tr></tbody></table>

correspond to the initial skill level of an average student at Stage 0 (where the students have not yet followed any courses). $\beta_c$ represents the individual fixed course effects. $D_{(c,s,t)}$ is a dummy binary variable that is equal to 1 if student s followed course c at Stage t. X represents the sum of all double and triple interaction effects of courses based on the groups. $\beta_i$ represents the effect of interest for specific courses. $\theta_s$ is a random effect of student s, and $\epsilon$ a residual term.

$$ \text{Skill}_{(c,s,t)} = \beta_0 + \sum_{c=1}^{C} \beta_c D_{(c,s,t)} + X + \beta_i I_{(s,t)} + \theta_s + \epsilon \quad (1) $$

Moreover, the distributions and effects of parameters used for the data generation are presented in Table 3.

All possible combinations are generated per stage and student (56 course sets
<table><caption>Table 3: Parameter distributions and values</caption><thead><tr><th>Parameter</th><th>Distribution</th><th>Effect</th></tr></thead><tbody><tr><td>β<sub>0</sub></td><td>-</td><td>1</td></tr><tr><td>β<sub>1</sub></td><td>-</td><td>0.40</td></tr><tr><td>β<sub>2</sub></td><td>-</td><td>0.45</td></tr><tr><td>β<sub>3</sub></td><td>-</td><td>0.50</td></tr><tr><td>β<sub>4</sub></td><td>-</td><td>0.44</td></tr><tr><td>β<sub>5</sub></td><td>-</td><td>0.37</td></tr><tr><td>β<sub>6</sub></td><td>-</td><td>0.63</td></tr><tr><td>β<sub>7</sub></td><td>-</td><td>0.33</td></tr><tr><td>β<sub>8</sub></td><td>-</td><td>0.11</td></tr><tr><td>S. Effect β<sub>xy</sub></td><td>-</td><td>0.025</td></tr><tr><td>L. Effect β<sub>xy</sub></td><td>-</td><td>0.05</td></tr><tr><td>S. Effect β<sub>xyz</sub></td><td>-</td><td>0.05</td></tr><tr><td>L. Effect β<sub>xyz</sub></td><td>-</td><td>0.1</td></tr><tr><td>No Effect I</td><td>-</td><td>0</td></tr><tr><td>S. Effect β<sub>i</sub></td><td>-</td><td>1</td></tr><tr><td>L. Effect I β<sub>i</sub></td><td>-</td><td>3</td></tr><tr><td>I<sub>st</sub></td><td>N(μ = 0, σ = 0.3)</td><td>-</td></tr><tr><td>θ<sub>s</sub></td><td>N(μ = 0, σ = 0.25)</td><td>-</td></tr><tr><td>ε</td><td>N(μ = 0, σ = 0.2)</td><td>-</td></tr></tbody></table>

at stage 1 and 10 course sets at stage 2 considering the previous choices at stage 1). For the scenarios with no interest for specific courses, the course sets are chosen randomly. On the other hand, when interest is present, the course sets are chosen through probabilities as previously shown in Figure 1. Consequently, the students' soft skills are generated following Equation 1. 2000 datasets (each with a structure as presented in Table 4) are generated per scenario (18000 in total).

<table><caption>Table 4: Dataset structure</caption><thead><tr><th>Student</th><th>Stage</th><th>Course 1</th><th>Course 8</th><th>Soft Skill</th></tr></thead><tbody><tr><td>1</td><td>0</td><td>0</td><td>0</td><td>1.21</td></tr><tr><td>1</td><td>1</td><td>0</td><td>0</td><td>2.74</td></tr><tr><td>1</td><td>2</td><td>0</td><td>1</td><td>4.33</td></tr><tr><td>..</td><td>..</td><td>..</td><td>..</td><td>..</td></tr><tr><td>800</td><td>1</td><td>1</td><td>1</td><td>2.65</td></tr><tr><td>800</td><td>2</td><td>1</td><td>0</td><td>3.49</td></tr></tbody></table>

### 3. Analysis

The generated data is analysed without interest given that it may prove difficult to estimate in real life and/or model its confounding relation. Models with and without propensity score covariates are fit in order to check whether the addition of propensity score predictors may help or not reduce bias in course
effects. The models are implemented with the lme4 package from R (Bates et al., 2015), which uses maximum likelihood methods.

### 3.1. Bias estimation

The course effect bias estimates are calculated by the relative difference between the model estimates and the real effects. This means that for each dataset there are 16 bias estimates (for each of the 8 courses, the bias is estimated using the model with and without propensity scores).

## 4. Results

The percentage of datasets, where the propensity score predictors help reduce the bias in course effects, is calculated by scenario. Moreover, in favour of simplicity, the percentages are averaged and aggregated by groups as shown in Table 5. It can be seen that in general the model without propensity scores is preferred across the scenarios. Nonetheless, the higher the effect of interactions (scenarios 2,5,8 and 3,6,9), the propensity models' preference in terms of bias reductions increases.

<table><caption>Table 5: The percentages in parentheses correspond to the relative amount of datasets benefitting from propensity scores in terms of bias reduction. The percentage below is the complement, where the models without the propensity scores produced less bias</caption><thead><tr><th></th><th>Group 1</th><th>Group 2</th><th>Group 3</th></tr></thead><tbody><tr><td>Scenario 1</td><td>(26.73%)<br>73.27%</td><td>(27.25%)<br>72.75%</td><td>(26.60%)<br>73.40%</td></tr><tr><td>Scenario 2</td><td>(37.25%)<br>62.75%</td><td>(32.33%)<br>67.68%</td><td>(37.85%)<br>62.15%</td></tr><tr><td>Scenario 3</td><td>(37.63%)<br>62.37%</td><td>(33.75%)<br>66.25%</td><td>(37.00%)<br>63.00%</td></tr><tr><td>Scenario 4</td><td>(25.20%)<br>74.80%</td><td>(24.25%)<br>75.75%</td><td>(25.12%)<br>74.88%</td></tr><tr><td>Scenario 5</td><td>(36.42%)<br>63.58%</td><td>(31.20%)<br>68.80%</td><td>(36.98%)<br>63.02%</td></tr><tr><td>Scenario 6</td><td>(36.77%)<br>63.23%</td><td>(30.03%)<br>69.98%</td><td>(36.51%)<br>63.48%</td></tr><tr><td>Scenario 7</td><td>(22.99%)<br>77.02%</td><td>(23.23%)<br>76.78%</td><td>(24.20%)<br>75.80%</td></tr><tr><td>Scenario 8</td><td>(36.43%)<br>63.57%</td><td>(29.38%)<br>70.63%</td><td>(37.00%)<br>63.00%</td></tr><tr><td>Scenario 9</td><td>(36.08%)<br>63.92%</td><td>(30.45%)<br>69.55%</td><td>(35.17%)<br>64.83%</td></tr></tbody></table>

## 5. Discussion

Even though the results show that models without propensity scores are favoured, it is important to remark that the higher effect of interaction terms,
the more preferable the propensity score models are. This may be due to the amount of fixed predictors to estimate. The interactions provide 9 fixed coefficients having a total of 18 coefficients without the propensity scores (the intercept, 8 course effects and 9 interactions). The propensity scores add 8 more coefficients in the regression (26 coefficients in total), and despite that fact, between 23% and 38% of the datasets in all scenarios, the propensity scores help reduce the bias of course effects estimates.

Nonetheless, there are several limitations to consider. First, the simulation design considers a single interest effect for specific courses instead of an interest effect per course. Second, the models are unidimensional with only one soft skill as the outcome. Third, the data generation process can be time-consuming and computationally expensive (the number of courses was selected to keep the number of possible combinations relatively low), though with parallelisation the generation time can be considerably reduced. Furthermore, in a real-life setting, the amount of students per stage may be smaller, the number of courses could be larger, the number of students per course may greatly differ producing unbalanced samples, there may also be constraints on the courses (pre-requisites or room capacity) making this problem more complicated.

# 6. Conclusions and future work

Some alternatives for future work would be to consider:

* The addition of interest effects per course.

* A multidimensional extension of the models by considering various soft skills where each course may affect one or more soft skills.

* The use of propensity scores as weights rather than predictors.

* The case of students who repeat their courses and consider it as an additional effect.

The simulation study sheds a light on the possible issues when modelling course effects over soft skills, and attempts to explore the course selection bias induced by the students' interests for specific courses. Propensity scores as predictors may reduce course selection bias, particularly when there are high effects of interactions between courses. We hope this article inspires researchers to further explore and analyse this issue.

# References

Almeida, F., Morais, J., 2021. Strategies for developing soft skills among higher engineering courses. Journal of Education 0. URL: https://doi.org/10.1177/00220574211016417.
Almonte, R., 2021. A Practical Guide to Soft Skills: Communication, Psychology, and Ethics for Your Professional Life. Routledge, Taylor & Francis Group. URL: https://doi.org/10.4324/9781003212942.

Austin, P., 2011. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate behavioral research 46, 399–424. doi:10.1080/00273171.2011.568786.

Bates, D., Mächler, M., Bolker, B., Walker, S., 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software 67, 1–48. doi:10.18637/jss.v067.i01.

Craig, P., Katikireddi, S., Leyland, A., Popham, F., 2017. Natural experiments: An overview of methods, approaches, and contributions to public health intervention research. Annual Review of Public Health 38. doi:10.1146/annurev-publhealth-031816-044327.

Deschacht, N., Goeman, K., 2015. Selection bias in educational issues and the use of heckman’s sample selection model, Leuven University Press, pp. 35–51.

Elze, M.C., Gregson, J., Baber, U., Williamson, E., Sartori, S., Mehran, R., Nichols, M., Stone, G.W., Pocock, S.J., 2017. Comparison of propensity score methods and covariate adjustment. Journal of the American College of Cardiology 69, 345–357. URL: https://www.jacc.org/doi/abs/10.1016/j.jacc.2016.10.060.

Jardim, J., Pereira, A., Vagos, P., Direito, I., Galinha, S., 2022. The soft skills inventory: Developmental procedures and psychometric analysis. Psychological Reports 125, 620–648. URL: https://doi.org/10.1177/0033294120979933.

Lecocq, A., Ammi, M., Bellarbre, E., 2014. Le score de propension : un guide méthodologique pour les recherches expérimentales et quasi expérimentales en éducation. Mesure et évaluation en éducation 37, 69–100. doi:https://doi.org/10.7202/1035914ar.

Williamson, E., Forbes, A., 2014. Introduction to propensity scores. Respirology 19. doi:10.1111 resp.12312.