## 5 Experimental Analysis
### Experimental Setup  
Using the synthetic data generator described in Section 4.2, we generated synthetic datasets consisting of 500 patients and also calculated the corresponding counterfactual outcomes of those BCPs. Notice that for calculating the counterfactual outcome of each sample, we re-calculated all of its variable values by assigning a change in the intensity of their physical activity, which is the treatment variable, while keeping the same value of the noise variables. In detail, we changed from high-intensity exercise to low-intensity exercise and vice versa, depending on the initial assignment of each individual. The ground truth effect of high-intensity exercise on the BMD loss of BCPs is calculated as 0.58, i.e., intense physical activity inhibits the loss of BMD, as has been underlined in the literature [61; 62; 63].  
We direct the reader to Appendix B for a detailed description of the implemented functional relationships of the variables as well as to [[https://github.com/kiriakidou/A-Causal-Inference-Methodology-to-Support-Research-on-Osteopenia-for-Breast-Cancer-Patients](https://github.com/kiriakidou/A-Causal-Inference-Methodology-to-Support-Research-on-Osteopenia-for-Breast-Cancer-Patients)]([https://github.com/kiriakidou/A-Causal-Inference-Methodology-to-Support-Research-on-Osteopenia-for-Breast-Cancer-Patients](https://github.com/kiriakidou/A-Causal-Inference-Methodology-to-Support-Research-on-Osteopenia-for-Breast-Cancer-Patients)) (accessed on 1 April 2021), for the generated datasets. At this point, it is worth mentioning that since no RWD are available, the utilized noises in our research were defined using input from domain experts in the REBECCA project.  
For each generated dataset, we consider two cases: (a) a non-imputed dataset in which no missing values exists and (b) an imputed dataset in which 30% of entries from the \"estrogens\" and \"calcitonin\" variables were randomly removed for simulating the case of the missing data problem, commonly encountered in practice when working with RWD. Lacking any additional evidence, we did not make any assumptions about the mechanisms underlying missingness in the generated data, so the data are Missing Completely at Random (MCAR) [64], i.e., the likelihood of a data point being missing is independent of both observed and unobserved data. This approach is commonly used when evaluating ML models with missing data [65].  
As regards data imputation, we experimented with several imputation techniques such as utilization of the mean value for each feature, \(k\)-NN for various values of parameter \(k\) and Random Forest; nevertheless, Extra-Trees was selected, since it provided the best overall performance for all causal inference models. In addition, the employment of any imputation technique does not affect the obtained findings in our experiments since similar conclusions could be made from the employment of any of the utilized imputation techniques (\"Data Curation\" stage of the proposed workflow). Finally, we highlight that 80% of the data were used for training the models (in-sample), while the rest 20% was used as a hold-out for evaluation (out-of-sample) [66].  
In the following text, we design and perform experiments considering four different case studies for highlighting the complications that occur when using RWD for causal inference.