# Exploring Performance of Configurable Software Systems: the JHipster Case Study

Édouard Guégain, Alexandre Bonvoisin, Mathieu Acher, Clément Quinton, Romain Rouvoy

► To cite this version:

Édouard Guégain, Alexandre Bonvoisin, Mathieu Acher, Clément Quinton, Romain Rouvoy. Exploring Performance of Configurable Software Systems: the JHipster Case Study. EASE'25 - 29th International Conference on Evaluation and Assessment in Software Engineering, Jun 2025, Istanbul, Turkey. pp.1-10. [hal-05003699v2](https://hal.science/hal-05003699v2)

HAL Id: hal-05003699

https://hal.science/hal-05003699v2

Submitted on 11 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Exploring Performance of Configurable Software Systems:
the JHipster Case Study

Edouard Guégain

Alexandre Bonvoisin

Univ. Lille, CNRS, Inria, Centrale Lille,
UMR 9189 CRISTAL, F-59000, Lille

France

Mathieu Acher

Univ Rennes, Inria, CNRS,
IUF, IRISA, Rennes

France

Clément Quinton

Romain Rouvoy

Univ. Lille, CNRS, Inria, Centrale Lille,
UMR 9189 CRISTAL, F-59000, Lille

France

## ABSTRACT

The performance of software systems remains a key concern in software engineering. Configurable software systems, with their numerous configurations, complicate the performance evaluation process. This paper investigates the impact of web stack configurations on performance, using the JHipster web stack generator as a case study. We analyze JHipster configurations to understand how component choices influence system performance and explore individual configuration options for their specific effects. Our study shows that correlations across performance indicators exist but are often weak, and different options affect performance unevenly, with some impacting one indicator minimally while significantly influencing another. We developed a performance model for JHipster to automate the identification of configurations optimized for specific metrics, identifying four configurations that outperform the current default. Overall, this study underscores JHipster's relevance as a use case for studying component-level variability in software systems and highlights the importance of selecting configurations based on performance indicators rather than preferred technologies.

## CCS CONCEPTS

* Software and its engineering → Software performance; 
* General and reference → Empirical studies.

## KEYWORDS

Variability, Performance, Empirical

### ACM Reference Format:

Edouard Guégain, Alexandre Bonvoisin, Mathieu Acher, Clément Quinton, and Romain Rouvoy. 2025. Exploring Performance of Configurable Software Systems: the JHipster Case Study. In *Proceedings of The 29th International Conference on Evaluation and Assessment in Software Engineering (EASE 2025)*. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

## 1 INTRODUCTION

Modern software systems must balance strict performance demands and provide superior user experiences, such as low response latency while also minimizing CPU, RAM utilization, and energy consumption, particularly in cloud environments [3]. Consequently, developers are incentivized to adopt technologies capable of meeting

these performance goals [24]. However, identifying the most suitable technologies for achieving these objectives is challenging, as performance assessments are influenced by runtime environments, input workloads, and the exponential growth of configuration possibilities in variable systems. In addition, some performance indicators are significantly more complex to measure than others. For instance, assessing power usage requires dedicated benchmarking setups, while metrics like binary size or the number of services are easier to obtain. As a result, developers often rely on default settings or explore only a subset of configurations [7, 22, 25].

This paper investigates the impact of configuration choices on system performance, exploring metrics such as binary size, execution time, and power consumption. We aim to help stakeholders determine (i) relevant performance indicators, (ii) their interrelations, and (iii) strategies for systematically designing high-performance configurations. We conduct our study on JHipster [10, 11], an open-source platform for developing modern web applications. With over 100,000 possible configurations [20], JHipster serves as an ideal use case for studying variability in software systems. We chose JHipster as it satisfies several critical criteria, such as open-source accessibility, automation, and extensive metrics for evaluation. Widely adopted in industry, JHipster exhibits over 56,000 commits, 700+ contributors, and 21,700+ GitHub stars, ensuring that its performance has a significant real-world impact. In particular, we make the following contributions:

- We examine JHipster's relevance as a use case to study software variability at the component level.

- We analyze how JHipster's configuration options affect various performance metrics.

- We identify correlations among performance indicators, uncovering trends related to specific metrics.

- We provide a replication package with a full dataset of performance indicators to support future research on configurable software.

## 2 RELATED WORK

Configurable systems are subject to combinatorial explosion: a system offering $n$ boolean options can lead to the generation of $2^n$ configurations. Their configuration space (i.e., the set of valid configurations) thus tends to be too large for an exhaustive analysis, as it is not possible to generate and test all the configurations. Understanding such configurable systems and their related performance is challenging, and is thus an active research field.

Please use nonacm option or ACM Engage class to enable CC licenses.

This work is licensed under a Creative Commons Attribution 4.0 International License.

EASE 2025, 17–20 June, 2025, Istanbul, Türkiye

© 2025 Copyright held by the owner/author(s).

ACM ISBN 978-x-xxxx-xxxx-x/YY/MM

https://doi.org/10.1145/nnnnnnn.nnnnnnn

<figure><img src="image_2.png" /></figure>
## 2.1 Performance of configurable software

Many efforts have been put into managing the performance of configurable software. A survey by Pereira et al. reports on 69 publications in that field [25], and shows that research has mainly addressed one of the following three areas, discussed thereafter.

**Performance prediction.** This research area aim to estimate the performance of configuration without measuring them. Several performance prediction approaches have been proposed in the literature [8, 15, 28]. Such approaches aim to build performance-influence models estimating the impact of each option on a system. These approaches rely on machine learning techniques to infer performance data from a sample of configurations. One of their main objectives is to detect feature interactions—as they can have a significant impact on performances—and provide more accurate predictions than approaches that do not consider such interactions. Relying on such performance models, Siegmund et al. predict the performance of configurations as the sum of the impact of each feature on performances [29, 31]. In particular, they empirically observed that different releases of the same system have different performances [16], and that performances of configurable systems have a high level of uncertainty [5]. Arcaini et al. attempted to account for such uncertainty [1].

**Performance optimization.** Many approaches have been proposed to locate near-optimal configurations w.r.t some performance indicators. Several studies provide deterministic approaches [6, 22, 23, 36] to tackle this challenge. Other authors, like Hierons et al., rely on genetic algorithms to minimize the number of measurements needed to optimize configurations [12], or leverage the performance prediction methods discussed above [30]. Siegmund et al. assess different machine learning techniques for estimating energy consumption for different configurable software systems and acknowledge that machine learning alone is insufficient to optimize software performance [27]. Recently, Weber et al. [33] studied whether there is a correlation between runtime performance and energy consumption of software configurations. Following a white-box approach, the authors also aim to identify relevant functions to obtain an accurate proxy for energy consumption. Our work aligns with their objectives, but there are notable differences in our study. For instance, while they only focus on the energy consumption of configurations, we consider a wider range of metrics, and we analyse features at component level rather than source-code level. In the context of JHipster, identifying functions at the source code level presents a challenge due to the involvement of multiple technologies (e.g., REST APIs, databases) and languages (e.g., Java, SQL, DockerFiles, ...). To gain a deeper understanding, we focus on a single case study, comprehensively covering a large portion of the configuration space by employing various workloads and indicators, thereby sacrificing some external generality for depth of analysis.

**Recommender systems.** In recent years, we observed an increased interest in studies applying tools and approaches to assist users during the configuration of their systems, based on their functional needs. For instance, Pereira et al. propose a visual recommender system [26] based on proximity and similarity between features. Similarly, Zhang et al. [35] use dynamic profiling and analyze the stack trace of the system to locate features that can be changed

without altering the functional behavior of the system. Other approaches, such as [19] or [14], aim at updating the configuration of the system while it is running, to adapt it to the evolution of its environment. Such approaches rely on the performance of a subset of configurations sampled from the configuration space, and extrapolate such data to the complete configuration space. In contrast to such work, our focus does not lie in predicting performance or sampling configurations, but rather in understanding the system's performance by conducting an exhaustive assessment. To the best of our knowledge, only a limited number of publications have conducted a comprehensive evaluation of an entire configuration space. In particular, exhaustive assessments in these works primarily serve to validate prediction or optimization approaches [7, 9, 18]. However, such papers did not conduct a specific analysis of the correlations and trends within performance indicators or the performance of individual features.

## 2.2 JHipster

JHipster has already been studied in previous research work, making its variability model partly available. In particular, Halin et al. were the first to consider JHipster as a relevant case study [11]. They proposed to build a Software Product Line (SPL) to manage the variability of JHipster, in order to leverage existing tools and approaches from the SPL community on this industrial project. Then, in [10], they compare how effective different sampling approaches are in detecting defects in the source code of JHipster.

The variability of JHipster has also been leveraged to assess analysis methods and feature models reasoning approaches. In particular, Horcas et al. explore the use of Monte Carlo Tree Search for analysis of feature models and compare the efficiency of different Monte Carlo algorithms [13], and validate their approach on JHipster. Such work was not focused on the performance or optimization of configuration. Only Munoz et al. attempted to assess and optimize configurations of JHipster [20]. They propose an extension of feature models to support quality-aware reasoning for edge computing and IoT, and provide tooling support to analyze and optimize such systems. In particular, JHipster is one of the case studies assessed against five quality attributes, namely energy footprint, time, dependency, usability, and security. However, no in-depth analysis of the performance impact of each attribute on each configuration is provided, as such data is only used to validate the optimization approach. Also, no conclusions are drawn from attribute correlations.

## 3 METHODOLOGY & SETUP

This paper investigates the performance of a real-world highly-configurable software, regarding various performance indicators. In particular, we aim to answer the following research questions:

**RQ 1:** *How are the different performance indicators related? Variability is known to impact configuration performances [25]. However, collecting a large set of performance indicators is a tedious and error-prone process. Thus, if some performance indicators are strongly correlated, and if this correlation is known and understood, the collection of some performance indicators may prove unnecessary.*
<figure><img src="image_3.png" /><figcaption>Figure 1: The variability of JHipster explored in this paper. Cross-tree constraints are hidden.</figcaption></figure>

**RQ 2:** *Do the different options of JHipster impact the performance of generated systems in different ways?* JHipster proposes different technologies to meet technical requirements. Setting such options may result in different performances. Thus, analyzing the performance of configurations *w.r.t* options they contain may highlight how such options can impact performances.

**RQ 3:** *How does the default configuration perform compared to all valid configurations?* Most users only use the default configuration for their systems [34]. Thus, performance issues in the default configuration can affect a large set of deployed systems.

This paper does not exhaustively test all 100,000 JHipster configurations, as our analysis consists in assessing the performance of the software stack in production. That is, our evaluation excludes options related to (i) building tools, such as Maven or Gradle, (ii) development databases (H2disk, H2mem, etc.), and (iii) virtualization and orchestration tools, as we assume such options do not relate to nor impact software performance in production. As depicted by Figure 1, the JHipster variability under study covers options related to the database system, the cache system, the search engine, the authentication method, and the usage of reactive processing. This feature model was built based on the official documentation of JHipster, by extending and updating previously published JHipster feature models and through source-code mining on the JHipster GitHub repository.<fnref n="1" /> This model defines 118 valid configurations, which were all run and tested to ensure that they actually work and that the selected options are properly included. While it does not represent the entire configuration space of JHipster, prior work showed that measuring performance of multiple products at once was feasible by sampling and analyzing a minimal set of products [6].

To answer our research questions, we empirically assessed the performances of all 118 configurations, built using version 7.9.3 of JHipster. This section introduces the performance indicators that were monitored and the experimental protocol we followed to yield our results. All the performance data that were monitored during our experiments are publicly available.<fnref n="2" />

## 3.1 Measured performance indicators

The configurations were analyzed *w.r.t* a set of performance indicators related to response time, energy consumption, and various static metrics were measured and analyzed. To ensure realistic simulations, response time and power usage are not limited to a single metric. Instead, multiple indicators that cover different usage scenarios of JHipster were considered. This approach allowed for evaluating JHipster’s performance under various conditions and workloads.

*Response time.* Response times were measured through Gatling, a benchmarking tool designed to simulate heavy loads on web applications and measure their performance and stability. In particular, we monitored requests, such as the authentication request (*auth.tion*), the first authenticated request (*auth.ted*), the retrieval of all objects belonging to a certain entity (*getall*), the creation of an object of a given entity (*create*), the retrieval of this object (*get*), and the deletion of this object (*delete*). For each indicator, we measured the mean response time.

*Power usage.* To account for execution time variations between configurations, the energy efficiency of a configuration is defined as a consumption rate, i.e., power usage, expressed in Watts. The power usage is captured during a fixed idle period *idle-total* once the stack is booted, and then during the Gatling workload, reported as *wl-total*. To increase the granularity of power indicators, the total power usage is further distributed into two indicators, CPU power usage, and RAM power usage. Thus, six indicators are monitored: *idle-total*, *idle-cpu*, *idle-ram*, *wl-total*, *wl-cpu*, and *wl-cpu*.

*Static indicators.* Some JHipster configurations include external services, such as Redis or Elasticsearch, which increase the number of concurrently running containers. Thus, the number of Docker images *service* and their *size* evolve with configurations, which may impact other performance indicators. The number of selected features *features*, which is different from the number of images, is also monitored. Finally, the stack’s boot time *boot-time* is also accounted for in this category, as it is independent of the workload.

<footnote n="1">¹https://github.com/jbeerster</footnote>

<footnote n="2">²https://zenodo.org/records/8140600</footnote>
<table><caption>Table 1: Overview of the measured performance indicators.</caption><thead><tr><th>Indicator</th><th>Unit</th><th>mean</th><th>std</th><th>min</th><th>25%</th><th>50%</th><th>75%</th><th>max</th><th>factor</th><th>Description</th></tr></thead><tbody><tr><td>size</td><td>MB</td><td>1364.71</td><td>605.44</td><td>689.00</td><td>766.25</td><td>1373.50</td><td>1687.00</td><td>2719.00</td><td>3.95</td><td>Total size of all containers</td></tr><tr><td>services</td><td>Ø</td><td>2.63</td><td>0.61</td><td>2.00</td><td>2.00</td><td>3.00</td><td>3.00</td><td>4.00</td><td>2.00</td><td>Number of containers</td></tr><tr><td>boot-time</td><td>s</td><td>12.96</td><td>11.85</td><td>3.53</td><td>5.78</td><td>6.98</td><td>9.80</td><td>37.68</td><td>10.68</td><td>Delay before stack is operational</td></tr><tr><td>features</td><td>Ø</td><td>3.95</td><td>0.99</td><td>2.00</td><td>3.00</td><td>4.00</td><td>5.00</td><td>5.00</td><td>2.50</td><td>Number of selected features</td></tr><tr><td>auth.tion</td><td>ms</td><td>82.36</td><td>5.33</td><td>76.00</td><td>79.00</td><td>81.00</td><td>85.00</td><td>104.00</td><td>1.37</td><td>User authentication</td></tr><tr><td>auth.ted</td><td>ms</td><td>7.98</td><td>3.68</td><td>5.00</td><td>6.00</td><td>7.00</td><td>8.00</td><td>25.00</td><td>5.00</td><td>First authenticated request</td></tr><tr><td>getall</td><td>ms</td><td>11.07</td><td>5.56</td><td>6.00</td><td>7.00</td><td>8.00</td><td>14.00</td><td>23.00</td><td>3.83</td><td>Fetching all objects of an entity</td></tr><tr><td>create</td><td>ms</td><td>13.59</td><td>5.20</td><td>7.00</td><td>9.00</td><td>13.50</td><td>18.00</td><td>29.00</td><td>4.14</td><td>Creating an object</td></tr><tr><td>get</td><td>ms</td><td>6.91</td><td>2.41</td><td>5.00</td><td>6.00</td><td>6.00</td><td>7.00</td><td>20.00</td><td>4.00</td><td>Fetching the created object</td></tr><tr><td>delete</td><td>ms</td><td>16.53</td><td>8.84</td><td>5.00</td><td>8.00</td><td>12.50</td><td>25.75</td><td>29.00</td><td>5.80</td><td>Deleting the creating object</td></tr><tr><td>idle-cpu</td><td>W</td><td>1.34</td><td>1.55</td><td>0.16</td><td>0.69</td><td>0.96</td><td>1.54</td><td>9.22</td><td>59.00</td><td>CPU when booted and idle</td></tr><tr><td>idle-ram</td><td>W</td><td>0.80</td><td>0.78</td><td>0.09</td><td>0.43</td><td>0.63</td><td>1.01</td><td>4.54</td><td>52.59</td><td>RAM when booted and idle</td></tr><tr><td>idle-total</td><td>W</td><td>2.14</td><td>2.32</td><td>0.27</td><td>1.10</td><td>1.54</td><td>2.52</td><td>13.60</td><td>50.96</td><td>CPU &amp; RAM when booted and idle</td></tr><tr><td>wl-cpu</td><td>W</td><td>4.70</td><td>1.10</td><td>3.30</td><td>4.08</td><td>4.44</td><td>5.07</td><td>8.94</td><td>2.71</td><td>CPU under load</td></tr><tr><td>wl-ram</td><td>W</td><td>2.31</td><td>0.63</td><td>1.56</td><td>1.98</td><td>2.27</td><td>2.60</td><td>4.99</td><td>3.21</td><td>RAM under load</td></tr><tr><td>wl-total</td><td>W</td><td>7.01</td><td>1.72</td><td>4.86</td><td>6.14</td><td>6.76</td><td>7.67</td><td>13.95</td><td>2.87</td><td>CPU &amp; RAM under load</td></tr></tbody></table>

## 3.2 Experimental setup

To ensure reproducibility, all experiments were conducted in the same technical setup. The experiments were performed on the Grid'5000 computing grid [2], on a single-socket server equipped with a 2.20 GHz CPU and 96 GB of memory, running the operating system Ubuntu 20-04. The entire server was allocated for this experiment, preventing alterations in performances caused by other users. The specific version of each software component is listed in our open data package. All valid configurations were built as docker images and stored on a repository for reuse.

The assessment of each configuration is automatically performed by the following process: First, the idle power usage of the device is monitored over a 30-second period. Then, the generated stack is started, and the startup duration is logged. Once the stack is ready to accept requests, the power usage is monitored over a 30-second period to compute the idle-total indicator. The system is warmed up with synthetic workload. Then, the stack is stressed with the load testing tool Gatling, while the power usage is monitored to compute the wl-total indicator. The Gatling workload is the default scenario generated by JHipster to prevent user bias. Specifically, Gatling generated 100 users evenly distributed over a one-minute time window, with each user performing a scenario composed of 19 operations. The response-time indicators are extracted from the reports generated by Gatling.

Energy consumption metrics are obtained using Running Average Power Limit (RAPL) [17]. This tool provides the total power usage of the computing package, and the specific power usage of the DRAM and CPU [4]. All the power usages discussed in this article are cleaned of the idle power usage of the device. The idle power usage of the device is measured before each workload to ensure the finest granularity. All experiments were performed after a warmup of the device, to avoid heat-related variations in the power usage [32]. Finally, the performance of each configuration is estimated as the median value over five replications of the measurements.

## 4 PERFORMANCE OF CONFIGURATIONS

This section analyzes the performance of the 118 configurations of JHipster and look into the correlation between performance indicators. Table 1 depicts the variation in performances across configurations, for all the measured indicators. This table summarizes, for each indicator, its unit, mean, standard deviation, minimum, maximum, and quartiles. The factor between the maximum value and minimum value is also displayed. While some performance indicators such as authentication response times are consistent across configurations, most exhibit substantial variations. In particular, the workload power usage (wl-total) varies by a factor of 3, the size by a factor of 4, and some boot times are 11 times longer than others. The total idle power usage, i.e., the power usage when the configuration is running with no workload, varies by factors as high as 51. Therefore, one can conclude that the appropriate configuration of JHipster is crucial as it can have a significant impact on the performance of the generated software system.

Beyond the difference in performances across configurations, some indicators exhibit clustering patterns, as depicted in Figure 2. Specifically, in Figure 2c the boot time follows a bimodal distribution: being either below 10 seconds, or more than 30. Similar behavior is visible in Figure 2d, with size, in Figure 2b with create and delete response times. Finally, in Figure 2a, most configurations are grouped around the lower values, while some outliers appear to have substantially higher power usage.

This distribution raises some questions. Are the fastest configuration of create and delete the same configuration? Are they also the fastest to boot? The most power-efficient? Such questions can be generalized to performance indicators with no obvious clusters, such as get. Answering such questions requires a deeper analysis of the correlation between all the measured indicators: if some or all of these performance indicators are strongly correlated, then the assessment and optimization of performance would be substantially simplified. Finally, it would be beneficial to identify if such behaviors are caused by specific options.
<figure><img src="image_4.png" /><figcaption>(a) Variation in power usage.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(c) Variation in boot time.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Variation in response time.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(d) Variation in total size.</figcaption></figure>

<caption>Figure 2: Variations in the measured indicators across configurations.</caption>

## 4.1 Correlation in indicator groups

We analyzed correlations within each group of indicators—i.e., between static indicators, energetic indicators, and response times. The objective is to identify indicators that can be estimated based on others, potentially serving as proxies for other indicators and simplifying the experimental setup and result analysis. As observed in Figure 2, some indicators exhibit a bimodal distribution. Therefore, all correlations reported in Figure 3 are Spearman Correlation coefficients, computed using the median value of five replications for each of the 118 configurations. To account for the increased risk of Type I errors (false positives) due to multiple correlation tests performed on the same dataset, we applied the Benjamini-Hochberg procedure to control the False Discovery Rate (FDR). The FDR-adjusted p-values are detailed in Figure 3b. Correlations coefficients higher than 0.75 are considered as absolutely strong, and correlations lower than 0.5 are considered as absolutely weak. Only the correlations with adjusted p-values below 0.05 are considered significant.

4.1.1 Correlation between static indicators. Static indicators include size, services, boot-time, and features. While all the correlations are positive, some are weaker: in particular, the correlations boot-time and services or features are 0.22 and 0.5, respectively. By contrast, boot-time is strongly correlated to size, by 0.72. The indicators services and features are correlated by 0.57. This can be explained by the design of the variability: the services are represented as features in the feature model, but not all features are services. Finally, services and features appear weakly correlated to size, by 0.5 and 0.36, respectively. Thus, a higher number of services or features tends to increase size, which in turn affects boot-time. Therefore, size may be a relevant indicator to optimize, to indirectly optimize services, boot-time, and features metrics.

4.1.2 Correlation between response times. The correlations between response times indicators are all positive, suggesting that faster configurations tend to be faster in all indicators. However, the correlations range from 0.13 to 0.9. The authentication and authenticated requests are strongly correlated—i.e., 0.73. They are only weakly correlated to other requests, such as get request (0.41 and 0.50) and create (0.23 and 0.19). The get and getall indicators only have correlations of lower magnitude, below 0.5, with the other response times. Finally, the create and delete requests are strongly correlated (0.9). Such a correlation tends to confirm the hypotheses formulated about the clustering of values in Figure 2b: the slowest configurations on create are also the slowest in delete. The delete request is only weakly correlated to other response times, with a correlation lower than 0.39. The create request has weaker correlations to the other response times, between 0.3 and 0.5.

4.1.3 Correlation between energetic indicators. All the energy-related indicators are strongly correlated, with correlations ranging from 0.7 to 1. The CPU and RAM power usage are strongly correlated, both in idle (0.95) and under load (0.96). Across idle and workload, the performance of the CPU and RAM are also correlated: their respective idle and workload power usages are correlated at 0.7 and 0.82, respectively. Consequently, the total power usage in idle is strongly correlated to the power usage under load (0.75). Since the total power usage is computed as the sum of the CPU and RAM power usage, these indicators are not independent and their correlations to the total power usage are not relevant in both idle and workload states.

## 4.2 Correlations across indicator groups

While some proxy indicators appeared inside each indicator group, assessing the performance of a configuration still requires a dedicated analysis for each indicator group. Therefore, to further simplify performance assessment and optimization, it appears relevant
<figure><img src="image_8.png" /><figcaption>Figure 3: Correlations between measured indicators and their significance.</figcaption></figure>

<figure><img src="image_9.png" /></figure>

to explore proxy indicators that enable the estimation of performance across different groups of indicators.

### 4.2.1 Static indicators and time.
The response time to create or delete requests have the strongest correlation to static indicators, in particular the number of services (0.72 for create and 0.83 for delete). The correlations between create and delete and other static indicators may be the consequence of the internal correlations between static indicators. Aside from the create and delete indicators, the response times are only weakly correlated with static indicators. The correlations range from -0.05 to -0.49. Thus, only services is a potential proxy indicator for create and delete, and static indicators do not provide a means to assess the response time of the other operations.

### 4.2.2 Static indicators and power.
The correlation between power indicators and size ranges from 0.76 to 0.85. In particular, size is strongly correlated to all energetic indicators (between 0.76 and 0.85), while boot-time is strongly correlated to the idle power usage (between 0.73 and 0.74). The correlation between the remaining static indicators service and features and the energetic indicators are weaker, between 0.17 and 0.45. However, service is overall more correlated to energetic under load (0.36 to 0.45) than in idle (0.23 to 0.28), whereas features exhibits the opposite behavior (0.26 to 0.34 in idle, and 0.17 to 0.21 under load). Therefore, size may be a proxy for idle and load power usages, and boot-time may be a proxy only for idle power usage.

### 4.2.3 Time and power.
The response times are systematically more strongly correlated to the power usage under load (between 0.31 and 0.61) than to the one in idle (between 0.01 and 0.44). The

behavior may be explained by the fact that both response times and power usage underquantify the behavior of the system at runtime. Furthermore, the power usage of the RAM is less correlated to response time than the power usage of the CPU. The power usage under load is more correlated to authentication, getall, create, and delete (between 0.47 and 0.61) than to authenticated and get (between 0.31 and 0.42).

## 4.3 Proxy indicators

Some of the analyzed indicators exhibit strong correlations. In particular, all the energetic indicators are strongly correlated with each other. The idle CPU and RAM power usage are strong predictors of the total idle power usage, and the CPU and RAM usage under load are strong predictors of the total power usage under load (with $R^2$ of respectively 0.99, 0.98, 0.98, and 0.96 in linear regression). The total power usage in idle is also a good predictor of the total power usage under load ($R^2$ = 0.71). Static indicators are also correlated with each other, and in particular size is a good proxy indicator for boot-time ($R^2$ = 0.68). Proxy indicators also appear between response times, in particular with authentication and authenticated ($R^2$ = 0.71) and between create and delete ($R^2$ = 0.84). However, proxy indicators within the same group of indicators have a limited interest. Indeed, except for boot-time, the ability to measure a given indicator of a group implies the ability to measure all indicators of that group.

Across indicator scopes, only a limited set of indicators are correlated: size and power usages, and some response times with service or the power indicators under load. However, such correlations do not convert into strong predictors. In particular, size
can only approximate the total power usage under load ($R^2$ = 0.29). In idle, size is a better predictor of the RAM power usage than the total power usage ($R^2$ of respectively 0.24 and 0.16). service provides better estimates of the responses times of create and delete ($R^2$ of respectively 0.51 and 0.68). Among response times, only authentication and getall allow for estimating the total power usage under load ($R^2$ of respectively 0.32 and 0.26). Other response times perform worse in such a task ($R^2$ < 0.1).

Thus, predicting an indicator using a proxy indicator remains a challenge across indicator groups. However, using a set of proxy indicators, rather than a single one, may compensate for such correlations. In particular, we explore predicting a given indicator using only indicator from other groups, e.g., predicting the total power usage of the system from the static and response time indicators. Table 2 compares prediction accuracy from each indicator, when using either all other metrics (i.e., metrics from all groups except the target) versus using only metrics from other groups. In addition, the coefficient of variation (column CV) for such indicators is provided, to assess whether such accuracy can be imputed to random changes. Predictions were performed using a random forest regressor. We observe that while most indicators achieve accuracy significantly better than random chance, some exceptions arise. For instance, authentication and features indicators have a low mean absolute error (MAE) but also a low CV, as the collected metrics show little variations, which reduces the practical value of predicting these indicators. Conversely, the idle power usage indicators exhibit higher MAE and CV, reflecting greater variability but also increased prediction difficulty. authenticated and getall tend to have higher MAE despite limited CV, indicating that such metrics are harder to model for the regressor.

Overall, limiting the training to indicators from other groups as a cost ranging from 1 to 22 percentage points in mean absolute error. Nonetheless, the resulting predictions still substantially outperform random chance, enabling the prediction of harder-to-predict indicators by leveraging other indicator groups.

**RQ 1:** Some indicators exhibit strong correlations, in particular when they quantify measurements of similar natures, allow the use of proxy indicators to simplify performance assessment and optimization. Such proxies can serve as warning signs that a system might exhibit poor performance in unmeasured indicators. However, they do not allow for quantifying precisely such a performance without specific measure. Correlations also exist across indicators of different natures, but they tend to be weaker and less frequent. Nonetheless, it is possible to predict such indicators using a compositing of proxies from other indicator groups.

# 5 PERFORMANCE OF OPTIONS

This section aims to understand the impact of options of JHipster on its performance and to discover means to create efficient configurations through relevant option selection. As the previous section highlighted that the CPU, the RAM, and the total power usage are strongly correlated (more than 0.96), only the latter is discussed in the remainder of this paper.

<table><caption>Table 2: Mean absolute error (MAE) of each indicator, predicted from metrics from all groups versus metrics from other groups.</caption><thead><tr><th>Indicators</th><th>MAE (%)<br>(All groups)</th><th>MAE (%)<br>(Other groups)</th><th>CV (%)</th></tr></thead><tbody><tr><td>size</td><td>6</td><td>5</td><td>44</td></tr><tr><td>services</td><td>7</td><td>6</td><td>23</td></tr><tr><td>boot-time</td><td>12</td><td>17</td><td>91</td></tr><tr><td>features</td><td>19</td><td>17</td><td>25</td></tr><tr><td>authentication</td><td>3</td><td>4</td><td>6</td></tr><tr><td>authenticated</td><td>13</td><td>21</td><td>46</td></tr><tr><td>getall</td><td>32</td><td>31</td><td>50</td></tr><tr><td>create</td><td>7</td><td>13</td><td>38</td></tr><tr><td>get</td><td>10</td><td>15</td><td>35</td></tr><tr><td>delete</td><td>10</td><td>17</td><td>53</td></tr><tr><td>idle-CPU</td><td>9</td><td>32</td><td>115</td></tr><tr><td>idle-RAM</td><td>11</td><td>27</td><td>97</td></tr><tr><td>idle-total</td><td>8</td><td>30</td><td>108</td></tr><tr><td>wl-CPU</td><td>3</td><td>7</td><td>23</td></tr><tr><td>wl-RAM</td><td>3</td><td>8</td><td>27</td></tr><tr><td>wl-total</td><td>2</td><td>8</td><td>24</td></tr></tbody></table>

## 5.1 Performance of individual options

This section compares the performance of various options available in JHipster. The detailed performances of each option are listed in Table 3.

**Authentication.** In our experiment, we analyzed two authentication methods: session and JWT. There is no significant difference w.r.t. energy consumption and static indicators between the two authentication methods, but some variations can be observed regarding response time. Specifically, the authentication and first authentication requests are slower with session than with JWT by 4% and 12%, respectively. Then, session is slightly faster on the remaining operations getall (5%), and less than 2% faster on the remaining operations create, get, and delete.

**Databases.** JHipster supports 8 databases: MySQL, Microsoft SQL Server (MSSQL), Postgres, MariaDB, Mongo, Cassandra, Neo4j, and Couchbase. The total size of the stack differs depending on the database. While most database footprints are between 690 Mb and 900 Mb, the average size of configurations is 1730 Mb for the ones containing Couchbase and 1925 Mb for the ones containing MSSQL. Thus, the choice of database appears to have a strong impact on the size of the system, and can partially explain the spread in size in Figure 2d. Finally, while most databases have boot times between 3.7 and 6.7 seconds, MSSQL increases this value to 34 seconds. It thus appears that MSSQL is responsible for the clustering visible in Figure 2c.

The selected database also affects response times. For instance, Neo4j is among the slowest across all response times, except for getall. Contrarily, Couchbase is among the fastest across all response times, but the slowest for getall.

As for energy indicators, Couchbase exhibits an outlying behavior: its power usage is higher in idle (13, 59 W in total) than under load (13, 51 W in total). This behavior is due to the design of Couchbase, which performs opportunistic indexation in idle to
<table><caption>Table 3: Average performance of configurations containing each option</caption><thead><tr><th></th><th></th><th>size</th><th>services</th><th>boot-time</th><th>features</th><th>authentication</th><th>auth-ted</th><th>getall</th><th>create</th><th>get</th><th>delete</th><th>idle-total</th><th>wl-total</th></tr></thead><tbody><tr><td rowspan="8">Database</td><td>Cassandra</td><td>907.00</td><td>3.00</td><td>4.05</td><td>2.00</td><td>80.00</td><td>8.00</td><td>8.00</td><td><strong>7.00</strong></td><td>6.00</td><td><strong>5.00</strong></td><td>1.72</td><td>7.37</td></tr><tr><td>Couchbase</td><td>1730.00</td><td><strong>2.00</strong></td><td>6.69</td><td>2.00</td><td>86.00</td><td><strong>6.00</strong></td><td>17.00</td><td><strong>7.00</strong></td><td><strong>5.00</strong></td><td><strong>5.00</strong></td><td>13.59</td><td>13.51</td></tr><tr><td>MariaDB</td><td>695.00</td><td><strong>2.00</strong></td><td>5.13</td><td>2.00</td><td>80.00</td><td>8.00</td><td><strong>6.00</strong></td><td>8.00</td><td>6.00</td><td>7.00</td><td>0.49</td><td><strong>4.93</strong></td></tr><tr><td>MongoDB</td><td>740.00</td><td><strong>2.00</strong></td><td><strong>3.73</strong></td><td>2.00</td><td><strong>77.00</strong></td><td>7.00</td><td><strong>6.00</strong></td><td>7.00</td><td>6.00</td><td>6.00</td><td>0.58</td><td>5.15</td></tr><tr><td>MSsql</td><td>1925.00</td><td><strong>2.00</strong></td><td>33.76</td><td>2.00</td><td>81.00</td><td>9.00</td><td>7.00</td><td>9.00</td><td>7.00</td><td>8.00</td><td>2.41</td><td>6.77</td></tr><tr><td>MySQL</td><td>765.00</td><td><strong>2.00</strong></td><td>5.39</td><td>2.00</td><td>82.00</td><td>10.00</td><td>7.00</td><td>9.00</td><td>7.00</td><td>9.00</td><td>1.07</td><td>6.82</td></tr><tr><td>Neo4j</td><td>869.00</td><td><strong>2.00</strong></td><td>6.68</td><td>2.00</td><td>86.00</td><td>12.00</td><td>10.00</td><td>12.00</td><td>9.00</td><td>9.00</td><td>1.19</td><td>6.45</td></tr><tr><td>PostgreSQL</td><td><strong>689.00</strong></td><td><strong>2.00</strong></td><td>5.16</td><td>2.00</td><td>80.00</td><td>8.00</td><td>7.00</td><td>8.00</td><td>7.00</td><td>7.00</td><td><strong>0.27</strong></td><td>5.04</td></tr><tr><td rowspan="2">Authentication</td><td>JWT</td><td>1358.88</td><td>2.67</td><td><strong>13.60</strong></td><td>4.06</td><td><strong>78.69</strong></td><td><strong>6.51</strong></td><td>10.57</td><td>13.29</td><td>6.39</td><td>16.98</td><td><strong>1.78</strong></td><td><strong>6.61</strong></td></tr><tr><td>Session</td><td><strong>1358.71</strong></td><td>2.67</td><td>13.64</td><td>4.06</td><td>83.39</td><td>7.43</td><td><strong>10.02</strong></td><td><strong>13.12</strong></td><td><strong>6.31</strong></td><td><strong>16.86</strong></td><td>1.82</td><td>6.66</td></tr><tr><td rowspan="2">Cache</td><td>No cache</td><td><strong>1353.62</strong></td><td><strong>2.50</strong></td><td><strong>12.94</strong></td><td><strong>2.50</strong></td><td>82.31</td><td>9.19</td><td><strong>6.62</strong></td><td><strong>13.00</strong></td><td>6.62</td><td><strong>16.75</strong></td><td><strong>1.51</strong></td><td><strong>6.49</strong></td></tr><tr><td>Any cache</td><td>1385.29</td><td>2.70</td><td>14.46</td><td>4.50</td><td><strong>80.96</strong></td><td><strong>6.49</strong></td><td>11.30</td><td>13.46</td><td><strong>6.34</strong></td><td>17.35</td><td>1.90</td><td>6.69</td></tr><tr><td rowspan="5">Cache</td><td>Caffeine</td><td><strong>1355.19</strong></td><td><strong>2.50</strong></td><td>13.22</td><td>4.50</td><td>79.25</td><td>6.12</td><td>7.25</td><td><strong>13.06</strong></td><td>6.25</td><td>16.62</td><td><strong>1.46</strong></td><td>6.31</td></tr><tr><td>Ehcache</td><td>1355.38</td><td><strong>2.50</strong></td><td><strong>13.10</strong></td><td>4.50</td><td><strong>79.06</strong></td><td><strong>6.06</strong></td><td>7.25</td><td><strong>13.06</strong></td><td><strong>6.12</strong></td><td><strong>16.38</strong></td><td>1.58</td><td><strong>6.29</strong></td></tr><tr><td>Hazelcast</td><td>1371.31</td><td><strong>2.50</strong></td><td>16.10</td><td>4.50</td><td>82.31</td><td>6.81</td><td>13.62</td><td>13.50</td><td>6.50</td><td>17.25</td><td>2.96</td><td>7.35</td></tr><tr><td>Infinispan</td><td>1367.88</td><td><strong>2.50</strong></td><td>16.36</td><td>4.50</td><td>80.25</td><td>6.12</td><td><strong>7.19</strong></td><td>13.81</td><td>6.19</td><td>17.12</td><td>1.63</td><td>6.36</td></tr><tr><td>Redis</td><td>1476.69</td><td>3.50</td><td>13.53</td><td>4.50</td><td>83.94</td><td>7.31</td><td>21.19</td><td>13.88</td><td>6.62</td><td>19.38</td><td>1.88</td><td>7.13</td></tr><tr><td rowspan="2">CouchbaseSE</td><td>No</td><td>1733.50</td><td>2.00</td><td>6.43</td><td><strong>2.50</strong></td><td><strong>86.50</strong></td><td>7.00</td><td>19.50</td><td>7.00</td><td>5.00</td><td>5.00</td><td><strong>13.49</strong></td><td><strong>13.34</strong></td></tr><tr><td>Yes</td><td>1733.50</td><td>2.00</td><td><strong>6.35</strong></td><td>3.50</td><td>87.00</td><td>7.00</td><td><strong>19.00</strong></td><td>7.00</td><td>5.00</td><td>5.00</td><td>13.59</td><td>13.85</td></tr><tr><td rowspan="2">Elasticsearch</td><td>No</td><td><strong>1024.48</strong></td><td><strong>2.14</strong></td><td><strong>12.84</strong></td><td><strong>3.52</strong></td><td><strong>81.96</strong></td><td><strong>7.95</strong></td><td>10.86</td><td><strong>9.39</strong></td><td><strong>6.98</strong></td><td><strong>8.86</strong></td><td><strong>1.45</strong></td><td><strong>6.16</strong></td></tr><tr><td>Yes</td><td>1694.95</td><td>3.14</td><td>13.87</td><td>4.52</td><td>82.50</td><td>8.11</td><td><strong>10.80</strong></td><td>18.50</td><td>7.02</td><td>25.45</td><td>2.04</td><td>7.38</td></tr><tr><td rowspan="2">Reactive</td><td>No</td><td><strong>1370.08</strong></td><td>2.42</td><td>10.42</td><td><strong>2.50</strong></td><td><strong>81.08</strong></td><td><strong>8.08</strong></td><td><strong>8.33</strong></td><td><strong>11.83</strong></td><td><strong>6.25</strong></td><td><strong>14.50</strong></td><td>3.44</td><td><strong>7.58</strong></td></tr><tr><td>Yes</td><td>1380.83</td><td>2.42</td><td><strong>9.37</strong></td><td>3.50</td><td>92.42</td><td>16.17</td><td>16.83</td><td>17.25</td><td>11.67</td><td>14.92</td><td><strong>3.24</strong></td><td>9.03</td></tr></tbody></table>

optimize response times under load. All other databases have idle power usage between 0.5 W and 2.4 W, except PostgreSQL with 0.27 W, and load power usage between 4.9 W and 6.8 W, except for Cassandra with 7 W.

**Hibernate cache.** JHipster supports the activation ofHibernate cache for SQL databases. This analysis only accounts for configurations whereHibernate cache is available and activated, while configurations where theHibernate cache is available and disabled serve as a baseline. Activating theHibernate cache appears to increase the boot time by 12% and the response time of create and delete requests by 3.5%. The response time for getall is increased by 71%. Such overhead is leveraged in reading operations: the response times for get and authenticated are reduced by 4% and 29%, respectively. While activating theHibernate cache substantially increases the idle power usage (+26%), it has a limited impact on the power usage under load (+3%).

**Cache provider.** When theHibernate cache is enabled, a cache provider must be selected among the five offered by JHipster. In Table 3, the "No cache" line refers to configurations where a cache system was available, but was not selected and thus acts as a baseline. The choice of cache system affects static indicators. Specifically, Redis increases the service count and the size of configurations. Infinispan and Hazelcast increase the boot time by 26% and 24%,

respectively, whereas other providers increase such boot time by less than 5%.

The choice of the cache system also impacts the response times. Some operations are consistently affected. For instance, response times for authenticated are reduced by up to 35%, and create are increased by up to 7%. However, some cache systems have outlying behavior. For instance, Hazelcast and Redis increase getall by respectively 106% and 220% while other systems only increase it by 10%. Similarly, Caffeine and Ehcache do not affect delete response times while other systems increase it by up 16%. Ehcache appears to have the best performance across response times, while Caffeine is a close second. Contrarily, Redis appears to provide the worst performance gains.

Finally, the choice of cache systems also impacts the energy efficiency of the stack. Hazelcast and Redis substantially increase the idle power usage, by 96% and 25%, respectively. Under load, such a difference is reduced to +13% and +10%, respectively. Ehcache and Infinispan increase the idle power usage by 5% and 8%, respectively, but reduce the power usage under load by -3% and -2%. Finally, Caffeine reduces the power usage by 3%, both in idle and under load. Overall, configurations containing Ehcache or Caffeine exhibit the lowest power usage, both idle and under load.

**Search engine.** JHipster supports Elasticsearch or Couchbase as search engines (S. E.), providing support to search content in the database. Each search engine is compared to its respective baseline
of configurations that could contain them but do not. Configura-
tions with CouchbaseSE are thus compared to configurations with
Couchbase and not CouchbaseSE, while configurations with Elas-
ticsearch are compared to configurations without Elasticsearch,
Couchbase, or Cassandra. CouchbaseSE is embedded in Couchbase,
and thus does not affect size and boot time, whereas Elasticsearch
increases them by 65% and 8%, respectively. Thus, Elasticsearch
can also be partially responsible for the spread in size visible in
Figure 2d. While both search engines have a limited impact on re-
sponse times, Elasticsearch increases the response time of create
by 97% and delete by 187%. Finally, enabling the Couchbase search
engine has only a limited impact on power usage: +1% in idle and
+4% under load. Contrarily, enabling Elasticsearch substantially
increases power usage, by 41% in idle and 20% under load.

*Reactive.* The Reactive option enables reactive programming for the application. Reactive allows for creating an asynchronous and non-blocking API, in order to improve the scalability of the system. Reactive has a limited impact on static indicators, by reducing the boot time by 10%. However, it substantially increases the response times. In particular, getall and get are increased by 102% and 87%, respectively. This difference is more limited for create, 46%. Only delete is unaltered. Furthermore, reactive reduces the power usage in idle by 6%, but increases the power usage under load by 19%.

**RQ 2:** The available options have different impacts on the performance of the generated system. This impact is not consistent across indicators: the same option can have a negligible impact on a given indicator, while substantially affecting another one. Thus, an informed feature selection per indicator is needed to achieve efficient configurations tailored to specific requirements.

5.2 Performance of default configuration

In JHipster, the default configuration is the one generated when
selecting the default option for each prompt in the JHipster CLI
tool. This configuration includes features like JWT, PostgreSQL,
and Ehcache, while the search engine and the reactive database
are disabled. As the default configuration, it may be the most used
configuration of JHipster [34]. To compare this with all other valid
configurations, their overall performance is aggregated into a single
value. This value is calculated as the average of the configuration's
percentile across all monitored performance indicators, where each
indicator is given equal weight in determining the overall perfor-
mance.

The ranks of the default configuration regarding each indica-
tor are detailed in Table 4. When evaluating the overall rank of
each configuration, with all metrics equally weighted, the default
configuration performs well compared to other valid configura-
tions. Its average percentage rank is 8.9%, indicating that, on av-
erage, it outperforms 91.1% of all other configurations across the
monitored indicators. However, four configurations achieve better
overall ranks, making them potential candidates for improving the
default configuration’s performance.

These candidate configurations share similarities with the cur-
rent default configuration, as shown in Table 4. The first candidate

uses MongoDB instead of PostgreSQL and thus has no cache sys-
tem, while candidates 2, 3, and 4 replace PostgreSQL with MariaDB,
Ehcache with Caffeine, or both. Candidate 1 exhibits worse ranking
in terms of size, authenticated response time, and w1-total
power usage. However, these regressions are outweighed by bet-
ter ranks in 6 other indicators, resulting in an average percentage
rank of 8.1%. Candidates 2, 3, and 4 are closer to the default con-
figuration and only affect the size, boot-time, idle-total, and
w1-total indicators. All three configurations improve boot-time.
Additionally, candidate 2 improves idle-total but worsens size
and w1-total, while candidate 4 improves w1-total and worsens
size and idle-total.

Candidate 3 stands out by improving all four indicators without
any regressions, allowing it to achieve an average percentage rank
of 8%. It thus make it the top-ranked configuration in our evaluation
and therefore, the best candidate for replacing the default configura-
tion. Specifically, this improvement would be implemented by only
replacing Ehcache with Caffeine. This results in a significant reduc-
tion of idle-total by 20.9%, while the benefits of this replacement
are more modest regarding w1-total, size, and boot-time, by re-
spectively 1.2%, 0.1% and 0.7% (other indicators remain unaffected).

While this optimization aimed to maximize a single objective (optimizing all 12 indicators equally) it can also be adapted to focus on different sets of indicators or assign varying weights to each, allowing for the maximization of specific performance profiles. This approach can be applied not only to the default configuration but also to any custom configuration defined by the practitioner. For instance, when optimizing specifically for boot-time, getall, and w1-total, the default configuration achieves an average percentage rank of 20% for such indicators but is outperformed by 10 configurations. The best-performing candidate configuration has an average percentage rank of 10% and outperforms the default configuration with improvements of 2.1%, 14.3%, and 7.4% in boot-time, getall, and w1-total, respectively.

**RQ 3:** The default configuration of JHipster offers strong performances as a balanced configuration. However, when comparing its average ranking across the considered indicators, it is outperformed by several other candidates. This approach can be generalized to optimize the default configuration for any chosen set of indicators and shows that it can be more relevant to select configurations based on indicators to optimize rather than the preferred technologies.

6 DISCUSSION AND LESSONS LEARNED

As an empirical study, the results presented in this article must be
interpreted in the context of the applied methodology.

**Threats to validity.** The results of this study cannot be gen-
eralized as a ground truth about the overall performance of each
analyzed services, as the configurations were assessed on a specific,
synthetic workload automatically generated by JHipster during
the creation of the project. This may affect the applicability of our
findings to other contexts and workloads, in particular regarding
the performance of individual options discussed in Section 5. More-
over, the performance model for each option was derived from
this specific workload, aiming to explore the potential for inferring
<table><caption>Table 4: Ranks and percentage rank of the default configuration and candidates configurations w.r.t all valid configurations</caption><thead><tr><th rowspan="2"></th><th colspan="2">Default</th><th colspan="2">Candidate 1</th><th colspan="2">Candidate 2</th><th colspan="2">Candidate 3</th><th colspan="2">Candidate 4</th></tr><tr><th>Rank</th><th>Percentage</th><th>Rank</th><th>Percentage</th><th>Rank</th><th>Percentage</th><th>Rank</th><th>Percentage</th><th>Rank</th><th>Percentage</th></tr></thead><tbody><tr><td>size</td><td>6</td><td>5.1</td><td>23</td><td>19.5</td><td>9</td><td>7.6</td><td>3</td><td>2.5</td><td>9</td><td>7.6</td></tr><tr><td>services</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td></tr><tr><td>boot-time</td><td>24</td><td>20.3</td><td>3</td><td>2.5</td><td>21</td><td>17.8</td><td>22</td><td>18.6</td><td>23</td><td>19.5</td></tr><tr><td>features</td><td>33</td><td>28.0</td><td>1</td><td>0.8</td><td>33</td><td>28</td><td>33</td><td>28.0</td><td>33</td><td>28.0</td></tr><tr><td>auth.tion</td><td>1</td><td>0.8</td><td>6</td><td>5.1</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td></tr><tr><td>auth.ted</td><td>1</td><td>0.8</td><td>39</td><td>33.1</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td><td>1</td><td>0.8</td></tr><tr><td>getall</td><td>12</td><td>10.2</td><td>1</td><td>0.8</td><td>12</td><td>10.2</td><td>12</td><td>10.2</td><td>12</td><td>10.2</td></tr><tr><td>create</td><td>9</td><td>7.6</td><td>1</td><td>0.8</td><td>9</td><td>7.6</td><td>9</td><td>7.6</td><td>9</td><td>7.6</td></tr><tr><td>get</td><td>6</td><td>5.1</td><td>6</td><td>5.1</td><td>6</td><td>5.1</td><td>6</td><td>5.1</td><td>6</td><td>5.1</td></tr><tr><td>delete</td><td>9</td><td>7.6</td><td>7</td><td>5.9</td><td>9</td><td>7.6</td><td>9</td><td>7.6</td><td>9</td><td>7.6</td></tr><tr><td>idle-total</td><td>17</td><td>14.4</td><td>10</td><td>8.5</td><td>8</td><td>6.8</td><td>12</td><td>10.2</td><td>18</td><td>15.3</td></tr><tr><td>wl-total</td><td>7</td><td>5.9</td><td>17</td><td>14.4</td><td>9</td><td>7.6</td><td>5</td><td>4.2</td><td>1</td><td>0.8</td></tr><tr><td>Average</td><td>10.5</td><td>8.9</td><td>9.6</td><td>8.1</td><td>9.9</td><td>8.4</td><td>9.5</td><td>8</td><td>10.3</td><td>8.7</td></tr></tbody></table>

high-performance configurations for a given workload and performance indicator. This approach may affect the generalizability of our results to other data models [21]. Additionally, all services were hosted on a single device, which may not accurately reflect production environments, especially in terms of response times and power usage. To address these limitations, future work will expand the scope by evaluating additional usage scenarios and adding additional execution contexts in the feature model.

**JHipster as case study.** This experiment investigates the suitability of JHipster as a use case for research on the performance of configurable systems at the component level. JHipster offers several features that proved useful in this experiment, particularly its toolchain, which simplifies converting configurations files into running systems with little technical overhead. Additionally, the built-in testing scenario generation, including performance tests via Gatling, allowed for an effective evaluation of the system's usability and performance. However, different configuration options offer distinct features, meaning not all configurations are functionally equivalent. Specifically, search engines offer an additional API endpoint for content lookups, which was not evaluated in this experiment. Thus, the metrics reported in this paper reflect the impact of each option on the overall system performance, rather than their individual performance for their respective responsibility. Such an experimental approach would require designing configuration-specific test scenarios based on the options they contain.

However, the simplicity of JHipster's toolchain does not apply to all configuration options. For instance, replacing the default docker-compose option with more complex alternatives such as micro-services or Kubernetes requires additional technical effort. Such alternatives were not explored in this experiment, which reduces the considered configuration space. Thus, no performance data was produced for these options, leaving their potential impact on overall system performance unknown.

In addition to these limitations, certain options or pairs of options were not considered in this experiment. The Cassandra database system is not compatible with Elasticsearch due to errors in the code generation system. Similarly, Couchbase, Neo4J, and Reactive

cannot be combined with the JWT option, and Reactive is incompatible with configurations containing Cassandra, Neo4j, Session, or a cache mechanism. The Oracle database was excluded due to licensing restrictions, and community-developed options such as GRPC and JavaEE were excluded as they are not officially supported.

As JHipster lacks a comprehensive feature model, such constraints were discovered either through documentation or trial and error. While some models, such as the one used in this paper, or by Halin et al. [11] are available, the JHipster community would benefit from an official and exhaustive feature model.

## 7 CONCLUSION

This article presents an empirical analysis of the variability and performance of the configurable stack generator JHipster, demonstrating its relevance as a use case for variability analysis. The study shows how JHipster's diverse configurations significantly impact performance across various indicators, with strong correlations observed among similar indicators, enabling proxy-based optimization. Despite weaker correlations between different types of indicators, it is still possible to predict them by leveraging a combination of proxy indicators rather than relying on a single one. This approach thus enables the estimation of complex-to-measure indicators using more easily obtainable ones. Performance variations are mainly driven by specific configuration options, which may affect indicators unevenly—an option with minimal impact on one may significantly influence another. This study and dataset on JHipster's performance indicators will contribute to future research on configurable software, especially in applying multi-objective optimization to balance trade-offs and identify configurations that achieve strong performance across multiple indicators.

## ACKNOWLEDGMENT

This work was partially funded by the ANR-19-CE25-0003 KOALA and ANR-21-CE25-0022 DISTILLER projects. Experiments presented in this article were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER, and several Universities, as well as other organizations (see https://www.grid5000.fr).
REFERENCES

[1] Paolo Arcaini, Omar Inverso, and Catia Trubiani. 2021. Automated model-based performance analysis of software product lines under uncertainty. In Proceedings of the 25th ACM International Systems and Software Product Line Conference - Volume A (Leicester, United Kingdom) (SPLC '21). Association for Computing Machinery, New York, NY, USA, 112.

[2] Daniel Balouek, Alexandra Carpen Amarie, Ghislain Charrier, Frédéric Desprez, Emmanuel Jeannot, Emmanuel Jeanvoine, Adrien Lèbre, David Margery, Nicolas Nicolaus, Lucas Nussbaum, Olivier Richard, Christian Pérez, Flavien Quesnel, Cyril Rohr, and Luc Sarzyniec. 2013. Adding Virtualization Capabilities to the Grid'5000 Testbed. In Cloud Computing and Services Science, Ivan I. Ivanov, Marten van Sinderen, Frank Leymann, and Tony Shan (Eds.). Communications in Computer and Information Science, Vol. 367. Springer International Publishing, 3–20. https://doi.org/10.1007/978-3-319-04519-1_1

[3] Alexandre Bonvoisin, Clément Quinton, and Romain Rouvoy. 2024. Understanding the Performance-Energy Tradeoffs of Object-Relational Mapping Frameworks. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). Rovaniemi, Finland, 11.

[4] Spencer Desrochers, Chad Paradis, and Vincent M. Weaver. 2016. A Validation of DRAM RAPL Power Measurements. In Proceedings of the Second International Symposium on Memory Systems (Alexandria, VA, USA) (MEMSYS '16). Association for Computing Machinery, New York, NY, USA, 455–470.

[5] Johannes Dorn, Sven Apel, and Norbert Siegmund. 2023. Mastering uncertainty in performance estimations of configurable software systems. Empirical Software Engineering 28, 2 (2023), 33.

[6] Édouard Guégain, Clément Quinton, and Romain Rouvoy. 2021. On reducing the energy consumption of software product lines. In Proceedings of the 25th ACM International Systems and Software Product Line Conference-Volume A. 89–99.

[7] Edouard Guégain, Amir Taherkordi, and Clément Quinton. 2023. Configuration optimization with limited functional impact. In International Conference on Advanced Information Systems Engineering. Springer, 53–68.

[8] Jianmei Guo, Krzysztof Czarnecki, Sven Apel, Norbert Siegmund, and Andrzej Wasowski. 2013. Variability-Aware Performance Prediction: A Statistical Learning Approach. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, 301–311.

[9] Edouard Guégain, Amir Taherkordi, and Clément Quinton. 2023. ICO: A Platform for Optimizing Highly Configurable Systems. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). 62–67.

[10] Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. Empirical Software Engineering 24 (2019), 674–717.

[11] Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Patrick Heymans. 2017. Yo variability! JHipster: a playground for web-apps analyses. In Proceedings of the Eleventh International Workshop on Variability Modelling of Software-intensive Systems. 44–51.

[12] Robert M. Hierons, Miqing Li, Xiaohui Liu, Sergio Segura, and Wei Zheng. 2016. SIP: Optimal Product Selection from Feature Models Using Many-Objective Evolutionary Optimization. ACM Trans. Softw. Eng. Methodol. 25, 2, Article 17 (April 2016), 39 pages.

[13] José-Miguel Horcas, José A Galindo, Ruben Heradio, David Fernandez-Amoros, and David Benavides. 2021. Monte Carlo tree search for feature model analyses: a general framework for decision-making. In Proceedings of the 25th ACM International Systems and Software Product Line Conference-Volume A. 190–201.

[14] José-Miguel Horcas, Mónica Pinto, and Lidia Fuentes. 2019. Context-aware energy-efficient applications for cyber-physical systems. Ad Hoc Networks 82 (2019), 15–30.

[15] Christian Kaltenecker, Alexander Grebhahn, Norbert Siegmund, and Sven Apel. 2020. The interplay of sampling and machine learning for software performance prediction. IEEE Software 37 (2020), 58–66. Issue 4.

[16] Christian Kaltenecker, Stefan Mühlbauer, Alexander Grebhahn, Norbert Siegmund, and Sven Apel. 2023. Performance evolution of configurable software systems: an empirical study. Empirical Software Engineering 28, 6 (2023), 152.

[17] Kashif Nizam Khan, Mikael Hirki, Tapio Niemi, Jukka K. Nurminen, and Zhonghong Ou. 2018. RAPL in Action: Experiences in Using RAPL for Power Measurements. ACM Trans. Model. Perform. Eval. Comput. Syst. 3, 2, Article 9 (mar 2018), 26 pages.

[18] Sergiy Kolesnikov, Norbert Siegmund, Christian Kästner, Alexander Grebhahn, and Sven Apel. 2019. Tradeoffs in modeling performance of highly configurable software systems. Software & Systems Modeling 18 (2019), 2265–2283.

[19] Andreas Metzger, Clément Quinton, Zoltán Ádám Mann, Luciano Baresi, and Klaus Pohl. 2024. Realizing self-adaptive systems via online reinforcement learning and feature-model-guided exploration. Computing 106, 4 (2024), 1251–1272.

[20] Daniel-Jesus Munoz, Mónica Pinto, and Lidia Fuentes. 2022. Quality-Aware Analysis and Optimisation of Virtual Network Functions. In Proceedings of the 26th ACM International Systems and Software Product Line Conference - Volume A

(Graz, Austria) (SPLC '22). Association for Computing Machinery, New York, NY, USA, 210–221.

[21] Stefan Mühlbauer, Florian Sattler, Christian Kaltenecker, Johannes Dorn, Sven Apel, and Norbert Siegmund. 2023. Analysing the Impact of Workloads on Modeling the Performance of Configurable Software Systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 2085–2097.

[22] Vivek Nair, Zhe Yu, Tim Menzies, Norbert Siegmund, and Sven Apel. 2020. Finding Faster Configurations Using FLASH. IEEE Transactions on Software Engineering 46, 7 (2020), 794–811.

[23] Rafael Olaechea, Steven Stewart, Krzysztof Czarnecki, and Derek Rayside. 2012. Modelling and Multi-Objective Optimization of Quality Attributes in Variability-Rich Software. In Proceedings of the fourth international workshop on nonfunctional system properties in domain specific modeling languages (NFPinDSML '12). Article 2, 6 pages.

[24] Zakaria Ournani, Romain Rouvoy, Pierre Rust, and Joel Penhoat. 2020. On reducing the energy consumption of software: From hurdles to requirements. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–12.

[25] Juliana Alves Pereira, Mathieu Acher, Hugo Martin, Jean-Marc Jézéquel, Goetz Botterweck, and Anthony Ventresque. 2021. Learning software configuration spaces: A systematic literature review. Journal of Systems and Software 182 (2021), 111044.

[26] Juliana Alves Pereira, Pawel Matuszyk, Sebastian Krieter, Myra Spiliopoulou, and Gunter Saake. 2016. A feature-based personalized recommender system for product-line configuration. In Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences. 120–131.

[27] Norbert Siegmund, Johannes Dorn, Max Weber, Christian Kaltenecker, and Sven Apel. 2022. Green Configuration: Can Artificial Intelligence Help Reduce Energy Consumption of Configurable Software Systems? Computer 55, 3 (2022), 74–81.

[28] Norbert Siegmund, Alexander Grebhahn, Sven Apel, and Christian Kästner. 2015. Performance-Influence Models for Highly Configurable Systems. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). 284–294.

[29] Norbert Siegmund, Sergiy S Kolesnikov, Christian Kästner, Sven Apel, Don Batory, Marko Rosenmüller, and Gunter Saake. 2012. Predicting Performance via Automated Feature-Interaction Detection, In 2012 34th International Conference on Software Engineering (ICSE) (Zurich, Switzerland). Proceedings of the 34th International Conference on Software Engineering, 167–177.

[30] Norbert Siegmund, Marko Rosenmüller, Martin Kuhlemann, Christian Kästner, Sven Apel, and Gunter Saake. 2012. SPL Conqueror: Toward optimization of non-functional properties in software product lines. Software Quality Journal 20 (2012), 487–517. Issue 3.

[31] Norbert Siegmund, Marko Rosenmüller, Christian Kästner, Paolo G Giarrusso, Sven Apel, and Sergiy S Kolesnikov. 2013. Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption. Information and Software Technology 55 (2013), 491–507. Issue 3.

[32] Yewan Wang, David Nörtershäuser, Stéphane Le Masson, and Jean-Marc Menaud. 2018. Potential effects on server power metering and modeling. Wireless Networks (2018), 1–8.

[33] Max Weber, Christian Kaltenecker, Florian Sattler, Sven Apel, and Norbert Siegmund. 2023. Twins or False Friends? A Study on Energy Consumption and Performance of Configurable Software. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 2098–2110.

[34] Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker. 2015. Hey, You Have given Me Too Many Knobs!: Understanding and Dealing with over-Designed Configuration in System Software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 307–319.

[35] Sai Zhang and Michael D. Ernst. 2014. Which Configuration Option Should I Change?. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 152–163.

[36] Ivan Švogor, Ivica Crnković, and Neven Vrček. 2019. An extensible framework for software configuration optimization on heterogeneous computing systems: Time and energy case study. Information and Software Technology 105 (2019), 30–42.