# Measure What Matters: A Dual Outcome Service Quality Model for Government Service Delivery

Willem Pieterson, Cindy Weng

► To cite this version:

Willem Pieterson, Cindy Weng. Measure What Matters: A Dual Outcome Service Quality Model for Government Service Delivery. 19th International Conference on Electronic Government (EGOV), Aug 2020, Linköping, Sweden. pp.138-150, 10.1007/978-3-030-57599-1_11. hal-03282778

HAL Id: hal-03282778

https://inria.hal.science/hal-03282778

Submitted on 9 Jul 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Measure what matters.

## A dual outcome service quality model for government service delivery

Willem Pieterson<fnref n="1" /> and Cindy Weng<fnref n="2" />

**Abstract.** Measuring customer service quality evaluations has been important since the rise of the service industry and many models in this area have been published. Most models focus on one outcome with a set of predictors. These outcomes are often ill defined and concepts are used interchangeably causing issues in creating good and consistent measures of quality. In this study we develop a new model combining multiple outcome variables and a series of predictors to show the interdependent nature of service outcomes. We test the model using machine learning based on survey responses from 3702 Dutch people. The results indicate that two types of outcome variables are important; quality of the outcome and satisfaction with the process. Each is predicted in different ways by four dimensions. This means governments could benefit from a better specification of the desired outcomes of service delivery and targeted measurement approaches.

**Keywords:** Service quality, service delivery, machine learning, gradient boosted decision trees

## 1 Introduction

Service delivery lies at the heart of the missions of many government agencies around the world and naturally governments have an interest in ensuring high quality service delivery. Finding an answer to what variables influence the quality of service delivery has been a prominent question in the services literature since the 1980s. During this time the first models defining outcomes of service delivery and possible determinants were published. SERVQUAL [1] is a prominent example. Soon different models appeared (e.g. SERVPERF [2]) arguing for the importance of other outcome variables.

Technological advancements (e.g. diffusion of the Internet) led to new types of models often used in this context, such as the technology acceptance model [3]. This focus on technology spawned new models geared towards electronic (government) service delivery, such as ‘eGovQual’ [4] and the model for evaluation of eGov services [5].

The majority of these models have in common that they use one outcome variable plus a series of determining variables. However, they vary in their choice of outcome

<footnote n="1">¹ Center for eGovernment Studies / Pieterson Strategic, London, United Kingdom
willem@pieterson.com</footnote>

<footnote n="2">² Microsoft Cloud Solutions, London, United Kingdom
cindy.weng@microsoft.com</footnote>
variable and often similar outcome variables are explained and/or measured slightly different. This trickles down to more practical research. For example, the bi-annual 'Citizen First' studies in Canada focus on 'satisfaction' as an outcome measure [6], while similar (annual) studies in the Netherlands used 'appreciation' [7]. This creates confusion about what outcomes are most important, as well as how to define and measure them. This creates challenges for researchers as it is ambiguous how measures should be defined and operationalized as well as practitioners who lack clear guidance on how to measure their service quality and create actionable outcomes. Furthermore, despite the creation of newer models, some of the older models are still often used raising questions about their usefulness (given their age) as well as the usefulness of newer models, given the enduring popularity of models such as SERVQUAL.

In this paper we investigate the status quo of service quality research and rather than testing existing models or creating a new theoretical model based on the existing literature, we decided to adopt the users' perspective and create and test a new model focused on service delivery outcomes and the main factors determining these outcomes.

## 1.1 Overview of the paper

The remainder of the paper is organized as follows. We first discuss the existing models and research on service quality in more detail. Subsequently, in section 3, we present the method of our study, followed by the results of our empirical work (section 4). We close the paper with our conclusions and points of discussion and recommendations.

# 2 Service quality in the literature

The topic of 'service quality' and how to measure it rose to prominence during the 1980s [1]. During this time, when the services industry came to maturity, the first models were developed that would describe certain outcomes of the service delivery process, as well as variables determining or predicting these outcomes. The most prominent example of these is SERVQUAL [1] which was published in 1988. This model suggests that the (perceived) quality of service delivery depends on five independent variables: reliability, assurance, tangibles, empathy, and responsiveness. Soon after, critiques on the model developed. A key argument is that 'quality' is not the dominant outcome of service delivery processes, but rather other variables, such as 'performance'. This led to models such as SERVPERF [2]. While developed in the context of private sector service delivery, these models were soon also used to study the quality of public sector service delivery [e.g. 8, 9]. Many of these older (competing) models are still used and cited in the literature on governmental service delivery [e.g. 10–12].

As these models were being adopted in the public sector, the services landscape itself started changing. The arrival of new service channels, fueled by the diffusion of the internet, led to new opportunities of both private and public sector agencies to improve the quality of their service delivery [13]. This led to two developments in studying the quality of service delivery in the public sector. The first is the study of the adoption of
these new service channels and the link between adoption models, such as the Technol-
ogy Acceptance Model [3] and service quality, for example by correlating variables
such as ‘ease of use’ of new channels with service quality [14, 15].

The second development is the creation of new, dedicated models that are targeted
towards the quality of online service delivery online. An example of a generic model
of this type is the E-S-Qual model [16], presented as an ‘electronic’ alternative to the
‘traditional’ SERVQUAL model. However, a breed of specific models for the govern-
ment and/or public sector arrived quickly. Examples are ‘eGovQual’ [4] and the model
for evaluation of eGov services [5], which, while both mentioning quality present dif-
ferent outcome variables of service delivery (in the context of service quality).

This, in sum, leads to a situation in which we can identify service shortcomings of
the existing models that focus on service quality. The first is that many models exist
that focus on different aspects of service delivery and these models all use different
predictor variables that are often derived from other types of models, such as those
focusing on adoption or acceptance of technology. Second, by creating the split be-
tween quality of online or electronic services versus quality of traditional services, these
models ignore the fact that on- and offline service channels are often used in ‘multiplex’
processes of services delivery, i.e. those situations in which citizens use multiple chan-
nels either in parallel or sequentially [17]. Third, many of the ‘older’ models, while
being critiqued, are still being used frequently, despite these shortcomings. Most mod-
els are never updated to reflect changing times. Fourth, while a multitude of models
exist and are often cited, many exist only in theoretical form and lack (repeated) empir-
ical testing (eGovQual is one example). Fourth, many models exist that focus on ‘qual-
ity’ of service delivery, definitions of quality vary and wildly varying measures for
quality exist. Fifth, and more generally, the world of government service delivery is
evolving rapidly. Many new service channels arrive on the scene frequently [18] and
focal points of government service delivery shift rapidly. Variables such as privacy
protection that feature heavily in many government service strategies nowadays (e.g.
the EU’s Talinn declaration) played hardly any role when most models were developed.

## 2.1 Research questions

The goal of our study is to develop a new service ‘quality’ model that links the key
outcomes of government service delivery to (predictor) variables that are relevant from
the perspective of the recipients of governmental service delivery: citizens and busi-
nesses (clients) in the Netherlands. The following research questions guide the study:

RQ1. What are the main outcome variables of government service delivery that cap-
ture the quality of service delivery from the perspective of clients?

RQ2. What are the main variables influencing or predicting these outcomes?

RQ3. How well does this model perform empirically?
# 3 Method

In order to answer our research questions, we followed an inductive two-step research approach [19], using mixed methods. In our case this approach combines qualitative and quantitative techniques. A benefit of this combination is that it leads to a more complete understanding of a research problem [20]. We collected data for our study through an exploratory sequential mixed method approach in which we first conducted a qualitative study and based on the outcomes of that study developed our research model and instrument. These were subsequently used in the quantitative study.

## 3.1 Research approach & setting

The study took place in the Netherlands, a country with both high levels of internet adoption and online service maturity (13th in the 2018 UN eGovernment ranking [21]). The country launched a digital strategy in 2018 ("NL DIGIBeter) which focuses on the role of technology in the Dutch society as a whole. It contains specific aims towards service delivery, such as improving accessibility, understandability of information and personalization of service delivery. The government body coordinating the program is the Ministry of the Interior. This organization asked for this study to be conducted in the context of monitoring success of the digital strategy. The study focused on governmental service delivery in general and was aimed at the entire population using government services (anyone aged 16 or higher), including citizens and businesses.

A research agency was involved in the study and they recruited participants, facilitated the qualitative study and collected data for the quantitative study. The authors were in charge of instrument development, supervision of data collection, and analyses. Both parts of the study included other topics as well.

## 3.2 Qualitative study

To gather insights in what clients perceive to be important variables and elements of public service quality, we decided to hold a series of focus groups. These focus groups are not merely a convenient way to gather insights from individual participants, but they “give rise synergistically to insights and solutions that would not come about without them” [22, p. 40]. A number between three and six groups is normally considered sufficient and groups should have between 7-10 participants [23].

We held five focus groups with a total number of 38 participants. Four of these focus groups consisted of citizens and one of representatives of small and medium businesses. The four citizen groups were split in two based on level of education (low/high). Within the groups we controlled for gender and age. The focus groups were held in April/May 2019 at various locations in the Netherlands to allow for geographical spread.

All focus groups were recorded (both audio and video) and notes were taken during the focus groups. The notes served as main source for the outcomes and the recordings served as reference material. As mentioned above, the interviews focused on many topics and in this paper the results of two interview questions are presented: a) what, in your eyes, determines whether or not service delivery is good (what are key results,
what is important in your eyes, when are you satisfied and is satisfaction the only im-
portant outcome)? And b) what are key elements that impact these outcomes (what is
important to you, what aspects related to the results or the process of service delivery
can you think of and how important are these)? A list of potential variables derived
from models discussed were used as examples of variables to stimulate the discussion.

3.3 Quantitative study

Based on the outcomes of the qualitative study, a model and survey instrument were
developed to test the relationships between outcomes of service delivery and relevant
predictor variables. The instrument contained more questions and focused on several
aspects of service delivery. For the purposes of this study, we focus on those questions
targeting outcomes and predictors of service delivery. The instrument was pre-tested
with 9 people to check for errors and interpretation issues and subsequently finalized.

Subsequently, the questionnaire was programmed and distributed among members
of the research agency's online panel. The structure of the questionnaire was such that
respondents were asked to indicate a) whether they had used government services in
the past 12 months and b) which type of services they had used. Respondents were
filtered based on these experiences to create a sample with relatively even distribution
of users across the spectrum of government services (and types of agencies). Thus, the
goal was not to get full and even representation across the (demographic groups in the)
population, but rather to reflect use of government services. After three weeks of data
collection, a total number of N=3702 complete responses were recorded, of which
n=3120 were citizens and n=582 were representatives of businesses.

The data were analyzed in SPSS and Python (using PyCharm), and a number of
analyses were done. First, we clustered the predictor variables into a number of dimen-
sions using factor analysis and reliability analyses of the derived constructs. Second,
we created a number of linear regression models that were tested in SPSS. While such
regression models are a good way to gain insights into linear causal relationships be-
tween variables, there are a number of issues with this approach in this context. The
first is that it assumes (relative) independence of the variables involved and collinearity
can cause issues [24]. In this case, it is likely that the different predictors (and outcomes)
of service quality co-vary and could be dependent. Furthermore, it assumes linearity of
the variables [25], which may not always be the case. Lastly, it is sensitive for sample
size (e.g. on producing biased R-squares) and the effects of error on the model, as well
as assuming exactly what (theoretical) relationships to expect in the model.

New types of analytics based on machine learning offer benefits over regression
models. These data-driven approaches infer relationships by finding the intrinsic splits
in the data, without relying on imposed relationships from theory. They use training
approaches to assess the relevance of these relationships [26]. In our situation, we chose
to apply gradient boosting to a decision tree model. The main reason is the relative
simplicity of the approach, coupled with the success of this approach in yielding robust
outcomes [26]. Furthermore, they are fast, input-agnostic, and work well in regression
situations [27]. We trained the model with a subsample of 1200 cases, keeping the re-
maining number of cases to validate the model. Model (split) scores were converted to
percentages of contribution to show their relative weight and the random mean square errors, and accuracy of the models was calculated to assess model fit.

# 4 Results

The results section consists of two parts. The first discusses the key outcomes from the qualitative focus groups. The second presents outcomes from the quantitative part.

## 4.1 Qualitative study

The qualitative study, as mentioned above focused on two key questions: 1) the question of what important outcomes of service delivery are and 2) what influences these outcomes. The first question yielded a lively discussion with an inconclusive outcome in terms of whether ‘quality’ is the most important outcome and what this quality exactly is supposed to mean. The discussion did suggest that two components are relevant to the final result, these are:

* The experience with the process of service delivery. This points to the way in which the process is designed and handled by the government agency and includes such variables such as *the friendliness of customer service agents, the ease with which information can be found and forms can be completed, the design of websites*. This has very little to do with the outcomes of the process. Several participants mention that this ‘experience’ could still lead to a positive service evaluation, even if the outcome is not the desired outcome (e.g. not getting a permit or benefit). We label this “**Satisfaction with the Process**” (Satisfaction Process).

* Besides the experience with the process, the final evaluation is tied to the actual outcome of the service delivery process. That has less to do with whether the result is what clients desire, but more with whether the results is correct, just, and fair. This includes such aspects as the *correctness of information, getting a response within (legal) limits and getting an answer to a question*. We label this variable the “**Quality of the Outcome**” (Quality Outcome).

Subsequently we asked about variables that influence these outcomes. The initial discussions (in which participants were asked to write down what they feel is important and then these were discussed in the group) yielded a list of 20 different aspects. We then asked participants in the different sessions to rate the importance of these aspects and discuss the outcomes. This resulted in a total of 18 different variables that were used in the questionnaire. The table below gives an overview of these variables

<table><caption>**Table 1.** Variable overview</caption><thead><tr><td>#</td><td>Name</td><td>Description</td><td>#</td><td>Name</td><td>Description</td></tr></thead><tbody><tr><td>1</td><td>Privacy</td><td>Whether privacy is well protected</td><td>10</td><td>Comprehension</td><td>Whether information is easy to comprehend</td></tr><tr><td>2</td><td>Security</td><td>Degree to which services are secure</td><td>11</td><td>Accessibility</td><td>Whether services are accessible to the user</td></tr></tbody></table>
<table><tr><td>3 Data control</td><td>Whether people have control over their own data when using government services</td><td>12 User Friendly</td><td>Whether services are designed to fit user needs</td></tr><tr><td>4 Trust</td><td>Trust levels in government and government services</td><td>13 Simplicity</td><td>Language and design are simple</td></tr><tr><td>5 Transparency</td><td>Degree to which there is transparency in the service delivery process</td><td>14 Pro-Active</td><td>Government service delivery is pro-active</td></tr><tr><td>6 Supportive</td><td>Whether people can get (customer) support from governments when needed</td><td>15 Personal</td><td>Service delivery is personalized</td></tr><tr><td>7 Responsibility</td><td>Degree to which governments take responsibility over delivering high quality services.</td><td>16 Speed</td><td>Service delivery is fast</td></tr><tr><td>8 Solution</td><td>Whether governments provide solutions when problems occur</td><td>17 Ease of Use</td><td>Government services are easy to use</td></tr><tr><td>9 Findability</td><td>Degree to which information and services can be found easily</td><td>18 Freedom Choice</td><td>of Users can choose how to obtain services.</td></tr></table>

In addition, the discussions suggested that *citizens* have different expectations of service delivery than *businesses*, suggesting clients' *role* influences outcomes. Furthermore, we observed possible differences between those users with much online experiences and use of technology in general (*media use*) and lastly, participants with different demographic characteristics (most notably *age*) appeared to value different aspects. Leading to the inclusion of *demography*, *media use* and *role* in the quantitative study.

## 4.2 Quantitative study

The first step in the quantitative analysis was to run a factor analysis to see whether the 18 variables can be grouped in fewer dimensions. A varimax rotated factor analysis suggests a four-factor solution that explains 72.30% of the variance in the data. Subsequently, we ran a reliability test on these four dimensions to test for their internal coherence. Cronbach alpha scores for each dimension range from α=0.83 to α=0.92, which is (much) higher than the commonly accepted threshold of α=0.65. **Table 2.** gives an overview of the dimensions and scores.

Based on the content of the dimensions, we created appropriate labels for these dimensions: 1) Security & Trust: this comprises those variables that relate to security, privacy, trust and control; 2) Help & Support: this entails those variables that focus on how governments support users; 3) Accessibility: this covers those variables that determine whether or not people can successfully use government services; 4) Service Design: this includes all variables that impact the user experience of service delivery.

<table><caption>**Table 2. Reliability of dimensions**</caption><thead><tr><th rowspan="2">Variable</th><th colspan="6">Dimensions</th></tr><tr><th>Security & Trust</th><th>Help & Support</th><th>Accessibility</th><th>Service Design</th><th>Security & Trust</th><th>Help & Support</th><th>Accessibility</th><th>Service Design</th></tr><tr><th></th><th>Scale if deleted</th><th>Variable</th><th>Scale if deleted</th><th>Variable</th><th>Scale if deleted</th><th>Variable</th><th>Scale if deleted</th><th>Security & Trust</th></tr></thead><tbody><tr><td>Privacy</td><td>0.83</td><td>Supportive</td><td>0.76</td><td>Findability</td><td>0.77</td><td>Pro-Active</td><td>0.66</td><td>Security</td></tr><tr><td>Security</td><td>0.84</td><td>Responsibility</td><td>0.67</td><td>Comprehension</td><td>0.79</td><td>Personal</td><td>0.56</td><td>Security</td></tr></tbody></table>
<table><tr><td>Data control</td><td>0.83</td><td>Solution</td><td>0.76</td><td>Accessibility</td><td>0.83</td><td>Speed</td><td>0.62</td></tr><tr><td>Trust</td><td>0.84</td><td></td><td></td><td>User</td><td>0.80</td><td>Ease of Use</td><td>0.64</td></tr><tr><td>Transparency</td><td>0.84</td><td></td><td></td><td>Friendly</td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td>Simplicity</td><td>0.77</td><td>Freedom of Choice</td><td>0.66</td></tr><tr><td colspan="8" style="text-align: center; font-style: italic;">Cronbach's alpha for dimensions</td></tr><tr><td>α=0.86</td><td></td><td>α=0.86</td><td></td><td>α=0.92</td><td></td><td>α=0.83</td><td></td></tr></table>

The reduction of the number of variables into four dimensions allows us to create the research model to be tested (Fig. 1). The first step of these tests was to run a series of two linear regression models. In the first model we used Quality of Outcomes as the dependent variable and Satisfaction with the Process, the four dimensions, and the personal characteristics as independent variables. In the second model, we swap Quality of Outcomes (now independent) and Satisfaction with the Process (now dependent). For both models we calculate the R-Square, F-score and P value as main parameters and subsequently the regression coefficients as estimates. Both models show high R-squares and have significant fit (Table 3), suggesting that the independent variables explain the dependent variables well.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Research Model</figcaption></figure>

<table><caption>Table 3. Regression model parameters</caption><thead><tr><th></th><th>Quality outcomes</th><th>Satisfaction process</th></tr></thead><tbody><tr><td>R square</td><td>0.72</td><td>0.73</td></tr><tr><td>F</td><td>907.78</td><td>954.14</td></tr><tr><td>Sig</td><td>.000</td><td>.000</td></tr></tbody></table>

Subsequently, the model estimates (Table 4) show that the Quality of Outcomes is predicted most strongly by the Satisfaction with the Process (β=0.74, p<.000) and the other way around (β=0.72, p<.000) suggesting a strong relationship between the two outcome variables. Furthermore, in both models do we find significant effects of the four dimensions on the dependent variables, but the strength of the coefficients varies between the two models. The role of personal characteristics is small (that of role is so small that it has been omitted from the table and subsequent analyses). We only find a small effect of media use on Satisfaction with the Process (β=-0.02, p=.017) and Age on the same dependent (β=0.03, p<.008).

<table><caption>Table 4. Regression model estimates</caption><thead><tr><th rowspan="2">Variables</th><th colspan="2">Quality outcomes</th><th colspan="2">Satisfaction process</th></tr><tr><th>Bèta</th><th>Sig</th><th>Bèta</th><th>Sig</th></tr></thead><tbody><tr><td>(constant)</td><td>.000</td><td>.022*</td><td>0.00</td><td>0.175</td></tr></tbody></table>
<table><tr><td>Quality outcomes</td><td>-</td><td>0.72</td><td>0.000**</td></tr><tr><td>Satisfaction process</td><td>0.74</td><td>0.000**</td><td>-</td></tr><tr><td>Dim 1. Security &amp; trust</td><td>0.07</td><td>0.000**</td><td>0.04</td><td>0.001**</td></tr><tr><td>Dim 2. Help &amp; support</td><td>0.04</td><td>0.001**</td><td>0.06</td><td>0.000**</td></tr><tr><td>Dim 3. Accessibility</td><td>0.04</td><td>0.001**</td><td>0.09</td><td>0.000**</td></tr><tr><td>Dim 4. Service design</td><td>0.03</td><td>0.008**</td><td>0.06</td><td>0.000**</td></tr><tr><td>Media use</td><td>-0.01</td><td>0.206</td><td>-0.02</td><td>0.017*</td></tr><tr><td>Age</td><td>0.01</td><td>0.565</td><td>0.03</td><td>0.008*</td></tr><tr><td>Education</td><td>-0.01</td><td>0.339</td><td>0.00</td><td>0.894</td></tr><tr><td>Gender</td><td>-0.02</td><td>0.071</td><td>0.01</td><td>0.164</td></tr></table>

The last step in the analyses was the calculation of the parameters for the Gradient Boosted Decision Tree (GBDT) model as well as the model contributions. For both models, we yield similar RMSEs between the training and the testing models and for both models the testing RMSEs (shown in **Table 5**) are close to 1, indicating a good model fit. Furthermore, the accuracy of both models is fairly high (62.5% for Quality of Outcomes and 64.34% for Satisfaction with the Process). In line with the regression models, this confirms that the independent variables predict the dependents well.

<table><caption>**Table 5.** Gradient Boosted Decision Tree model parameters</caption><thead><tr><th></th><th>Quality outcomes</th><th>Satisfaction process</th></tr></thead><tbody><tr><td>Accuracy</td><td>62.52%</td><td>64.34%</td></tr><tr><td>Root Mean Squared Error (RMSE)</td><td>0.994667</td><td>0.977235</td></tr></tbody></table>

The model contributions (Fig. 2) follow that line. We see that the largest model contributions are those of the other outcome variables, confirming that high quality outcomes lead to more satisfaction with the process and vice versa. However, the GBDT results show much bigger model contributions of the four dimensions and it highlights the differences between the two models.

It appears that Quality of Outcomes is determined (in ascending order) by variables related to Security & Trust (Dim1), Service Design (Dim4), Help & Support (Dim2) and Accessibility (Dim3). That order is different for Satisfaction with the Process (Dim4, Dim3, Dim2, Dim1, respectively). In both models we find a moderate effect of Age and a

smaller effect of Media Use. The other variables play a much smaller role.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Relative model contributions</figcaption></figure>

<footnote>*Significant at p=0.05, **Significant at p=0.01</footnote>
# 5 Conclusions and Discussion

This empirical study among 3,702 citizens and businesses in the Netherlands was aimed at the development and test of a new service quality model for governmental service delivery. Three research questions guided this aim. The first research question (RQ1) asked about the main outcome variables that capture the quality of service delivery from clients' perspectives. Our research suggests two main variables are important:

* Quality of Outcomes. This refers to the degree to which clients receive an outcomes that is of high quality, i.e. free of errors, according to the norms, fair and just.

* Satisfaction with the Process. Whether clients are satisfied with the service delivery process, i.e. friendly staff, user friendly forms, well-designed websites.

These variables are not independent: a high level of satisfaction with the process will lead to a more positive evaluation of the quality of outcomes and vice versa. This highlights the importance of designing processes and the effect of the ‘customer experience’ on the outcomes of the process. Furthermore, it highlights how a focus on *quality* of outcomes can have a positive effect on the evaluation of the process as a whole.

RQ2 focused on the variables influencing these outcomes. We identified a total of 18 different variables that clustered in four different dimensions. All of these dimensions have a significant impact on the outcome variables across the different types of models we tested. However, we do find in our GBDT that the impact of the dimensions on the outcomes varies. For example we see a strong effect of ‘security & trust’ on the quality of outcomes and we find the strongest effect of ‘service design’ on the satisfaction with the outcomes. Security and trust is the least important dimension for satisfaction with the outcomes. This suggests clear differences between how different variables and dimensions contribute to certain outcomes of service delivery.

In general, the role of personal characteristics is less important. Age has some impact as well as the degree to which (online) media are used. This probably relates to peoples' digital skills and experiences in using (online) channels for service delivery [28].

The last research question (RQ3) concerns the performance of our model(s). In both tests they perform well. We find good model fit measures, as well as good r-squares and accuracies. These measures are by no means perfect, but do provide an excellent starting point for future iterations of the model.

In sum, in tying these back to our aim, we have created a new series of models to measure the outcomes of government service delivery based on the inputs of citizens and businesses in the Netherlands, and have tested this model empirically using a large sample in the Netherlands using several analytical approaches. The models yield significant results. This results in several implications for practitioners and researchers.

From a practical standpoint, our research shows that clients typically discriminate between two types of outcomes or measures of ‘quality’. This is in contrast to more common approaches in which one outcomes variable is specified and often measured using wildly differing measures. Our research shows, these outcome measures are different and are determined by different variables. The implication is that practitioners need to be aware of the outcomes they want to measure and ensure the questions they ask match these outcomes. Second, our research shows that different types of variables influence different outcomes. For example, service design has a strong influence on the
satisfaction with the process. Security and trust related variables influence the quality of the outcome. This means that governments need to be aware of the variables their policies influence and how these relate to outcomes. In other words: we give input in the types of levers governments have to improve certain types of outcomes. Lastly, while our research shows that the role of personal characteristics is relatively small, age and media use do have a role, suggesting that digital divide related variables remain important when considering the quality of service delivery as a whole.

For researchers and as points of discussion, we hope that our study gives further impetus to the development of new service quality models that combine different types of outcomes with multiple input variables. We find the determining variables to be quite different from those used in previous models and this suggests not only that a) the older, still often used, models have an 'expiration date', but b) moreover that newer models need to evolve and constantly be updated to reflect the evolving service delivery landscape. Second, while our model uses a large number of variables, the results show that we are not able to explain all variance in the data or accurately predict all outcomes. Thus, more or different variables could be included to improve these outcomes. Third, we tested the model once, in one country. Replication and validation in other settings could help strengthening this work. Lastly, we used an online research panel, inevitably meaning that people without online access were not included, thus we miss out on an important part of the population. This needs to be rectified in future work.

**Acknowledgement.** We would like to thank the Ministry of the Interior for funding this work. The Ministry of the Interior had no role in the preparation of this publication.

## References

1. Parasuraman, a, Zeithaml, V. a, Berry, L.L.: SERQUAL: A Multiple-Item scale for Measuring Consumer Perceptions of Service Quality, (1988). https://doi.org/10.1016/S0148-2963(99)00084-3.

2. Cronin, J.J., Taylor, S.A.: Measuring Service Quality: A Reexamination and Extension. J. Mark. (1992). https://doi.org/10.2307/1252296.

3. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. Manag. Inf. Syst. (1989). https://doi.org/10.2307/249008.

4. Papadomichelaki, X., Mentzas, G.: e-GovQual: A multiple-item scale for assessing e-government service quality. Gov. Inf. Q. 29, 98–109 (2012). https://doi.org/10.1016/j.giq.2011.08.011.

5. Alanezi, M., Kamil, A., Basri, S.: A proposed instrument dimensions for measuring e-government service quality. Int. J. u-and e-Service. (2010).

6. Institute for Citizen-Centered Service: Citizen First 2018. , Ontario (2018).

7. Research, I.: De kwaliteit van de overheidsdienstverlening 2015. (2016).

8. Donnelly, M., Dalrymple, J.F., Wisniewski, M., Curry, A.C.: The portability of the SERVQUAL scale to the public sector. In: Total Quality Management (1995). https://doi.org/10.1007/978-94-011-0539-2_39.

9. Shieff, D.: Service Quality Components and Group Criteria in Local Government. Int. J. Serv. Ind. Manag. (1993). https://doi.org/10.1108/09564239310044280.
10. Melo, A.I., Goncalo, S., Lima, R.: Measuring the Quality of Health Services Using SERVQUAL: Evidence From Portugal. In: Handbook of Research on Modernization and Accountability in Public Sector Management. pp. 300–3187. IGI Global (2018).

11. Setiajit, D.G., Utomo, A.: A Servqual Measurement of Public Service from Motor Vehicle Taxation Office (SAMSAT) In Indonesia. J. Karya Dosen ITN Malang. April, (2017).

12. Ocampo, L., Alinsub, J., Casul, R.A., Enquig, G., Luar, M., Panuncillon, M., B., Ocampo, C.O.: Public service quality evaluation with SERVQUAL and AHP-TOPSIS: A case of Philippine government agencies. Socioecon. Plann. Sci. 68, (2019).

13. van Deursen, A.J.A.M., Pieterson, W.: The Internet as a service channel in the public sector. In: ICA Conference., Dresden, Germany (2006).

14. Nguyen, M.H.: A Study on Evaluation of E-Government Service Quality. Int. J. Soc. Manag. Econ. Business, Eng. (2014).

15. Sang, S., Lee, J.D.: A conceptual model of e-Government acceptance in public sector. In: Proceedings of the 3rd International Conference on Digital Society, ICDS 2009 (2009). https://doi.org/10.1109/ICDS.2009.30.

16. Parasuraman, A., Zeithaml, V.A., Malhotra, A.: E-S-QUAL a multiple-item scale for assessing electronic service quality. J. Serv. Res. (2005). https://doi.org/10.1177/1094670504271156.

17. Madsen, C.Ø., Hofmann, S., Pieterson, W.: Channel Choice Complications: Exploring the Multiplex Nature of Citizens’ Channel Choices. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 11685 LNCS, 139–151 (2019). https://doi.org/10.1007/978-3-030-27325-5_11.

18. Pieterson, W., Ebbers, W.E., Madsen, C.Ø.: New Channels, New Possibilities: A Typology and Classification of Social Robots and Their Role in Multi-channel Public Service Delivery, (2017). https://doi.org/10.1007/978-3-319-64677-0_5.

19. Blaikie, N.: Approaches to social inquiry. Polity (2010).

20. Creswell, J.W.: Research Design. Qualitative, Quantitative, and Mixed Methods Approaches. SAGE, London, UK (2014).

21. United Nations: E-Government Survey 2018. (2018). https://doi.org/e-ISBN: 978-92-1-055353-7.

22. Brown, J.S., Collins, A., Duguid, P.: Situated Cognition and the Culture of Learning. Educ. Res. (1989). https://doi.org/10.3102/0013189X018001032.

23. Krueger, R. A., Casey, M.A.: Planning the Focus Group Study. In: Focus Groups A Practical Guide for Applied Research (2015).

24. Wold, S., Ruhe, A., Wold, H., Dunn, III, W.J.: The Collinearity Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses. SIAM J. Sci. Stat. Comput. (1984). https://doi.org/10.1137/0905052.

25. Wetherill, G.B., Seber, G.A.F.: Linear Regression Analysis. J. R. Stat. Soc. Ser. A. (1977). https://doi.org/10.2307/2345290.

26. Natekin, A., Knoll, A.: Gradient boosting machines, a tutorial. Front. Neurorobot. (2013). https://doi.org/10.3389/fnbot.2013.00021.

27. Hastie, T., Tibshirani, R., Friedman, J.: Springer Series in Statistics. (2009). https://doi.org/10.1007/b94608.

28. Ebbers, W.E., Jansen, M.G.M., Pieterson, W.J., van de Wijngaert, L.A.L.: Facts and feelings: The role of rational and irrational factors in citizens’ channel choices. Gov. Inf. Q. 33, 506–515 (2016). https://doi.org/10.1016/j.giq.2016.06.001.