# Identifying Usability and Productivity Dimensions for Measuring the Success of Mobile Business Services

Authors: Maiju Vuolle; Anne Aula; Minna Kulju; Teija Vainio; Heli Wigelius
Journal: Advances in Human-Computer Interaction
 (2008)

---

## Abstract

This paper reviews existing measures used for evaluating the usability of information systems and those used for evaluating the level of the productivity of a company. We discuss the usefulness of the existing measures in the mobile business context, where both mobility and work-context pose specific demands for the mobile business services. The review showed that the existing measures rarely consider the great contextual variation caused by the mobility of the services and the demands this poses on usability; which, in turn, affects productivity. To build a measurement tool that better meets the requirements of mobile business services, we completed case studies on two mobile business services, one used in passenger transport and the other in construction sites. Based on the understanding gained from the case studies, we propose a list of dimensions and items addressing both usability and productivity aspects that work as the basis for a multidisciplinary measurement tool.


---

## Body

## 1. Introduction

Mobile business services have great potential for benefiting both the
individual workers and the companies deploying them. These business-oriented
services are used via mobile devices (e.g., mobile phone or PDA), for example,
to access corporate information systems and the Internet or for collecting data
in the field (e.g., [1, 2]). Both the companies developing and deploying these
services would benefit from a tool that measures their success. This tool could
be used for benchmarking and evaluating the new versions of the service, for
evaluating the success of the service against competitors, and for evaluating
the possible benefits the service could provide for the end users and companies
deploying the service.Despite the potential
benefits, it remains a challenge to measure how successful the services
actually are. One challenge in providing a tool for measuring success lies in
the fact that success means different things for the different stakeholders:
the users of the service might emphasise the easiness of use while improved
productivity is often mentioned as one of the main benefits from the
perspective of the company (e.g., [3, 4]). These views on success are not necessarily
contradictory; often the easy-to-use service is also more successful
financially. For example, if the service is easy to use, the need for training
decreases and then users can focus on working instead of taking part in
training courses or seeking online help [5]. The bottom line is that the success of a mobile
business service use cannot be determined by utilizing one factor alone—both usability
and business aspects need to be considered together.Another challenge in the
development of a tool for measuring success lies in the fact that such tool
needs to take into account the context. From a technical view, mobile context
includes, for example, the network infrastructure, the broader computational
system, and the actual physical environment [6]. According to Vartiainen [7], connections (e.g., Internet, WLAN, broadband),
devices, as well as applications and services construct virtual spaces that
enable communication and collaboration. Moreover, Vartiainen [7] defines physical environments as settings and
environments such as home, offices, cars, and hotels, where the mobile services
are used. In addition to environmental context (including physical and social
contexts), mobile context also includes personal context from the user’s point
of view [8]. According to Lee etal. [8], emotion, time, and movement are three important
factors related to personal context.On the basis of examining work
in a mobile context, recent studies concluded that there is the so-called
“anytime, anywhere” paradox [9], that is, time frames of mobile work are not always
negotiable, nor are the places where mobile work take place. On the other hand,
Sørensen etal. [10] stated that in the mobile work context, supporting
smart work processes is the key in keeping the dynamic work environment in
control. This control, in turn, ensures safer and more economic work
environment [10]. Moreover, we believe that the complexity of the
mobile work context has an effect on measuring the success factors. Mobility is
not only “a matter of location in time and space” [11] as the mobility can be spatial, temporal, and
contextual [12].In this paper, we will first present
a review on the state of the art of the measurement tools for usability and
productivity. To better understand whether the current tools are sufficient for
the mobile business context, we will present findings from two case studies.
One of the case studies was done on a mobile taxi dispatching system and
another on a mobile safety-measurement service used at construction sites. In
the case studies, we explored factors that have an effect on the success of
mobile business services. In addition, we will highlight the shortcomings of
the current way of measuring success—why some measures
are not suitable or optimal in this context and why none of the existing
measurement tools are sufficient alone. As a result, the paper presents a pool
of items that will be used for developing a questionnaire for evaluating some
key success factors of mobile business services from the individual user’s
point of view.


## 2. Measuring Usability

According to ISO 9241 [13] standard, usability is an extent to which the
intended users of a product achieve specified goals in an effective, efficient
and satisfactory manner within a specified context of use. This section
presents an overview on the existing tools and metrics for measuring usability
and discusses their limitations in the context of mobile business services.


### 2.1. Measuring User’s Performance

Effectiveness—or the accuracy and completeness with which
intended goals are achieved—can be measured by the percent or number of
errors, the success-failure ratio, or the percentage of tasks completed.
Efficiency is a measure of the amount of human, economic, and temporal resources
that are expended in attaining a required level of product effectiveness.
Measures of efficiency include time taken to complete tasks, time spent using
help or documentation, and estimations of subjective workload (mental effort).
Satisfaction refers to the comfort in using the system and its overall functionality. Useful measures of user satisfaction include the
number of times the interface misleads the user or the user loses control of
the system, the percent of favourable/unfavourable comments, and ratings on post-test
questionnaires.According to Nielsen [14], another way to define usability is through the
concepts of learnability, memorability, efficiency, subjective satisfaction,
and low rate of errors. For example, learnability can be measured as the
beginners’ time to attain the average skills with the system and efficiency by
considering how fast and accurately a practiced user can work with the system.
In relation to errors, it is important to separate between errors that are
repairable (users can recover from them) and errors that lead to a catastrophic
failure.


### 2.2. Measuring User’s Subjective Opinions

Quantitatively, user’s
perceived usability ratings are often collected using questionnaires. A number
of usability questionnaires have been carefully developed and tested for their
psychometric qualities, such as (SUS) [15], Software Usability Measurement Inventory (SUMI) [16], Questionnaire for User Interface Satisfaction (QUIS)
[17], The After-Scenario Questionnaire (ASQ), Post-Study
System Usability Questionnaire (PSSUQ), The Computer System Usability
Questionnaire (CSUQ) (all three by [18]), and measurement scales for perceived usefulness and
ease of use [19]. The questionnaires do not only focus on user
satisfaction, but also study how users perceive the other attributes of
usability.The questionnaires differ in
the number and contents of the constructs they measure. For example, SUS
consists of 10 statements that cover several aspects of usability, such as the
need for support, training, and complexity; QUIS, on the other hand, studies
eleven dimensions, such as screen factors, terminology, system feedback,
learning factors, and system capabilities. In both SUS and QUIS, the items
focus on the use of thesystem (e.g., “Overall reactions to software,”
“System speed”) and on the details of the user interface or system
functionality (e.g., “I found the various functions of the system were well
integrated”). On the other hand, the three items in ASQ focus on tasks (e.g., “Overall, I am satisfied with the ease of completing the tasks in this
scenario”); PSSUQ and CSUQ include items with system, task, and interface
focus. The measurement scales for perceived usefulness and ease of use by Davis’s Technology
Acceptance Model (TAM) [19] focus specifically on the use of the system in the
context of work tasks.There are also some models and
questionnaires that are specifically developed for mobile devices and services.
For example, the applied TAM for Mobile
services (TAMM) extends the original TAM with two new perceived product
characteristics affecting the intention to use:trust and ease of adoption. In
addition, TAMM model redefines the theme of usefulness as value to the user [20]. In addition, the Value-based Adoption Model (VAM)
extends TAM by explaining customer’s mobile internet adoption from the
perceived value perspective including usefulness,
enjoyment, fee, and technicality [21]. The Mobile Phone Usability Questionnaire (MPUQ)
evaluates the usability of mobile phones for the purpose of making decisions
among competing variations in the end-user market, alternatives of prototypes
during the development process, and evolving versions during an iterative
design process. In addition, the questionnaire can serve as a tool for
identifying diagnostic information to improve specific usability dimensions and
related interface elements [22]. Moreover, Wehmeyer [23] has developed an initial instrument for assessing
users’ attachment to their mobile devices by three dimensions: symbolism,
aesthetics, and perceived necessity. However, all these previously mentioned methods
are developed for consumer products
and services. Thus, there is still a need for analysing the characteristics of
mobile business services that are
used in a work context. The networked
nature of the mobile work system requires further extensions to user acceptance
and usability measures.


### 2.3. Combining Objective and Subjective Measures

The usability research
publications often report results in relation to the three factors of usability
mentioned in the ISO standard [13], namely, satisfaction, efficiency, and effectiveness.
However, these constructs are often reported separately: one system is reported
as more efficient than the other, whereas, in relation to effectiveness, the
results do not differ between the interfaces. Subjective satisfaction is often
used to give additional testimony on the superiority of one interface over the
other. A reasonable question is whether all the different aspects of usability
need to be studied or whether one would be sufficient: if effectiveness,
efficiency, and subjective satisfaction all point to the same direction, maybe
it is enough to simply (and cheaply) ask the user how much they liked the
service and use this score as an indication of the usability of the service.Nielsen and Levy [24] reviewed 57 studies and concluded that although there
is a clear association between subjective satisfaction and task performance,
there are still many cases where the “users prefer systems that are measurably
worse for them” [24]. Later, Frøkjær etal. [25] showed that correlations between the three aspects of
usability are dependent on the application domain, the user’s experience, and
the use context, and that no simple correlation between the three can be
expected. Especially, for complex “nonroutine” tasks, efficiency measures as
indicators of usability are useless if the task completion (effectiveness) is
not taken into account [25]. Hornbæk [26] also encourages the measurement of all the three
aspects of usability with both subjective and objective methods. However, Hornbæk
[26] does not limit the subjective measures to subjective
satisfaction. Instead, many objective measures, such as task time, can also be
evaluated subjectively: the objective task time with System A may be
faster than with System B, but
the perceived time may be well faster with System B. The implications
such results warrant are dependent on the situation, but overall, the results
can give important information on the usability of the system.


## 2.1. Measuring User’s Performance

Effectiveness—or the accuracy and completeness with which
intended goals are achieved—can be measured by the percent or number of
errors, the success-failure ratio, or the percentage of tasks completed.
Efficiency is a measure of the amount of human, economic, and temporal resources
that are expended in attaining a required level of product effectiveness.
Measures of efficiency include time taken to complete tasks, time spent using
help or documentation, and estimations of subjective workload (mental effort).
Satisfaction refers to the comfort in using the system and its overall functionality. Useful measures of user satisfaction include the
number of times the interface misleads the user or the user loses control of
the system, the percent of favourable/unfavourable comments, and ratings on post-test
questionnaires.According to Nielsen [14], another way to define usability is through the
concepts of learnability, memorability, efficiency, subjective satisfaction,
and low rate of errors. For example, learnability can be measured as the
beginners’ time to attain the average skills with the system and efficiency by
considering how fast and accurately a practiced user can work with the system.
In relation to errors, it is important to separate between errors that are
repairable (users can recover from them) and errors that lead to a catastrophic
failure.


## 2.2. Measuring User’s Subjective Opinions

Quantitatively, user’s
perceived usability ratings are often collected using questionnaires. A number
of usability questionnaires have been carefully developed and tested for their
psychometric qualities, such as (SUS) [15], Software Usability Measurement Inventory (SUMI) [16], Questionnaire for User Interface Satisfaction (QUIS)
[17], The After-Scenario Questionnaire (ASQ), Post-Study
System Usability Questionnaire (PSSUQ), The Computer System Usability
Questionnaire (CSUQ) (all three by [18]), and measurement scales for perceived usefulness and
ease of use [19]. The questionnaires do not only focus on user
satisfaction, but also study how users perceive the other attributes of
usability.The questionnaires differ in
the number and contents of the constructs they measure. For example, SUS
consists of 10 statements that cover several aspects of usability, such as the
need for support, training, and complexity; QUIS, on the other hand, studies
eleven dimensions, such as screen factors, terminology, system feedback,
learning factors, and system capabilities. In both SUS and QUIS, the items
focus on the use of thesystem (e.g., “Overall reactions to software,”
“System speed”) and on the details of the user interface or system
functionality (e.g., “I found the various functions of the system were well
integrated”). On the other hand, the three items in ASQ focus on tasks (e.g., “Overall, I am satisfied with the ease of completing the tasks in this
scenario”); PSSUQ and CSUQ include items with system, task, and interface
focus. The measurement scales for perceived usefulness and ease of use by Davis’s Technology
Acceptance Model (TAM) [19] focus specifically on the use of the system in the
context of work tasks.There are also some models and
questionnaires that are specifically developed for mobile devices and services.
For example, the applied TAM for Mobile
services (TAMM) extends the original TAM with two new perceived product
characteristics affecting the intention to use:trust and ease of adoption. In
addition, TAMM model redefines the theme of usefulness as value to the user [20]. In addition, the Value-based Adoption Model (VAM)
extends TAM by explaining customer’s mobile internet adoption from the
perceived value perspective including usefulness,
enjoyment, fee, and technicality [21]. The Mobile Phone Usability Questionnaire (MPUQ)
evaluates the usability of mobile phones for the purpose of making decisions
among competing variations in the end-user market, alternatives of prototypes
during the development process, and evolving versions during an iterative
design process. In addition, the questionnaire can serve as a tool for
identifying diagnostic information to improve specific usability dimensions and
related interface elements [22]. Moreover, Wehmeyer [23] has developed an initial instrument for assessing
users’ attachment to their mobile devices by three dimensions: symbolism,
aesthetics, and perceived necessity. However, all these previously mentioned methods
are developed for consumer products
and services. Thus, there is still a need for analysing the characteristics of
mobile business services that are
used in a work context. The networked
nature of the mobile work system requires further extensions to user acceptance
and usability measures.


## 2.3. Combining Objective and Subjective Measures

The usability research
publications often report results in relation to the three factors of usability
mentioned in the ISO standard [13], namely, satisfaction, efficiency, and effectiveness.
However, these constructs are often reported separately: one system is reported
as more efficient than the other, whereas, in relation to effectiveness, the
results do not differ between the interfaces. Subjective satisfaction is often
used to give additional testimony on the superiority of one interface over the
other. A reasonable question is whether all the different aspects of usability
need to be studied or whether one would be sufficient: if effectiveness,
efficiency, and subjective satisfaction all point to the same direction, maybe
it is enough to simply (and cheaply) ask the user how much they liked the
service and use this score as an indication of the usability of the service.Nielsen and Levy [24] reviewed 57 studies and concluded that although there
is a clear association between subjective satisfaction and task performance,
there are still many cases where the “users prefer systems that are measurably
worse for them” [24]. Later, Frøkjær etal. [25] showed that correlations between the three aspects of
usability are dependent on the application domain, the user’s experience, and
the use context, and that no simple correlation between the three can be
expected. Especially, for complex “nonroutine” tasks, efficiency measures as
indicators of usability are useless if the task completion (effectiveness) is
not taken into account [25]. Hornbæk [26] also encourages the measurement of all the three
aspects of usability with both subjective and objective methods. However, Hornbæk
[26] does not limit the subjective measures to subjective
satisfaction. Instead, many objective measures, such as task time, can also be
evaluated subjectively: the objective task time with System A may be
faster than with System B, but
the perceived time may be well faster with System B. The implications
such results warrant are dependent on the situation, but overall, the results
can give important information on the usability of the system.


## 3. Measuring Productivity

It is widely agreed that IT
investments in general have an impact on the output of the company and thus, on
productivity (e.g., [27]). Productivity is a multidimensional term and its
meaning can vary depending on the context in which it is used [28]. Productivity can be defined simply as output
divided by the input that is used to generate the output: input
consists of materials, labour, capital, energy, and so forth, and output
consists of products, services or information (e.g., [28–30]). This section provides an overview on the existing
tools for measuring productivity of a company. In addition, the limitations of
measuring productivity improvement resulting from the use of mobile business
services are discussed.


### 3.1. Traditional
Productivity Measurement

Traditionally, productivity
measures have been categorised as either total productivity measures or partial
productivity measures [29]. In practise, the measurement of total productivity
(i.e., comparing all outputs with all inputs) is difficult as different types
of outputs and inputs cannot be summed up and compared directly. Therefore, partial
productivity measures are commonly used. Partial productivity ratio can be
calculated by dividing the total output by one input or some portion of inputs [29]. Labour productivity is a most common example of this
kind of measure and it is usually used when measuring the impacts of IT investments.
Labour productivity is the ratio calculated by dividing the total output (e.g.,
number of products) by labour input (e.g., man-hour) [31].Moreover,indirect productivity measurement can be
used when the data needed for partial productivity measures is not available.
The idea behind indirect productivity measures is that certain symptoms or
phenomena are related to problems in productivity. They include, for example,
high defect rates, machine defects, unused capacity, high material scrap,
unnecessary transports, poor atmosphere, and long waiting times (e.g., [30, 32]). Indirect productivity measures focus on these
factors which are closely related to productivity and which can be used to
explain the reasons behind the changes in productivity [31].According to Misterek etal. [29], the problem with productivity measures is that they
do not take into account changes in the quality of the inputs or outputs.
Especially in service business, quality and productivity cannot be dealt
separately [33]. Moreover, measuring the effects of mobile business
services on productivity with these traditional measures may be difficult
because of business process changes and indirect influences (e.g., improved
customer satisfaction) these services cause [34]. For example, Brynjolfsson and Hitt [35] noted that information technology investment may
create intangible assets which in turn may have a positive impact on the
productivity of the company but are not included in traditional evaluations. Including
intangible aspects could provide a more accurate and realistic view of returns
as a result of the investment [36].


### 3.2. Subjective
Productivity Measurement

The use of the traditional
productivity measurement methods presented above requires collecting
quantitative data from operations. Sometimes the inputs and output are not
easily quantifiable and it may be difficult or even impossible to collect the
data that is needed for productivity measurement. For example, in service work,
inputs and outputs are usually intangible and the quality may vary a lot. In
these cases,subjective productivity measurement is one possible method
to collect the needed information about the level of or problems in
productivity. Subjective productivity measures are based on personnel’s
subjective assessments and data is usually collected using surveys or interviews
(see, e.g., [31, 37]). As traditional objective productivity measures,
also subjective measures can be direct or indirect. Direct productivity
measures can be, for example, partial productivity ratios (objective) or survey
questions regarding employees’ perceptions of their own productivity
(subjective). Indirect productivity measures can be, for example, unused
capacity, unnecessary transports, and long waiting times (objective) or survey
questions concerning employees’ perceptions of waiting times needed
(subjective) [31].As subjective measurement is
also a common way of measuring usability, it could provide a basis for
combining usability and productivity perspectives.  In addition, it could be a good way of
including intangible aspects (such as quality) when measuring the productivity
impacts of mobile business services.


## 3.1. Traditional
Productivity Measurement

Traditionally, productivity
measures have been categorised as either total productivity measures or partial
productivity measures [29]. In practise, the measurement of total productivity
(i.e., comparing all outputs with all inputs) is difficult as different types
of outputs and inputs cannot be summed up and compared directly. Therefore, partial
productivity measures are commonly used. Partial productivity ratio can be
calculated by dividing the total output by one input or some portion of inputs [29]. Labour productivity is a most common example of this
kind of measure and it is usually used when measuring the impacts of IT investments.
Labour productivity is the ratio calculated by dividing the total output (e.g.,
number of products) by labour input (e.g., man-hour) [31].Moreover,indirect productivity measurement can be
used when the data needed for partial productivity measures is not available.
The idea behind indirect productivity measures is that certain symptoms or
phenomena are related to problems in productivity. They include, for example,
high defect rates, machine defects, unused capacity, high material scrap,
unnecessary transports, poor atmosphere, and long waiting times (e.g., [30, 32]). Indirect productivity measures focus on these
factors which are closely related to productivity and which can be used to
explain the reasons behind the changes in productivity [31].According to Misterek etal. [29], the problem with productivity measures is that they
do not take into account changes in the quality of the inputs or outputs.
Especially in service business, quality and productivity cannot be dealt
separately [33]. Moreover, measuring the effects of mobile business
services on productivity with these traditional measures may be difficult
because of business process changes and indirect influences (e.g., improved
customer satisfaction) these services cause [34]. For example, Brynjolfsson and Hitt [35] noted that information technology investment may
create intangible assets which in turn may have a positive impact on the
productivity of the company but are not included in traditional evaluations. Including
intangible aspects could provide a more accurate and realistic view of returns
as a result of the investment [36].


## 3.2. Subjective
Productivity Measurement

The use of the traditional
productivity measurement methods presented above requires collecting
quantitative data from operations. Sometimes the inputs and output are not
easily quantifiable and it may be difficult or even impossible to collect the
data that is needed for productivity measurement. For example, in service work,
inputs and outputs are usually intangible and the quality may vary a lot. In
these cases,subjective productivity measurement is one possible method
to collect the needed information about the level of or problems in
productivity. Subjective productivity measures are based on personnel’s
subjective assessments and data is usually collected using surveys or interviews
(see, e.g., [31, 37]). As traditional objective productivity measures,
also subjective measures can be direct or indirect. Direct productivity
measures can be, for example, partial productivity ratios (objective) or survey
questions regarding employees’ perceptions of their own productivity
(subjective). Indirect productivity measures can be, for example, unused
capacity, unnecessary transports, and long waiting times (objective) or survey
questions concerning employees’ perceptions of waiting times needed
(subjective) [31].As subjective measurement is
also a common way of measuring usability, it could provide a basis for
combining usability and productivity perspectives.  In addition, it could be a good way of
including intangible aspects (such as quality) when measuring the productivity
impacts of mobile business services.


## 4. Exploring the Success Factors of Mobile Business Services

The main aim of this paper was to study the factors that have an effect
on the success of mobile business services and to construct a preliminary measurement
tool for measuring the level of success. Given the variability of tools already available for measuring usability
and productivity (the ones reviewed in previous sections and much more), why
would anyone want to develop yet another set of tools? For us, there were two
main reasons: first, the current tools do not address the specific context we
are interested in, namely,mobile services for business use, and
second, the current tools only address either usability—or productivity-related
issues—and we strongly
feel that both of them need to be addressed in order to measure success. In order to
understand the characteristics and success factors of mobile services that are
used in a mobile work context, we conducted two case studies.In the first case study, we examined a mobile business service used for
taxi booking and dispatching. Taxi drivers use the service constantly in their
daily work while driving. The service is used, for example, for accepting
incoming requests, searching customers’ addresses and other locations on the
map, and messaging to the dispatching centre and other taxis. The service is
used on a fixed in-car device. In the second case study, we studied a mobile
business service that is used at construction sites for weekly safety
measurements. Safety delegate observes various safety issues at the
construction site and uses the mobile service to document the observations.
This service is used on a mobile phone.


### 4.1. Methods and Participants

Our goal
is to gain understanding of the phenomena, that is, measuring the success
factors of mobile business services, and we are using qualitative research
methods to gain that understanding. Therefore, our study is based on an
interpretive research paradigm whereby we aim at understanding the context of
the system better (see [7]) (cf. the positivist paradigm where the assumption
is that the measurable properties of the phenomenon are not dependent of the
single observer). As data collection methods, we used focus groups, interviews,
and observations. The data collection took place during the spring 2006.For
the first case study, we conducted two focus groups with a total of nine taxi
drivers and two pair interviews with dispatchers to analyze the success factors
in their mobile work context while using the taxi order booking and dispatching
system. To
evaluate the service from the perspective of new users, we observed the
training of almost 50 taxi driver candidates during three three-hour training
sessions. During these sessions, the candidates used the mobile service to
complete typical usage scenarios in a classroom setting. To evaluate the flow
of interaction and other usability issues of the taxi drivers’ system, two
usability specialists spent one hour using a demonstration system to complete
typical taxi driver’s tasks (such as accepting a customer order, locating an
address from the map, and changing the settings of the interface). To gain a
wide business perspective on the themes related to the success of the mobile
business service, we interviewed the taxi company’s managing director and the
chairman of the board.For the second case study, we had one group discussion at a construction
site, where we discussed the use of the service with the representatives of the
construction company and the service provider. In addition, we interviewed one
user at another construction site and observed three users at two different
construction sites while using the mobile service to perform safety
measurement. To better understand the challenges and opportunities that the new
service brings along, we also observed the traditional safety measurement
process where the safety measurement was done by filling in a paper form. After
the observations, we asked the users to fill in the SUS questionnaire in order to get information
about the usefulness of the SUS questionnaire and its limitations in this
context.


### 4.2. Data Analysis

The focus group interviews in
the first case study and all the other interviews were recorded with a digital
voice recorder and transcribed for further analysis. The transcribed interviews
were analysed qualitatively in order to find important themes related to mobile
work from them (content analysis). Each interview was analysed at least by two
researchers. When observing the users, the data was collected by the
researchers (by taking notes). The observations were analysed so that the
observed process was deeply understood: the roles and tasks of the actors, how
they performed their work, which actors were involved in the process, and so
forth. SUS questionnaires were analysed following its guidelines. We also compared the results of
the questionnaires and the observations in line. After the data collection phase, the researchers had
a group discussion where the goal was to elicit a common understanding on the
important themes.


## 4.1. Methods and Participants

Our goal
is to gain understanding of the phenomena, that is, measuring the success
factors of mobile business services, and we are using qualitative research
methods to gain that understanding. Therefore, our study is based on an
interpretive research paradigm whereby we aim at understanding the context of
the system better (see [7]) (cf. the positivist paradigm where the assumption
is that the measurable properties of the phenomenon are not dependent of the
single observer). As data collection methods, we used focus groups, interviews,
and observations. The data collection took place during the spring 2006.For
the first case study, we conducted two focus groups with a total of nine taxi
drivers and two pair interviews with dispatchers to analyze the success factors
in their mobile work context while using the taxi order booking and dispatching
system. To
evaluate the service from the perspective of new users, we observed the
training of almost 50 taxi driver candidates during three three-hour training
sessions. During these sessions, the candidates used the mobile service to
complete typical usage scenarios in a classroom setting. To evaluate the flow
of interaction and other usability issues of the taxi drivers’ system, two
usability specialists spent one hour using a demonstration system to complete
typical taxi driver’s tasks (such as accepting a customer order, locating an
address from the map, and changing the settings of the interface). To gain a
wide business perspective on the themes related to the success of the mobile
business service, we interviewed the taxi company’s managing director and the
chairman of the board.For the second case study, we had one group discussion at a construction
site, where we discussed the use of the service with the representatives of the
construction company and the service provider. In addition, we interviewed one
user at another construction site and observed three users at two different
construction sites while using the mobile service to perform safety
measurement. To better understand the challenges and opportunities that the new
service brings along, we also observed the traditional safety measurement
process where the safety measurement was done by filling in a paper form. After
the observations, we asked the users to fill in the SUS questionnaire in order to get information
about the usefulness of the SUS questionnaire and its limitations in this
context.


## 4.2. Data Analysis

The focus group interviews in
the first case study and all the other interviews were recorded with a digital
voice recorder and transcribed for further analysis. The transcribed interviews
were analysed qualitatively in order to find important themes related to mobile
work from them (content analysis). Each interview was analysed at least by two
researchers. When observing the users, the data was collected by the
researchers (by taking notes). The observations were analysed so that the
observed process was deeply understood: the roles and tasks of the actors, how
they performed their work, which actors were involved in the process, and so
forth. SUS questionnaires were analysed following its guidelines. We also compared the results of
the questionnaires and the observations in line. After the data collection phase, the researchers had
a group discussion where the goal was to elicit a common understanding on the
important themes.


## 5. Results

In this section, we will present the dimensions related to the success
of mobile business services we found in the case studies. In Section5.2, we will
present the initial version of the questionnaire for measuring the success of
mobile business services.


### 5.1. Success Factors Revealed in the Case Studies

In both of the case studies,
we found that the mobile business service needs to fulfill similar usability
goals than other systems. These services need to be effective, efficient,
pleasant, and easy to use to make it easier for the mobile workers to accept
the new service in their daily work. From the companies’ point of view,
improved productivity was seen as an important success factor. This is also in
line with previous studies (see, e.g., [3, 4, 11]). In both of the cases, the mobile business service
has helped the users in performing their work more productively. Moreover, when
the users of the services are mobile, safety is the most important aspect of success. Taxi drivers are driving and
construction workers walking around the construction sites while using the
services. In both of these cases, users need to observe the environment and
minimise their interaction with the service to avoid accidents.In addition to the general usability goals, productivity and safety, we
found thatlearnability is important
for the success of the service. Earlier, Nielsen [14] has emphasised the role of learnability in usability
evaluation. In addition, Zhang and Adipat [38] as well as Turel [39] have discussed the importance of learnability in
mobile usability. In business use, severe errors should not happen even when
the users are learning to use the service and learning should not take too much
time. The learning phase may be expensive for both the company deploying the
service (work tasks cannot be done as efficiently as possible) as well as for
the service provider. For the service provider, the costs are due, for example,
to numerous calls to the helpline and extended need for on-site support.Before using the safety measurement service, the users have toinstall it into their mobile devices.
This phase was found to play a critical role in the success of the service. If
the users did not succeed with installation, they could easily switch back to
using the traditional pen-and-paper method to perform their work task. For the
construction company, this meant that the benefits of the service were not
achieved. The installation phase was not as critical in the taxi case, because
the taxi drivers did not install the services themselves. However, it should be
noted that installation is a one-time activity and increasingly automated for
the users. After successful installation, the users are not likely affected by
the possible difficulties in the installation process too much. The most
important thing is that the users can get the service installed.In the taxi case, one factor having a large effect on the users’
opinions of the service was the level ofsupport,
whether the users felt that they receive enough support and that their problems
and ideas are heard and acted upon. Support had also an important role in the construction
case where the users of the mobile safety measurement service needed additional
help while on the move. For example, during one observation, the user did not
know how to answer an incoming call without distracting the safety measure and at
the end, he needed to start the measurement again. This resulted in there being
two measurements in the system and the user support needed to combine the two
measurements together.Moreover, we found that mobile
business services need to take into account issues related to themobile context. In previous studies, understanding
the context is also seen essential for the success of mobile service design [8, 40]. In addition, Kemppilä and Lönnqvist [31] stated that there is a need to go deeper than just
tracking the physical attributes to be able to understand the effects of the
context better. In analysing the context of work, we need to understand (1)
structure or cues that invites users to enter information spaces or tasks
(entry points), (2) the mental and physical spaces users are constructing while
accomplishing tasks (action landscapes), and (3) artefacts and environmental
structures that help users manage the complexity of their tasks (coordinating
mechanism) [31].Exploring the context of work,
the environmental context (including surrounding environment and physical
elements but also organizational factors) sets requirements for the design of
mobile business services. Sometimes, changes in the context are challenging to
manage, for example, sunshine or coldness when using mobile services.
Organizations provide the work spaces and technology, such as network
connections and devices for their employees. In the taxi case study, taxi
drivers needed to cope with the changing environmental context and we found
that connections were lost from time to time when changing location, cold
weather slowed down the device, and sunshine made it difficult to read the
screen. In the construction case, lost connection also distracted the main
task, that is, the safety measurement process and small keys on the mobile
phone were difficult to use, especially in the winter-time with gloves. In
addition, new systems need to be compatible with the old ones.


### 5.2. Development of a Measurement Tool

In order to develop a proper
measurement tool for mobile business services, we decided to develop an initial
version of questionnaire. A questionnaire as a research method is relatively
efficient and cheap. The advantage of a usability questionnaire, compared to
laboratory evaluations, expert reviews, and checklists usages, is that a
questionnaire is reasonable easy and quick to be carried out and also the real
end users of the product are involved in the process. The measures made with a
questionnaire can be used, for example, to make comparison of user satisfaction
between different versions of the same service or with competing services
(benchmarking).To begin the development of
the questionnaire, we took an existing questionnaire, SUS, which we tested in
our second case study with construction workers. This questionnaire is simple
and quick to fill in, but unfortunately its usefulness for the current context
was found to be limited as it does not take into account the mobility of the
service or the business context.  For
example, the claim “I think that I would like to use this system frequently” is
confusing as the users must use the mobile service to perform their work tasks.
We also noticed that some of the statements in SUS were ambiguous as “I found
the various functions in this system were well integrated”.From the case studies, several
success factors were identified: effectiveness, efficiency, user satisfaction,
ease of use, safety, learnability, ease of installation, support, factors
related to mobile context, and productivity. Effectiveness, efficiency, and
user satisfaction are also emphasised in the ISO standard’s definition of
usability [13]; efficiency, learnability, and satisfaction are
emphasised by Nielsen [14], and ease of use is, for example, part of TAM [19]. In addition, installation and support (technical
manuals and online help) are part of QUIS [17].Some of the above-mentioned issues are addressed by the existing questionnaires
(SUMI, QUIS, PSSUQ, SUS, USE questionnaire, TAM, and MPUQ), but they do not
specifically focus on the issues that are relevant for the mobile work context.
Thus, we revised the otherwise-relevant questions from the existing tools to
better measure the success factors that are important for mobile business
services. We also added new dimensions of success, namely, safety, mobile
context, and productivity.The existing usability measures
have been mainly developed for desktop computers or consumer markets and used
in usability tests. In contrast, the usability measures for mobile business
services must take into account the different contexts of use and the workload
the use of the service poses, among other things. Mobile business services can
be used with different devices, such as mobile phones or fixed in-car systems,
and the usability measures need to take into account the appropriateness of the
device to the users’ task. In addition, the existing tools do not take into
account the fact that the service is typically only part of the story; the
service may fail to support the work task because of problems in some other
parts of the network that is needed to support the use of the service. The mobile service rarely acts as an independent tool
in the working environment but information needs to be transferred somewhere.
Thus, it is important that the mobile service and the device enable fast and
easy connection with other devices and users. To address these issues, the
dimension ofmobile context includes
questions related to the actual physical context, that is, mobile device and environment,
as well as the overall system context
(see, e.g., [6]).In the context of consumer services, subjective satisfaction and ease of use
are extremely important factors as they have a large effect on theacceptability and adoption of the service [20]. In addition, the users of consumer
services often have the freedom of choice—if they do not
like the current service, they can choose another one. In business context, the
end users (workers) often do not have a choice of not using the system and
unfortunately, they often do not even have a say when the company is deciding
on which service deploy. Despite of that (or rather, because of that), we feel that subjective
satisfaction is important to be taken into account also in the business context, as it is likely
to have an effect on other measures, such as the overall work satisfaction
and productivity. In order to evaluate the productivity impacts, we added
both direct and indirect subjective productivity measures under the theme
“mobile work productivity.” In addition, especially the questions under the
themes “efficiency” and “effectiveness” are closely related to
productivity.The statements
from the existing questionnaires were selected based on the themes and their
relevance to business use. The selected statements were modified so that
“mobile service” and “work tasks” were emphasised. It should be noted that
these themes and questions need to be further adapted based on the context in
which they are applied. For example, if the service
is intended for taxi drivers and used inside of the vehicles, there is no need
to ask if the service does not do that well in dust or while walking. In
addition, some themes are more relevant in the deployment phase than in the
continued use. For example, ease of installation and learnability are crucial
in the early stages of usage, whereas the effects on efficiency and
productivity may be observed over longer periods of use.To
summarise, we identified a total of ten different dimensions from our case
studies and existing literature that are likely to have a significant effect on
the success of mobile services. To meet the demands of measuring the success
factors of mobile business services, we found that the existing questionnaires
are not sufficient alone. Thus, the following factors were added to the
repertoire:safety, mobile context, and
impact on mobile work productivity. The list of dimensions and some
examples of the revised and added statements for the questionnaire on users’
perceived usability and productivity impacts of the mobile service in work
tasks are listed in Table 1.Table 1
The dimensions for the
questionnaire on the success of mobile business services and examples of statements.
DimensionsExamples of statementsSourceInstallationThe mobile service was easy to assemble, install, and/or setup into the mobile device[17, 22]LearnabilityIt was easy to learn to use the mobile service[16, 17, 22, 41]Experience with other mobile services (or products)
made it easier to operate this serviceEase of useIt is easy to move from one part of a task to the next with the mobile service[16, 22]EfficiencyI can complete my work tasks quickly by using the service with a mobile device[18]The mobile service responds quickly to my actions[16, 17]EffectivenessThe work task sometimes fails because of the mobile serviceThe mobile service enables quick, effective and economical performance of work tasks[18]User satisfactionI would recommend the mobile service also for others doing the same work[16]Using the mobile service in performing the work tasks is pleasing[16]Factors related to Mobile work contextThe battery capacity of the mobile device is sufficient for the use of the service for work tasksThe screen size of the mobile device is adequate for
using the service for work tasksInputting information to the mobile device is easySometimes, the environment (such as coldness, sunshine, darkness) makes the use of the service difficultThe mobile device suits well for performing my work tasks while on the moveExchanging and transmission of data between the mobile service and other products (e.g., computer, other mobile devices) is easy[16, 17, 22]Switching between the different applications (e.g., calls and work task) in the mobile device is easySafetyThe use of the mobile service has caused me safety risks while on the moveI sometimes have to fully concentrate on using the mobile service and cannot observe the environmentIt is easy to perform work tasks in a hurry with the mobile serviceSupportI always know who to ask for help if I have problems performing work tasks with the mobile serviceThe help information given by the mobile service is
useful[16]Impacts on Mobile work productivityUsing the mobile service in my job reduces travelling from and to the office during the work dayThe mobile service helps me complete my work tasks quicklyUsing the mobile service in my job increases my
productivity[19]Using the mobile service in my job increases my work motivationThe dimensions listed in Table1 are the ones
that we recognised from the case studies to be important determinants of the
success of mobile business services. However, the dimensions and example items
should not be treated as a ready-to-use questionnaire: the dimensions need to
be prioritised for each case to focus on the most important aspects of success
and more items should possibly be added to the most important dimensions to
capture all the aspects of them. In its current state, this questionnaire does
not propose a way to collect the ratings from the users. Although we expect
Likert-scales to be a useful way to collect ratings, more research is needed to
determine the most appropriate way.


## 5.1. Success Factors Revealed in the Case Studies

In both of the case studies,
we found that the mobile business service needs to fulfill similar usability
goals than other systems. These services need to be effective, efficient,
pleasant, and easy to use to make it easier for the mobile workers to accept
the new service in their daily work. From the companies’ point of view,
improved productivity was seen as an important success factor. This is also in
line with previous studies (see, e.g., [3, 4, 11]). In both of the cases, the mobile business service
has helped the users in performing their work more productively. Moreover, when
the users of the services are mobile, safety is the most important aspect of success. Taxi drivers are driving and
construction workers walking around the construction sites while using the
services. In both of these cases, users need to observe the environment and
minimise their interaction with the service to avoid accidents.In addition to the general usability goals, productivity and safety, we
found thatlearnability is important
for the success of the service. Earlier, Nielsen [14] has emphasised the role of learnability in usability
evaluation. In addition, Zhang and Adipat [38] as well as Turel [39] have discussed the importance of learnability in
mobile usability. In business use, severe errors should not happen even when
the users are learning to use the service and learning should not take too much
time. The learning phase may be expensive for both the company deploying the
service (work tasks cannot be done as efficiently as possible) as well as for
the service provider. For the service provider, the costs are due, for example,
to numerous calls to the helpline and extended need for on-site support.Before using the safety measurement service, the users have toinstall it into their mobile devices.
This phase was found to play a critical role in the success of the service. If
the users did not succeed with installation, they could easily switch back to
using the traditional pen-and-paper method to perform their work task. For the
construction company, this meant that the benefits of the service were not
achieved. The installation phase was not as critical in the taxi case, because
the taxi drivers did not install the services themselves. However, it should be
noted that installation is a one-time activity and increasingly automated for
the users. After successful installation, the users are not likely affected by
the possible difficulties in the installation process too much. The most
important thing is that the users can get the service installed.In the taxi case, one factor having a large effect on the users’
opinions of the service was the level ofsupport,
whether the users felt that they receive enough support and that their problems
and ideas are heard and acted upon. Support had also an important role in the construction
case where the users of the mobile safety measurement service needed additional
help while on the move. For example, during one observation, the user did not
know how to answer an incoming call without distracting the safety measure and at
the end, he needed to start the measurement again. This resulted in there being
two measurements in the system and the user support needed to combine the two
measurements together.Moreover, we found that mobile
business services need to take into account issues related to themobile context. In previous studies, understanding
the context is also seen essential for the success of mobile service design [8, 40]. In addition, Kemppilä and Lönnqvist [31] stated that there is a need to go deeper than just
tracking the physical attributes to be able to understand the effects of the
context better. In analysing the context of work, we need to understand (1)
structure or cues that invites users to enter information spaces or tasks
(entry points), (2) the mental and physical spaces users are constructing while
accomplishing tasks (action landscapes), and (3) artefacts and environmental
structures that help users manage the complexity of their tasks (coordinating
mechanism) [31].Exploring the context of work,
the environmental context (including surrounding environment and physical
elements but also organizational factors) sets requirements for the design of
mobile business services. Sometimes, changes in the context are challenging to
manage, for example, sunshine or coldness when using mobile services.
Organizations provide the work spaces and technology, such as network
connections and devices for their employees. In the taxi case study, taxi
drivers needed to cope with the changing environmental context and we found
that connections were lost from time to time when changing location, cold
weather slowed down the device, and sunshine made it difficult to read the
screen. In the construction case, lost connection also distracted the main
task, that is, the safety measurement process and small keys on the mobile
phone were difficult to use, especially in the winter-time with gloves. In
addition, new systems need to be compatible with the old ones.


## 5.2. Development of a Measurement Tool

In order to develop a proper
measurement tool for mobile business services, we decided to develop an initial
version of questionnaire. A questionnaire as a research method is relatively
efficient and cheap. The advantage of a usability questionnaire, compared to
laboratory evaluations, expert reviews, and checklists usages, is that a
questionnaire is reasonable easy and quick to be carried out and also the real
end users of the product are involved in the process. The measures made with a
questionnaire can be used, for example, to make comparison of user satisfaction
between different versions of the same service or with competing services
(benchmarking).To begin the development of
the questionnaire, we took an existing questionnaire, SUS, which we tested in
our second case study with construction workers. This questionnaire is simple
and quick to fill in, but unfortunately its usefulness for the current context
was found to be limited as it does not take into account the mobility of the
service or the business context.  For
example, the claim “I think that I would like to use this system frequently” is
confusing as the users must use the mobile service to perform their work tasks.
We also noticed that some of the statements in SUS were ambiguous as “I found
the various functions in this system were well integrated”.From the case studies, several
success factors were identified: effectiveness, efficiency, user satisfaction,
ease of use, safety, learnability, ease of installation, support, factors
related to mobile context, and productivity. Effectiveness, efficiency, and
user satisfaction are also emphasised in the ISO standard’s definition of
usability [13]; efficiency, learnability, and satisfaction are
emphasised by Nielsen [14], and ease of use is, for example, part of TAM [19]. In addition, installation and support (technical
manuals and online help) are part of QUIS [17].Some of the above-mentioned issues are addressed by the existing questionnaires
(SUMI, QUIS, PSSUQ, SUS, USE questionnaire, TAM, and MPUQ), but they do not
specifically focus on the issues that are relevant for the mobile work context.
Thus, we revised the otherwise-relevant questions from the existing tools to
better measure the success factors that are important for mobile business
services. We also added new dimensions of success, namely, safety, mobile
context, and productivity.The existing usability measures
have been mainly developed for desktop computers or consumer markets and used
in usability tests. In contrast, the usability measures for mobile business
services must take into account the different contexts of use and the workload
the use of the service poses, among other things. Mobile business services can
be used with different devices, such as mobile phones or fixed in-car systems,
and the usability measures need to take into account the appropriateness of the
device to the users’ task. In addition, the existing tools do not take into
account the fact that the service is typically only part of the story; the
service may fail to support the work task because of problems in some other
parts of the network that is needed to support the use of the service. The mobile service rarely acts as an independent tool
in the working environment but information needs to be transferred somewhere.
Thus, it is important that the mobile service and the device enable fast and
easy connection with other devices and users. To address these issues, the
dimension ofmobile context includes
questions related to the actual physical context, that is, mobile device and environment,
as well as the overall system context
(see, e.g., [6]).In the context of consumer services, subjective satisfaction and ease of use
are extremely important factors as they have a large effect on theacceptability and adoption of the service [20]. In addition, the users of consumer
services often have the freedom of choice—if they do not
like the current service, they can choose another one. In business context, the
end users (workers) often do not have a choice of not using the system and
unfortunately, they often do not even have a say when the company is deciding
on which service deploy. Despite of that (or rather, because of that), we feel that subjective
satisfaction is important to be taken into account also in the business context, as it is likely
to have an effect on other measures, such as the overall work satisfaction
and productivity. In order to evaluate the productivity impacts, we added
both direct and indirect subjective productivity measures under the theme
“mobile work productivity.” In addition, especially the questions under the
themes “efficiency” and “effectiveness” are closely related to
productivity.The statements
from the existing questionnaires were selected based on the themes and their
relevance to business use. The selected statements were modified so that
“mobile service” and “work tasks” were emphasised. It should be noted that
these themes and questions need to be further adapted based on the context in
which they are applied. For example, if the service
is intended for taxi drivers and used inside of the vehicles, there is no need
to ask if the service does not do that well in dust or while walking. In
addition, some themes are more relevant in the deployment phase than in the
continued use. For example, ease of installation and learnability are crucial
in the early stages of usage, whereas the effects on efficiency and
productivity may be observed over longer periods of use.To
summarise, we identified a total of ten different dimensions from our case
studies and existing literature that are likely to have a significant effect on
the success of mobile services. To meet the demands of measuring the success
factors of mobile business services, we found that the existing questionnaires
are not sufficient alone. Thus, the following factors were added to the
repertoire:safety, mobile context, and
impact on mobile work productivity. The list of dimensions and some
examples of the revised and added statements for the questionnaire on users’
perceived usability and productivity impacts of the mobile service in work
tasks are listed in Table 1.Table 1
The dimensions for the
questionnaire on the success of mobile business services and examples of statements.
DimensionsExamples of statementsSourceInstallationThe mobile service was easy to assemble, install, and/or setup into the mobile device[17, 22]LearnabilityIt was easy to learn to use the mobile service[16, 17, 22, 41]Experience with other mobile services (or products)
made it easier to operate this serviceEase of useIt is easy to move from one part of a task to the next with the mobile service[16, 22]EfficiencyI can complete my work tasks quickly by using the service with a mobile device[18]The mobile service responds quickly to my actions[16, 17]EffectivenessThe work task sometimes fails because of the mobile serviceThe mobile service enables quick, effective and economical performance of work tasks[18]User satisfactionI would recommend the mobile service also for others doing the same work[16]Using the mobile service in performing the work tasks is pleasing[16]Factors related to Mobile work contextThe battery capacity of the mobile device is sufficient for the use of the service for work tasksThe screen size of the mobile device is adequate for
using the service for work tasksInputting information to the mobile device is easySometimes, the environment (such as coldness, sunshine, darkness) makes the use of the service difficultThe mobile device suits well for performing my work tasks while on the moveExchanging and transmission of data between the mobile service and other products (e.g., computer, other mobile devices) is easy[16, 17, 22]Switching between the different applications (e.g., calls and work task) in the mobile device is easySafetyThe use of the mobile service has caused me safety risks while on the moveI sometimes have to fully concentrate on using the mobile service and cannot observe the environmentIt is easy to perform work tasks in a hurry with the mobile serviceSupportI always know who to ask for help if I have problems performing work tasks with the mobile serviceThe help information given by the mobile service is
useful[16]Impacts on Mobile work productivityUsing the mobile service in my job reduces travelling from and to the office during the work dayThe mobile service helps me complete my work tasks quicklyUsing the mobile service in my job increases my
productivity[19]Using the mobile service in my job increases my work motivationThe dimensions listed in Table1 are the ones
that we recognised from the case studies to be important determinants of the
success of mobile business services. However, the dimensions and example items
should not be treated as a ready-to-use questionnaire: the dimensions need to
be prioritised for each case to focus on the most important aspects of success
and more items should possibly be added to the most important dimensions to
capture all the aspects of them. In its current state, this questionnaire does
not propose a way to collect the ratings from the users. Although we expect
Likert-scales to be a useful way to collect ratings, more research is needed to
determine the most appropriate way.


## 6. Discussion

Previous research on usability
measurement tools has focused mostly on desktop environments. Usability is a
critical issue in any service, but when poor usability may affect the security
of the mobile users (and their customers) or the productivity of their work,
the issue becomes even more important. In addition, when usability may affect
not only the business operations of the company who produced the service but
also the companies deploying it, the business impact of usability may be noticeably
large. Thus, when considering the success of mobile business services, we claim
that usability, context of work,
and productivity must be taken into account. Thus far, measuring the success of
mobile business services has received little attention. There are several
independent measurement tools for usability designers and business managers. To
the best of our knowledge, this is the first attempt to combine the dimensions
of usability and productivity into a single measurement tool for mobile
business services.In this paper, we identified
ten success factors of mobile business services through two case studies by
combining the approaches of usability and productivity measurement. These
success factors include the ease of installation, learnability, ease of use,
efficiency, effectiveness, user satisfaction, factors related to mobile
context, safety, support, and mobile work productivity. Most of these factors
are included in the current tools for usability measurement. Thus, the existing
usability questionnaires were used as the basis for the current tool although
they needed to be revised for the current purposes. In addition, themes needed
to be added to capture all the important factors in relation to the success of
mobile business services, such as the issues related to the mobile device,
environment, and the overall system, as well as safety, support, and mobile
work productivity.We expect the results of the questionnaire
to be useful, for example, in analysing and benchmarking the level in which the
users of the service perceive the usability and productivity of the service to
be. Moreover, by reusing the questionnaire, results may be used for detecting
changes, for example, in satisfaction and productivity over time. The results
not only help the service developers in developing more successful services,
but they can also be useful for the company using the service. For example,
information about the level of perceived productivity and problems related to
productivity in mobile work can be used as diagnostic tools to improve the
productivity over time.In this paper, we are
proposing that a questionnaire could be used to measure the success of mobile
business services. We acknowledge the limitations of the questionnaire approach
(it measures the users’ perception of things rather than the objective truth,
users may have difficulties interpreting the questions)—but we claim that
the benefits of this approach overweight the limitations: questionnaire is a
fast data-collection method, it can address several different themes, and it
can provide quantifiable information about the themes in question. However, we
do not claim that questionnaire (or any other single data collection method)
should be used alone. Objective performance data (task time, customers/day, or
revenue) should naturally be used as indicators of success as well. However,
relying only on sales or objective efficiency may result in the less successful
parts of the service going unnoticed—and potential for
further development not being realised.In our two-year MOMENTO
project, the next step is to test the suggested themes and items of the
questionnaire in several case studies. In addition to making sure that the dimensions
are sufficient and applicable to different domains where mobile business
services are used, we will address the issues related to the success of the
individual items, the optimal number of items per dimension, and the scoring of
the questionnaires. In addition, the validity and reliability of the
questionnaire will be evaluated (see, e.g., [41]).In this paper, we have mainly addressed the
success factors from a single worker’s viewpoint. Other themes that emerged in
the case studies were related to measuring success factors at the
organizational level and in a more objective manner. These issues will be
explored and identified in more detail in our future work.



---

*Source: 680159-2008-09-23.xml*