# Analyzing Accessibility Barriers Using Cost-Benefit Analysis to Design Reliable Navigation Services for Wheelchair Users

Benjamin Tannert, Reuben Kirkham, Johannes Schöning

► To cite this version:

Benjamin Tannert, Reuben Kirkham, Johannes Schöning. Analyzing Accessibility Barriers Using Cost-Benefit Analysis to Design Reliable Navigation Services for Wheelchair Users. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.202-223, 10.1007/978-3-030-29381-9_13. hal-02544586

HAL Id: hal-02544586

https://inria.hal.science/hal-02544586

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Analyzing Accessibility Barriers using Cost-Benefit Analysis to Design Reliable Navigation Services for Wheelchair Users

Benjamin Tannert<fnref n="1" />, Reuben Kirkham<fnref n="2" />, Johannes Schöning<fnref n="1" />

**Abstract.** This paper explores 'A to B' routing tools designed to chart accessible routes for wheelchair users. We develop and present a novel measurement framework based upon cost-benefit analysis in order to evaluate the real-world utility of routing systems for wheelchair users. Using this framework, we compare proposed routes generated by accessibility tools with the pedestrian routes generated by Google Maps by means of conducting expert assessments of the situation on the ground. Relative to tools aimed at pedestrians, we find that these tools are not significantly more likely to produce an accessible route, and more often than not, they present longer routes that arise from imaginary barriers that do not exist in the real world. This analysis indicates how future routing tools for wheelchair users should be designed to ensure that they genuinely ameliorate the effects of accessibility barriers in the built environment.

**Keywords:** Accessibility; Disability; Wheelchair Users; Pedestrian Navigation; Routing;

## 1 Introduction and Motivation

Getting from 'A to B' is an important task in day-to-day life. It is a fundamental part of being integrated into wider society. For most people, this is something that can usually be taken for granted: the existing pedestrian paths are all accessible to them, and modern tools (e.g. Google Maps) operate seamlessly to plot an accurate route for them to travel to their desired destination. However, the same cannot be said for people with mobility impairments and other disabilities (e.g. people with visual impairments) that impact their movement within the built environment [8]. One group who are particularly disadvantaged in this regard are wheelchair users, who require a smoothly surfaced set of paths in order to safely drive a wheelchair. This is a substantial group of people with disabilities with a specific and well-defined set of accessibility needs, with there being 65 million people worldwide that are reliant on a wheelchair in their day-to-day lives [28]. This is more common in westernized countries: e.g. the Federal Statistical Office in Germany indicates that 1.5 million people use a wheelchair in

<footnote n="1">¹University of Bremen, Bibliothekstr. 5, 28359 Bremen, Germany
{btannert,schoening}@uni-bremen.de</footnote>

<footnote n="2">²Monash University, 900 Dandenong Rd, Caulfield East VIC 3145
reuben.kirkham@monash.edu</footnote>
Germany on a day-to-day basis [37], with a similar picture in the UK (1.2 million people [49]) and the USA (3.3 million people [48])

In response, there has been a range of research aimed at documenting the built environment, so accessibility barriers can be identified (and thus be avoided or addressed). One aspect of this work focuses upon the accessibility of sidewalks, including the detection of surface quality for wheelchair users. These measurements have been made using both technical means such as depth cameras or acceleration sensors (e.g. [11, 13, 16, 17, 24]), or direct inspections, be they via crowdsourcing [45], or more formal inspections by experts (e.g. [46, 51]). These approaches either suffer from limited accuracy (i.e. the automated or crowdsourced approaches) or a paucity of coverage (in respect of the formal inspections by experts) [19].

Accordingly, there is an inevitable limitation in respect of a routing system for wheelchair users: it will always be operating on the basis of imperfect and incomplete information (and thereby make misdirection's, see **Figure 1** for an example). An effective routing tool will be optimised to manage the risk that arises from this uncertainty. There have been nascent efforts to build routing systems for wheelchair users, including OpenRouteService [50], Routino [52], and Google has recently offered a separate prototype to serve wheelchair users [33]. There are also predecessors to these systems that were developed as academic prototypes (e.g. [4, 21, 26, 41]). Yet one limitation of these existing works is that they have not been subject to a real-world evaluation approach.

This paper makes two primary contributions. First, drawing upon both the literature concerning the experiences and needs of wheelchair users and separately the tool of cost-benefit analysis from economics, we develop a real-world evaluation approach for 'A to B' routing systems for wheelchair users. Second, we apply this framework to compare Google Maps – a *pedestrian* routing tool (and the most widely used mapping tool [30]) -- with two different Wheelchair Routing systems in a real-world scenario. As with [42], our emphasis is upon the metrics and appropriate means for evaluation. We find that Google Maps is more effective than the existing Wheelchair Routing systems in identifying accessible routes for wheelchair users. By analysing how Google Maps is more effective, we are able to propose algorithmic and practical improvements to future 'A to B' systems for wheelchair users, as well as proposing new directions for research in respect of the documentation of accessibility barriers.

## 2 'A to B': The Task of Wheelchair Routing

We frame the problem of wheelchair accessibility in respect of the 'A to B' problem, setting out the specific types of wheelchairs and wheelchair users, the effect of less than ideal driving conditions, and how this might in practice be mitigated. It must be emphasised that this is distinct to the *at A* or *at B* problem, which considers the accessibility of a location when someone *arrives* at it (e.g. does it have elevators to travel between floors?) and which is evaluated by a different set of tools (examples of which include Wheelmap [23] or Euan's Guide [47]). The aim is to broadly frame and capture the problem in question, so that we can fully justify our proposed approach towards the identification and measurement of barriers.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** A real example of a misdirection made by a wheelchair routing tool. On the left side of the street there is a narrow pavement with a horizontal slope (purple – diagonal strips). In addition, there is a street sign (yellow – horizontal strips) and a bin (red - squares) which hamper the use of this sidewalk. On the right side the narrow pavement has not only a horizontal slope but is also blocked by a bike (orange - full) and a car (pink – vertical strips). Even without the temporary barriers this sidewalk would not be usable for electric wheelchair drivers because of their narrow width. For manual wheelchair users it would only be possible with increased power and by travelling on the road itself (which itself presents substantial safety issues).</figcaption></figure>

**2.1 Living with and Navigating Around Barriers in the Built Environment**

In day-to-day life, an important task is getting from ‘A to B’, where A is one geographical location and B is another geographical location. This can be done using a broad range of transportation, including public transport or a private car, but in practice, there will be a portion of most trips that involve navigation of pedestrian routes of some kind (e.g. there are no roads leading to a location, a limited number of car parks and so forth). For a wheelchair user, those pedestrian routes may contain one or more obstacles: if they do, then they are either impassable, inconvenient or uncomfortable depending upon the person. The problem therefore is a matter of determining which route to take and how someone can know this in advance of traversing it, much as someone would use Google Maps. The distinction is that mapping software needs to take into account the barriers in determining which route to take and there is less of a possibility to ‘ask for directions’ (because most people are not wheelchair users and are therefore not familiar relevant with the barriers in the built environment). An example of a scenario involving a range of obstacles encountered after the failure of a routing system is provided in **Figure 1**.

There are substantial practical effects upon the day-to-day lives of manual wheelchair users. The most notable recent work that directly considers ‘A to B’ routing
systems is [12], which provides an extensive account of the way that Google Maps and other means are adapted by those with mobility impairments in order to assist with navigation (e.g. by performing ‘reccies’ to check routes in advance, or using Google Streetview to check a location): that work also proposed approaches for redesigning such systems. What is clear is that the uncertainty imposed upon people with mobility impairments – in respect of the accessibility of routes that are not already known to the person in question – has a substantial impact on their day-to-day lives, not only for the reasons given in [12], but also the wider socio-economic difficulties that often impact upon disabled people (e.g. a disproportionate number are on social security [22] and have low incomes, limiting the availability of alternatives such as taxis or more advanced mobility equipment).

## 2.2 Wheelchairs and Wheelchair users

There are a multiplicity of underlying reasons why someone might be a wheelchair user. The common ground is that they are unable (or less able) to ambulate using their legs due to one or more impairments. The most usual case is manual wheelchairs driven using the arms of the wheelchair user: this group make up around 85% of all wheelchair users [35]. Manual wheelchairs generally comprise two large rear wheels (with grips known as ‘pushrims’) for propulsion, supported by two smaller caster wheels at the front of the chair [5]. Typically, these wheelchairs are highly customised to the specific anatomical requirements of the individual user [39].

In some cases, there are additional features, for instance a one-armed propulsion mechanism for someone who has had a stroke (and only has full use of one side of their body), or the use of a battery pack for someone who has limited strength and less ability to climb gradients. In others, a manual wheelchair is propelled by a caregiver or assistant, for those who may be unable to operate the chair themselves (e.g. due to a comorbid cognitive impairment): in this case there can be considerable physical demands placed on this operator [1, 2]. For persons who lack physical strength, coordination or control in respect of their upper arms, or those who simply lack physical stamina due to a health condition, an electric wheelchair or scooter would be used: as with manual wheelchairs, there can be a considerable degree of customisation involved in the design of these systems: they are bespoke to the user [18, 20, 39]. In general, an electric wheelchair or scooter is a four wheeled vehicle which is powered electrically by one or more motors, and normally controlled by some kind of joystick or steering wheel.

## 2.3 The Direct Effect of an Individual Barrier in the Built Environment

Despite this breadth in wheelchair users and wheelchairs, there is a point of considerable commonality. This starts with the *ideal* driving conditions for *any* kind of wheelchair which are the same in general, which in turn means that an accessibility barrier can be well-defined with reference to be a *deviation* from this ideal. To be specific, these ideal driving conditions are a sufficiently flat, smooth surface (albeit with sufficient grip), with sufficient space to maneuver their chair. Anything outside of that has consequences for wheelchair users: the precise effects are different depending on the
individual and their abilities. They range from simple inconvenience, onto discomfort, gradual injury from repetitive interaction with the environment (or a more immediate injury were a feature to cause an accident), to impassibility. To provide a concrete understanding, we now describe the main types of barriers and the different effects that they can have.

* **Surface gradient.** If a slope is too steep, physical fitness limits driving, it also involves undue effort and exertion for those who are able to navigate this [6]. In respect of gradients perpendicular to the direction of travel (i.e. with a notable camber), these can be uncomfortable, especially for (two-handed) manual wheelchair users, as they have to propel the wheelchair in an unbalanced manner [14].

* **Surface deformation.** These typically take the form of an abrupt discontinuity in the surface, which raises the risk of a person driving an electric wheelchair getting stuck, or a manual wheelchair user falling out of the wheelchair. It is not just the risk in tripping that is detrimental, but also the additional concentration required in an effort to avoid these hazards (not all of which are readily visible).

* **Rough surface.** As well as being a potential trip hazard and increasing the work required by a manual wheelchair user, it also increases harmful whole-body vibration, with a long term negative effect upon health [9, 44].

* **Lack of a dropped curb (or curb cut).** These are impassable for existing electric wheelchairs but can be passed (uncomfortably) by 'hopping' in a manual wheelchair (presuming a suitably fit rider).

* **Stairs.** For obvious reasons, these are generally impassable to a wheelchair user.

* **Narrow pavement or passageway.** Wheelchairs are wider than most pedestrians and need a certain width in order to travel a given path. Accordingly, if a pavement is not wide enough (or is obstructed by a transient barrier such as a parked car or wheelie-bin), then a wheelchair driver may have to use the road (with the obvious risks) or the route is entirely inaccessible.

These features exclude barriers that go beyond wheelchair users: for example, whether there are sufficient tactile markings for visually impaired people [31], or if there is sufficient crowd management for people with relevant impairments (e.g. age related frailty or anxiety). By focussing upon a relatively fixed set of concrete barriers, our approach avoids subjectivity and enables a focus on the mechanical accessibility issues that apply specifically for wheelchair users.

# 3 Developing a Measurement Framework

## 3.1 The task of a (Wheelchair) Routing System

It is necessary to briefly set out the task of a pedestrian routing system and provide some terminology. A routing systems task is to plot a route between geographical location A and geographical location B. This route will be a sequence of paths. An *optimal*
route will be the one which allows the person to *most efficiently* traverse 'A to B'. Typically, 'most efficiently' means the *shortest distance*, however it can be important to take into account the *complexity* of the route (because navigation can itself take additional time and effort, especially when a route is unfamiliar or more complex [10, 15]). In some cases, other factors might be optimised, for example the flatness of the route. We will say a route is *effective* if it is traversable by the person traversing it, i.e. it is *not impassable*. For someone who is not a wheelchair user, that means that the route *exists* and is *not obstructed*, for example by construction work. For someone who is a wheelchair user, this means that the route is not obstructed *for them*: in addition to that, it means that there are no *accessibility barriers for them* (i.e. those features as identified in Section 2.3 or a subset of them, depending on the person) impeding their route. The task of a wheelchair routing system is therefore to provide an *optimal* route from 'A to B' that (i) *exists*, (ii) is *not obstructed* and (iii) has *no accessibility barriers* for the wheelchair user in question.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Examples of the types of errors that can be made by an 'A to B' wheelchair routing system. A Type I error (top), is where a longer route is proposed that presumably arises from a virtual or imaginary barrier. A Type II error (bottom) is where the routing algorithm sends a user down an inaccessible route, when there is an alternative accessible one.</figcaption></figure>
## 3.2 What errors can be made by a Wheelchair Routing System?

Following on from the task of a routing system, it is important to be more specific as to what an error actually is and how it might arise. With respect to a given wheelchair user there are two primary types of errors that can be made by such a tool: A *Type 1 error*, where a longer route is proposed that presumably arises from a ‘virtual’ or ‘imaginary’ accessibility barrier which does not physically exist (i.e. a suboptimal route), and a *Type 2 error*, where a route is declared to be wheelchair accessible, but in fact has substantial obstacles within it. These errors have different consequences. In respect of a Type 1 error, the wheelchair user unnecessarily spends more time going to their destination than they otherwise would. In some cases, this may also mean additional time navigating, because longer routes on average are more likely to be complicated with an increased number of turns, in turn requiring further in-situ references to the route in question [43]. In respect of a Type 2 error, the result is the person has to do a ‘go around’, i.e. amend their journey in real time. A ‘go around’ has two implications, both of which substantially reduce efficiency: the first is that time is wasted travelling a route that is inaccessible; the second is that time has to be taken in order to identify or investigate an alternative route. **Figure 2** depicts each type of error in turn.

## 3.3 Cost-Benefit Analysis

*Cost-benefit analysis* is a widely-used comparative approach used to make decisions between options based on economic analysis. More specifically, it is defined as “*the systematic and analytical process of comparing benefits and costs in evaluating the desirability of a project or programme*” [32]. In our case, this is whether a wheelchair user should use a given routing algorithm. Cost-benefit analysis is an *objective* balancing exercise: it involves determining the costs and benefits of a given option. However, there are a variety of ways as to which this might be done: for example, there are different ways of measuring cost and benefit, including in respect of the units of analysis employed (e.g. time, financial cost). This means a bespoke approach must be designed for each individual scenario. This is what we do next.

## 3.4 Making the Subjective Objective: Classifying Individual Routes for Populations

One difficulty in developing any framework is the diverse nature of wheelchair users and their particular impairments in respect of accessing the built environment: this is captured in **Section 2.2**. Moreover, how this can impact upon a given individual is innately subjective. However, this does not mean we cannot adopt an objective approach towards evaluating wheelchair routing systems. This is because we are considering *populations* of individuals, not individuals per se, because the systems are supposed to be designed for wheelchair users at large. The remedy is to consider extrema, because it offers a strong indication in respect of the intermediate space which many wheelchair users fit within. For our purposes, this means the following broad categories of wheelchair users, with **C1** and **C3** capturing the extrema:
* **C1 (Electric / All users)**: A route which is fully accessible for an electric wheelchair or scooter user, which means it does not contain one or more errors identified in **Section 2.3**. Note: a route that is accessible for an electric wheelchair user will almost always be accessible for most manual wheelchair users.

* **C2: (Manual, Not Electric)** A route that has one or more errors noted in C1, but is not inaccessible to a manual wheelchair user (i.e. there was no need for a 'go around' by them).

* **C3 (Not Accessible for all Wheelchair Users)**: A route which is impassable even to a highly skilled and able manual wheelchair user without any impairments asides in respect of using their legs to ambulate.

These categories can be readily considered pairwise to produce Type I and Type II errors, (i) Scenario 1 by comparing C1 vs C2 and C3 combined (hereon **electric** wheelchair user accessibility), and separately (ii) Scenario 2 by comparing C1 and C2 vs C3 (hereon **manual** wheelchair user accessibility). This provides two different scenarios where a cost-benefit analysis can be performed on a set of routing algorithms.

**3.5 Drawing this together: How to perform a Cost-Benefit Analysis**

For our purposes, cost-benefit analysis takes two routing algorithms at a time, RA1 and RA2 and then compares them. To do so, one needs to prepare a *sample*<fnref n="1" /> of ‘A’s and ‘B’s: i.e. two different geographic locations and then use RA1 and RA2 in turn to compute routes. These routes can then be evaluated in turn, by computing (i) the length and complexity of the routes in question and (ii) the proportion of routes in which there is an accessibility barrier encountered, with reference to Scenario 1 and Scenario 2. A simple cost-benefit analysis would then involve computing -- for each element of the sample -- the additional distance and complexity between RA1 and RA2 by quantifying it as a numerical time, before deducting time for additional go-arounds.

For example, one could take distance at 6km/h (the standard speed for moving in a wheelchair – thus 6000m = 1 hour), complexity at 5 seconds per turn (to take account of additional consultations of a smartphone) and a time penalty of 5 minutes for en-countering an accessibility barrier. Now suppose RA1 was 24 km (so 4 hours) over a sample of routes and RA2 was 30 km (so 5 hours), but RA1 involved 30 more turns than RA2 and encountered 16 extra barriers, then RA1 would perform marginally worse than RA2, because the 30 extra turns (150 seconds or 2 and half extra minutes) and the 16 additional detours (80 minutes) add up to more time taken in total.

<footnote n="1">¹ For an individual user, what would matter is the sample of routes (rather than focussing on one route) in any event. This is because an effective tool would be used regularly for navigation, as opposed on a 'one off' basis.</footnote>
# 4 Methodology: Applying our Framework in an Investigation of Wheelchair Routing Algorithms

As mentioned before, our method is a Cost-Benefit Analysis, following the framework set out in the foregoing section. Any analysis is based upon data. This work is no exception. We therefore collected a dataset for this investigation to which cost-benefit analysis could then be applied. There are three facets to this exercise: (i) the design of an expert assessment of routes to obtain a ground truth; (ii) the determination of how routes were to be generated using existing navigation tools and then (iii) how these routes were to be sampled (given the burdens of performing an expert investigation).

## 4.1 Making an Expert Assessment of a Route

Our goal was to produce an expert accessibility assessment of the routes generated by the mapping algorithms in order to act as an effective ground truth. Two manual wheelchair users with expertise in accessibility assessments independently traversed each of the routes in question to ensure that all the potential issues were captured. This ‘double annotation’ is in line with best practice when collecting accurate datasets [29]. The assessments took place in daytime to ensure sufficient visibility so that no features were occluded from the assessment. Whilst travelling the route, a detailed record was taken of all potential barriers (see the discussion in **Section 2.3** above) within the built environment for wheelchair users of all abilities. If the route was impassable in a manual wheelchair, this was recorded and then a ‘go around’ was performed to travel as much of the proposed route as possible. To help ensure the necessary objectivity, there was a detailed checklist that had to be completed for each trip. Any discrepancies were resolved between the two annotators. The resulting exercise involved the collection of an extensive amount of data, with 160km of routes being directly assessed by our two experts.

## 4.2 Navigation Tools

There are two existing systems for routing Wheelchair users from ‘A to B’, OpenRoute-Service (‘ORS’) [50] and Routino [52], which (i) presently operate in German cities and (ii) which also have a publicly available API that enables automated access to them. For each of these platforms, we generated 2715 routes. We adopt the approach towards generating origin-destination pairs as per Tannert et al. [38] This approach involves determining all possible routes ‘A’ and ‘B’ within a city, where in ‘A’ is a publicly available bathroom, and ‘B’ is a point of interest. Bathrooms are used because their locations are critical for wheelchair users in two respects: (i) the nature of many mobility impairments means an increased need to visit the bathroom (e.g. Spina Bifida is well known for having such an incidence [40]) and (ii) public accessible toilets are a rarity when compared to the facilities available for non-wheelchair users. Bremen was chosen because it is heavily populated, as well as having a variety of terrain (e.g. a medieval city centre) as to present a diverse range of challenges for a routing algorithm.
<figure><img src="image_4.png" /><figcaption>**Fig 3.** These images provide examples of the generated routes in respect of each case. At the top is Case 1 (near 100% overlap/ MAX), the middle is Case 2 (near 50% overlap/MID) and the bottom is Case 3 (near 0% overlap/MIN).</figcaption></figure>

## 4.3 Sampling approach

Our sampling approach has two goals. The first was to allow for an effective test of whether Type I or Type II errors (as defined in **Section 3.2**) impeded routes, so we
could gain an insight into how (or more accurately why) a system offered the routes that it did. The second was to ensure that the overall sample was reasonably representative of how a wheelchair user might use the system, thereby enabling cost-benefit analysis. The challenge was to have a process that efficiently used expert assessments, given the degree of effort involved in collecting (and annotating) the data, whilst also meeting these goals. The way we achieved this was to have three different groups of routes, of which 15 examples were sampled (making 45 pairs for each comparison, totalling around 160km of routes evaluated). These groups of routes were determined on the basis of the degree of overlap or alignment between RA1 and RA2 (where RA2 was always Google Maps): examples of each can be found in **Figure 3**:

1. **MAX (overlap)** the case where the route was closest to the one generated by Google Maps for fully ambulant pedestrians (i.e. starting from 100% overlap and descending until we have 15 routes). This is designed to help identify Type II errors, because it is highly probable that routes for wheelchair users that are equal to the routes of pedestrians could create a virtual barrier of Type 2.

2. **MID (overlap)** those in the middle (with ~50% overlap). This was added to provide a more balanced sample for cost-benefit analysis, given that cases 1 and 3 were extrema.

3. **MIN (overlap)** those routes with the least overlap (i.e. starting from 0% overlap and ascending until we had 15 routes). This is designed to help identify Type I errors, because if the routes for wheelchair users and pedestrians are almost different it is very possible that there could occur virtual barriers of Type 1.

To compute the degree of overlap between routes we used a fine grid to represent each path (with 10m by 10m cells) in order to identify which elements were overlapping and thus estimate the % overlap (which is different from the difference in distance) [34].

# 5 Results

## 5.1 The Routes Chosen by the Algorithms

We begin by summarizing the general (non-accessibility related) features of the routes we sampled. As one might expect (Figure 4), the routes for the minimum overlap (whose average length is over 1.5km) with Google Pedestrian are significantly longer than those with a maximum overlap (all of which are under 500m), albeit there is substantial variance in lengths. Another key feature (Figure 4) is that the routes generated by the wheelchair routing systems are on average both longer (in distance) and more complex (they have substantially more turns, indeed on average there are more than twice as many in each route). This is also to be expected: the algorithms of ORS<fnref n="2" />, and Routino<fnref n="3" /> both focus on (i) avoiding steps and (ii) providing paved routes whereby the ORS route has to be 100% paved and for the Routino route 90% must be paved.

<footnote n="2">² https://github.com/GIScience/openrouteservice-docs</footnote>

<footnote n="3">³ https://github.com/twinslash/routino/blob/master/xml/routino-profiles.xml</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig. 4:** A summary for the routes generated by each system, in respect of each sample, with its paired pedestrian route in Google Maps. Each graph presents the mean values, with the error bars being one standard error. A proportion of the variance is explained by the fact that Google had up-to-date information on construction sites that necessitated substantial details, but neither wheelchair routing system was aware of them.</figcaption></figure>
## 5.2 Are the Routes generated by a Wheelchair Routing System more likely to be Accessible?

The confusion matrices in **Figure 5** provide a comparison between Google and both of the mapping tools for each of the settings in question. The difference between the two approaches is mostly marginal, asides Routino's relative failure in respect of electric wheelchair users, and ORS being better (by providing three more accessible routes compared to Google) for manual wheelchair users. In total 39 of the 45 ORS routes were accessible for manual wheelchair users and 29 for electric wheelchair users. Out of the 45 Routino routes, 34 were accessible for manual wheelchair users and only 21 for electric wheelchair users. For electric wheelchair users, both tools less often provide a more accessible route, meaning that the tools were (insofar as our sample was concerned) less accurate than using Google Maps, especially in respect of Routino. This appears to be for the reason that (i) the routes in question are longer and involve more turns and (ii) there is more likely to be a missing dropped curb within a given route (this only needs to happen once for the route to fail): see **Table 1** which shows the increased number of average number of curbs and missing curbs (especially in respect of the longer routes with less overlap: i.e. Routino<sub>min</sub> and ORS<sub>min</sub>).

<figure><img src="image_6.png" /><figcaption>**Fig. 5:** Confusion Matrices comparing Google Maps Pedestrian Routing with the specialist Wheelchair Routing systems. The general picture is that under each condition, there is little or any practical difference between using a Wheelchair Routing system and simply using a general Pedestrian Routing system in terms of the accessibility (:Acc) or inaccessibility (:In) of the routes in question. The majority of the results are within the leading diagonal, i.e. neither system proposed a route that encountered an accessibility barrier.</figcaption></figure>

## 5.3 Our Cost-Benefit Analysis

In **Table 2** we compute the *additional time cost* (denoted [Z] in the table) per additional accessible route identified. We make assumptions that generally favour the ‘A to B’ systems, including that it would take 5 minutes (on average) to perform a ‘go around’ in respect of a given accessibility barrier and that it would take only five seconds extra per turn to navigate the route in question. We also provide the *success probability* (*denoted by [X] in Table 2 for Google, and [Y] for the respective routing system*): namely the proportion of routes that were completely passable for each type of wheelchair user.
<table><caption>**Table 1:** This table provides a summary of the number of missing dropped curbs in respect of each routing algorithm. It can be seen that Google Maps is generally more successful in avoiding them.</caption><thead><tr><th rowspan="2">Routes</th><th colspan="2">Total Number of Missing Dropped Curbs</th><th colspan="2">Total Number of Dropped Curbs</th><th colspan="2">Total Number of Curbs</th></tr><tr><th>Wheelchair Routing System</th><th>Google</th><th>Wheelchair Routing System</th><th>Google</th><th>Wheelchair Routing System</th><th>Google</th></tr></thead><tbody><tr><td>Routino_Max</td><td>11</td><td>11</td><td>40</td><td>40</td><td>51</td><td>51</td></tr><tr><td>Routino_Middle</td><td>7</td><td>4</td><td>109</td><td>116</td><td>116</td><td>120</td></tr><tr><td>RoutinoEVENT</td><td>16</td><td>4</td><td>280</td><td>240</td><td>296</td><td>244</td></tr><tr><td>ORS EVENT</td><td>5</td><td>5</td><td>23</td><td>23</td><td>28</td><td>28</td></tr><tr><td>ORS EVENT</td><td>4</td><td>8</td><td>127</td><td>158</td><td>131</td><td>166</td></tr><tr><td>ORS EVENT</td><td>15</td><td>6</td><td>251</td><td>216</td><td>266</td><td>222</td></tr><tr><td>RoutinoEVENT</td><td>11.33</td><td>6.33</td><td>143</td><td>132</td><td>154.3</td><td>138.3</td></tr><tr><td>ORS EVENT</td><td>8</td><td>6.33</td><td>133.7</td><td>132.3</td><td>141.7</td><td>138.7</td></tr><tr><td>Overall EVENT</td><td>9.67</td><td>6.33</td><td>138.3</td><td>132.2</td><td>148</td><td>138.5</td></tr></tbody></table>

<table><caption>**Table 2:** This table details the underlying cost-benefit analysis of each Wheelchair Routing Algorithm compared to Google Maps. As the Increased Accessibility Probability for Electric Wheelchairs is negative to begin with (and the routing tools produce routes that are already on average longer), the table is only complete for Manual Wheelchair Users. The same modelling assumptions are used as in our example in **Section 3.5**.</caption><thead><tr><th>Scenario</th><th>Google Success Probability [X]</th><th>'A to B' Success Probability [Y]</th><th>Average Additional Distance</th><th>Average Additional Turns</th><th>Increased Accessibility Probability</th><th>Time lost/ extra Accessible Route (at 6kph)</th><th>Time lost/ extra accessible route (assume 5sec/turn)</th><th>Total Time Cost per Extra Accessible Route [Z]</th></tr></thead><tbody><tr><td>ORS Electric</td><td>30/45</td><td>29/45</td><td>182m</td><td>4.11</td><td>-1/45</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>ORS Manual</td><td>36/45</td><td>39/45</td><td>182m</td><td>4.11</td><td>3/45</td><td>27m 45s</td><td>5 m 8 s</td><td>32m 53s</td></tr><tr><td>Routino Electric</td><td>29/45</td><td>21/45</td><td>118m</td><td>12.07</td><td>-8/45</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>Routino Manual</td><td>33/45</td><td>34/45</td><td>118m</td><td>12.07</td><td>1/45</td><td>53m 6s</td><td>44 m 15 s</td><td>1h 37m 21s</td></tr></tbody></table>
The best-case scenario is the Manual Wheelchair user with no other mobility impairments, as we have already found that the systems showed no improvement *at all* for Electric Wheelchair users (making the result of any cost benefit analysis be that Google Maps is more effective). For manual wheelchairs the ORS would need to be over *six* times more effective simply just to break-even (it takes over 30 minutes extra travelling time to avoid one accessibility barrier, so dividing by 5 minutes – our assumed time to perform a ‘go around’ gives us the figure in question), let alone to actually have a substantive impact on the wheelchair users chances of getting from ‘A’ to B’ in a timeous fashion. The picture is worse with Routino, which would need to be *over* 19 times more effective. We find that neither routing tool is effective compared to Google Maps under a cost-benefit analysis, even if profoundly more favorable assumptions were applied.

## 5.4 Virtual Barriers

As explained in our introduction, our concern was that these systems may be sending their users around the houses (i.e. on longer routes) to deal with imaginary accessibility barriers (i.e. making Type 1 errors). **Table 3** makes it clear that this is so. As can be seen from **Figure 6** the most aggressive interventions appear to be made on the occasions where only the wheelchair routing system produces an accessible route (especially with respect to Routino): similarly, the increased distance of using a wheelchair routing system is markedly reduced on occasions where only Google Maps is accessible. This suggests where longer routes are being used, there is a real accessibility barrier in play, for Google Maps, we observed this was where a construction site was missed by the Wheelchair routing tool (in total, we identified four such occasions).

# 6 Discussion

## 6.1 Google (Normally) Knows Best

The result of our cost-benefit analysis is surprising: it was better to use Google Maps than a specialist routing tool to travel as a wheelchair user, even for manual wheelchair users. As Google Maps was able to identify transient barriers, such as construction sites (whereas the wheelchair mapping tools directed our user straight through them), it has an evident informational advantage over the alternative tools. This informational advantage partly explains why Google Maps is more effective.

Accordingly, rather than striking out alone for wheelchair users, it might be more appropriate to focus upon improving Google Maps and similar platforms, or at the least, integrate these additional sources of information into the analysis offered by a wheelchair routing tool. This is particularly so given that Google Maps is more developed: it provides a range of features both in the moment (where someone discovers a barrier) and offline (e.g. Street View for inspections), meaning that the foregoing analysis is an underestimate of the increased burden imposed by wheelchair-routing tools. Whilst Google Maps may have a disadvantage in that it is not Open Source (unlike the two
<figure><img src="image_7.png" /><figcaption>**Fig. 6:** These two charts illustrate the additional travelling distance involved in using the tools in question, grouped by the same categories used in the confusion matrices.</figcaption></figure>

<table><caption>**Table 3:** This table enumerates the Types of Errors made in line with our Taxonomy in **Section 3.2.** of this document.</caption><thead><tr><th></th><th>Type I Error</th><th>Type II Error</th></tr></thead><tbody><tr><th>ORS</th><td>4</td><td>7</td></tr><tr><th>Routino</th><td>6</td><td>12</td></tr></tbody></table>

tools we used), the reason for its advantage is the level of resource which arises from it not be purely focused upon Wheelchair users: there is no reason to expect that this advantage would not apply to similarly resourced Open Source tools.

## 6.2 Real versus Synthetic Evaluations

There are a number of existing works that analyze the efficacy of a given routing algorithm on the basis of a synthetic evaluation. This includes two works which were the basis for the ORS system itself [26, 27]. Our investigation shows that in the real world, these synthetic analyses are not even sufficient to ensure that the routes offered are accessible, let alone the most effective ones. There are two specific flaws with these tools (i) they do not pick accessible routes often enough (which would have been identified if this was tested with a direct inspection) and (ii) they are not being optimized using metrics that are orientated from the perspective of the end user (i.e. they do not involve an effective cost-benefit analysis).

These are synthetic problems of different flavours, but ultimately the lesson is that some form of physical inspection process (i.e. obtaining ground truth) is required when evaluating any 'A to B' tool, as well as taking into account the real data that is likely to
be available when navigating, rather than some idealized expectation of it. There is a further implication, namely that these tools cannot be used as proxies for the general accessibility of a city as proposed in [38], because (unlike what someone would reasonably expect) they simply do not sufficiently reflect the real world.

## 6.3 Reporting Routing Options

There is always a risk that accessibility data will be inaccurate: even for expert assessments, accessibility information is as only as good as the time of the assessment. In practice, simply due to resource constraints, most data is not even gathered by experts, making it inherently unreliable at the point of collection. The real concern is one of high-level strategy: Google Maps (and similar systems) may offer more direct (and more up to date) information about additional barriers (e.g. construction sites) and routes, but there is a need to merge that with reliable accessibility information.

How should these two information sources be combined? We suggest that a starting point should be the lived experiences of wheelchair users using ‘A to B’ tools, as sharply captured in [12]. These experiences are multifarious in nature, being a combination of the impairment that the person in question has, as well as their wider life circumstances and the particular city that they are based. Ultimately, this must be an exercise in exposing choices and the risk associated with them, so wheelchair users can decide in advance whether its worth (for example) the risk of being ‘X minutes late’ or having to go Y metres further, rather than having to do ‘reccies’ in advance or laboriously exploring routes using Google StreetView [12]. Given that Wheelchair users sometimes use existing networks, the information arising from this could also be provided, including perhaps the sharing of location and acceleration data to the mutual benefit of other wheelchair users.

Making meaningful choices would involve a system exposing what it knows about the accessibility of given segments, before presenting it as a range of choices. Taking into account that Google Maps is likely to suggest a shorter route, the choices would in effect amount to proposed ‘detours’, together with the information known about that detour (e.g. how much longer, what accessibility barrier it might be avoiding and the probabilities involved). Overall, this would be an exercise in exposing trade-offs that recognizes the imperfection inherent in any accessibility tool. Other advanced algorithms have been successful in a range of complex route-planning contexts (e.g. for ships at sea [3], pedestrians travelling at night [7] or personalized routing [36]), so adapting such approaches may be a partial remedy for wheelchair mapping as well.

## 6.4 Cost-Benefit Analysis and the Bigger Picture

In so far as we know, this paper is the first attempt at using cost-benefit analysis in analyzing individual assistive technologies. It is therefore important to recap the benefits of this approach. In this case, the further analysis allowed for a consideration that took into account a more holistic set of factors and considered the true utility of a system to someone with a disability. Whilst any cost-benefit analysis (like any model) has its limitations (it only captures what is modelled), the success in this example may mean
that it could be appropriate for it to be considered more widely in respect of assistive technologies and thereby capture the wider benefits to individual lives. We consider this approach to be complementary to qualitative methods used to understand individual experiences and an efficient alternative where these studies might not be easily available (e.g. due to the well-known difficulty of easily accessing a pool of participants due to delays in ethical approval [25], or where a longer term consideration is required in advance of deployment), or at preliminary stages in design (thereby ensuring a better design to be tested and explored in a qualitative study or deployment).

## 6.5 Limitations

Our investigation was constrained to wheelchair users, meaning that its findings may not generalize to people with other impairments that limit their opportunity to access the built environment. This paper focusses on one city with a diverse landscape: future work may compare the effects identified within this paper with other cities. In particular, given that our city may have had limited accessibility, a more accessible city may well produce different results and be more (or less) effective a place for using ‘A to B’ tools (on the other hand, it is possible that the trade-off could in fact be worse). Similarly, where a city has been more (or less) thoroughly documented in respect of accessibility, this would also potentially impact on performance. For the comparison of the routes we chose Google Maps because it is the most widely used tool: future work may benefit with a comparison between other mapping tools (e.g. Apple Maps).

# 7 Conclusion and Future Work

We have explored the viability of ‘A to B’ mapping tools for wheelchair users, by developing a novel framework based on cost-benefit analysis in order to explore their real-world efficacy. In respect of existing tools aimed at wheelchair users the overall result is unfortunate: these tools are of little or no practical benefit when compared to Google Maps, with Routino also presenting routes that were less accessible overall. In arriving at this conclusion, we have also determined that the approaches used to evaluate ‘A to B’ tools are potentially misleading, and an economic approach based upon cost-benefit analysis from the perspective of wheelchair users would be more appropriate instead. Our analysis also suggests more appropriate means of presenting the data generated by these systems that reflects their real-world performance and the uncertainty inherent in that. It is hoped that this work would be the basis for ‘A to B’ mapping tools which are of genuine and long-lasting utility to Wheelchair users going forwards.

## Acknowledgements

The project was supported through the Volkswagen foundation through a Lichtenberg professorship and the University of Bremen. Reuben Kirkham was supported by an EPSRC Doctoral Prize Fellowship.
References

1. Abraham, B., Davidson, R.I.: A best space for assisted wheelchair users. British Journal of Occupational Therapy. 80, 3, 163–172 (2017).

2. Abraham, B.B. et al.: A novel four-caster manual vehicle manoeuvring investigation: Higher loading-weights require larger turning spaces. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine. 229, 5, 403–416 (2015).

3. Andersson, P., Ivehammar, P.: Cost Benefit Analysis of Dynamic Route Planning at Sea. Transportation Research Procedia. 14, 193–202 (2016). https://doi.org/10.1016/j.trpro.2016.05.055.

4. Beale, L. et al.: Mapping for wheelchair users: Route navigation in urban spaces. The Cartographic Journal. 43, 1, 68–81 (2006).

5. Boninger, M.L. et al.: Manual wheelchair pushrim biomechanics and axle position. Archives of physical medicine and rehabilitation. 81, 5, 608–613 (2000).

6. Choi, Y.O. et al.: Effects of ramp slope on physiological characteristic and performance time of healthy adults propelling and pushing wheelchairs. Journal of physical therapy science. 27, 1, 7–9 (2015).

7. Fang, Z. et al.: An artificial bee colony-based multi-objective route planning algorithm for use in pedestrian navigation at night. International Journal of Geographical Information Science. 31, 10, 2020–2044 (2017). https://doi.org/10.1080/13658816.2017.1346795.

8. Froehlich, J.E. et al.: Grand challenges in accessible maps. interactions. (2019).

9. Garcia-Mendez, Y. et al.: Health risks of vibration exposure to wheelchair users in the community. The journal of spinal cord medicine. 36, 4, 365–375 (2013).

10. Gartner, G. et al.: Human-centred mobile pedestrian navigation systems. Mitteilungen der Österreichischen Geographischen Gesellschaft. 153, 237–250 (2011).

11. Hara, K. et al.: Combining crowdsourcing and google street view to identify street-level accessibility problems. In: Proceedings of the SIGCHI conference on human factors in computing systems. pp. 631–640 ACM (2013).

12. Hara, K. et al.: The design of assistive location-based technologies for people with ambulatory disabilities: A formative study. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 1757–1768 ACM (2016).

13. Holloway, C., Tyler, N.: A micro-level approach to measuring the accessibility of footways for wheelchair users using the Capability Model. Transportation planning and technology. 36, 7, 636–649 (2013).

14. Ishida, T. et al.: Evaluation of sidewalk unevenness based on wheelchair traveling resistance. Transportation Research Record: Journal of the Transportation Research Board. 1956, 68–75 (2006).

15. Ishikawa, T. et al.: Wayfinding with a GPS-based mobile navigation system: A comparison with maps and direct experience. Journal of Environmental Psychology. 28, 1, 74–82 (2008).

16. Iwasawa, Y. et al.: Combining Human Action Sensing of Wheelchair Users and Machine Learning for Autonomous Accessibility Data Collection. IEICE Transactions on Information and Systems. 99, 4, 1153–1161 (2016).
17. Iwasawa, Y., Yairi, I.E.: Life-logging of wheelchair driving on web maps for visualizing potential accidents and incidents. In: Pacific Rim International Conference on Artificial Intelligence. pp. 157–169 Springer (2012).

18. Kirkham, R. et al.: Build Me a Ubicomp: Bespoke Ubiquitous Accessibility in Live Television Production Environments. Pervasive and Ubiquitous Computing. 19, 5–6, 853–880 (2015).

19. Kirkham, R. et al.: WheelieMap: An Exploratory System for Qualitative Reports of Inaccessibility in the Built Environment. In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. p. 38 ACM (2017).

20. Maltais, C. et al.: Assessment of geometric and mechanical parameters in wheelchair seating: a variability study. IEEE Transactions on Rehabilitation Engineering. 7, 1, 91–98 (1999).

21. Matthews, H. et al.: Modelling Access with GIS in Urban Systems (MAGUS): capturing the experiences of wheelchair users. Area. 35, 1, 34–45 (2003).

22. Meyer, B.D., Mok, W.K.: Disability, earnings, income and consumption. Journal of Public Economics. (2018).

23. Mobasheri, A. et al.: Wheelmap: The wheelchair accessibility crowdsourcing platform. Open Geospatial Data, Software and Standards. 2, 1, 27 (2017).

24. Mourcou, Q. et al.: Wegoto: A Smartphone-based approach to assess and improve accessibility for wheelchair users. In: IEEE EMBS 2013. pp. 1194–1197 IEEE (2013).

25. Munteanu, C. et al.: Situational Ethics: Re-thinking Approaches to Formal Ethics Requirements for Human-Computer Interaction. In: CHI 2015. pp. 105–114 (2015). https://doi.org/10.1145/2702123.2702481.

26. Neis, P.: Measuring the reliability of wheelchair user route planning based on vol-unteered geographic information. Transactions in GIS. 19, 2, 188–201 (2015).

27. Neis, P., Zielstra, D.: Generation of a tailored routing network for disabled people based on collaboratively collected geodata. Applied Geography. 47, 70–77 (2014).

28. Newdisability.: Wheelchair Statistics: How Many Wheelchair Users Are There? http://www.newdisability.com/wheelchairstatistics.htm.

29. Nowak, S., Rüger, S.: How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In: Proceedings of the international conference on Multimedia information retrieval. pp. 557–566 ACM (2010).

30. Panko, Riley: The Popularity of Google Maps: Trends in Navigation Apps in 2018, https://themanifest.com/app-development/popularity-google-maps-trends-navigation-apps-2018.

31. Parkin, J., Smithies, N.: Accounting for the needs of blind and visually impaired people in public realm design. Journal of Urban Design. 17, 1, 135–149 (2012).

32. Quah, E., Haldane, J.: Cost-benefit analysis. Routledge (2007).

33. Sasha Blair-Goldensohn: A better world for wheels on Google Maps. https://www.blog.google/products/maps/better-world-wheels.google-maps/, (2017).
34. Shirabe, T.: A method for finding a least-cost wide path in raster space. International Journal of Geographical Information Science. 30, 8, 1469–1485 (2016).

35. Smith, E.M. et al.: Prevalence of wheelchair and scooter use among community-dwelling Canadians. Physical therapy. 96, 8, 1135–1142 (2016).

36. Socharoentum, M., Karimi, H.A.: Multi-modal transportation with multi-criteria walking (MMT-MCW): Personalized route recommender. Computers, Environment and Urban Systems. 55, 44–54 (2016). https://doi.org/10.1016/j.compenvurbsys.2015.10.005.

37. Statistisches Bundesamt: https://www.desta-tis.de/DE/Publikationen/Thematisch/Gesundheit/BehinderteMenschen/SozialSchwerbehinderteKB5227101159004.

38. Tannert, B., Schöning, J.: Disabled, but at what cost?: an examination of wheelchair routing algorithms. In: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services. p. 46 ACM (2018).

39. Trefler, E., Taylor, S.: Prescription and positioning: evaluating the physically disabled individual for wheelchair seating. Prosthetics and Orthotics International. 15, 3, 217–224 (1991).

40. Verhoef, M. et al.: High prevalence of incontinence among young adults with spina bifida: description, prediction and problem perception. Spinal Cord. 43, 6, 331 (2005).

41. Völkel, T., Weber, G.: RouteCheckr: Personalized Multicriteria Routing for Mobility Impaired Pedestrians. In: ASSETS 2008. pp. 185–192 ACM, New York, NY, USA (2008). https://doi.org/10.1145/1414471.1414506.

42. Ward, J.A. et al.: Performance Metrics for Activity Recognition. ACM Trans. In-tell. Syst. Technol. 2, 1, 6:1–6:23 (2011). https://doi.org/10.1145/1889681.1889687.

43. Winter, S.: Modeling costs of turns in route planning. GeoInformatica. 6, 4, 345–361 (2002).

44. Wolf, E. et al.: Vibration exposure of individuals using wheelchairs over sidewalk surfaces. Disability and Rehabilitation. 27, 23, 1443–1449 (2005).

45. AXS Map, https://www.axsmap.com/.

46. DisabledGo, http://www.disabledgo.com/.

47. Euan's Guide - Disabled Access Reviews, https://www.euansguide.com/.

48. KDS Smartwheelchair https://kdsmartchair.com/blogs/news/18706123-wheelchair-facts-numbers-and-figures-infographic.

49. NHS Wheelchair Services Statistics: https://www.england.nhs.uk/wheelchair-services/.

50. OpenRouteService (https://www.openrouteservice.org).

51. PhotoRoute, http://www.photoroute.com/.

52. Routino (https://www.routino.org/).