DennisOSRM/Project-OSRM — pull request #114 by emiltin (2012-02-11 13:28:54+00:00)
Title: cleaned up cucumber tests

here's a cleaned up version of cucumber tests - single commit rebased on your master.

it's quite a big chunk with many pieces, so take a look, and hopefully we can iron out any issues, so it can become a help in finding and fixing bugs.

Prerequisites:
ruby: should be preinstalled on most systems.
gems: cucumber, rake, osmlib-base. the easiest and safest is to first install the 'bundler' gem, then run 'bundle install' from the project folder.
osmosis: command line tool for converting and cropping osm files. if you're on mac you can install it via homebrew with 'brew install osmosis'

if you don't want to run tests, you should still be able to build and run osrm without installing any of the above.

SConstruct:
the SConstruct file had to rearranged a bit to get os x compilation to work. other platforms should not be affected, but i couldn't test it on those other platforms. compilation on mac does not currently use openmp.

Files:
test are located in features/ and organized into feature files, each containing scenarios.

test and sandbox are now two separate folders. test is where all the tests are run. sandbox is where you can use rake task to download real osm data, run the server manually to test in a browser, etc.

speedprofile.ini has been split into separate files, stored in speedprofiles/

both test/ and sandbox/ folder contains a .stxxl config file. the one in the test folder uses a 1MB disk, the one in sandbox uses a 1GB disk.

Mechanics:
for each test scenario, an osm data file is constructed based on the description in the test scenario. the osm file is converted to protobuffer and preprocessed with osrm-extract and osrm-prepare.

the resulting osrm files are cached, so that processing is only repeated if either the osm data, the speedprofile or the binaries have changed. this helps speed up the running of tests a lot.

test/server.ini and test/speedprofile.ini is now rewritten to ensure that the right data files are used.

osrm-routed is then launched, and routes are requested via http calls to port 5000 on localhost, and the response is compared to the expected result.

all request are timed out after 5 seconds. both timeouts and crashes are caught and reported.

to ensure a clean run of every test, all processes named osrm-extract, osrm-prepare and osrm-routed are killed before and after each test using a signal 9, no matter who launched them. i have only tested this on mac os x 10.7. it should work on linux, but process management can often behave differently depending on platform. test are not yet expected to work on windows, for example the unix shell command 'ps' is used. i assume there will be other issue too, but if course it should be possible to get it working.

Writing tests:
tests come in two main flavors. one where you specify data and test routes separately, and one where you just provide a table of ways and their tags to be tests.

with the first type, you setup the data by a small 'node map', and a list of ways (and optionally relations). finally you specify a list of start/end pairs, and the expected route, described by the ways traversed:

Scenario: Zig Zag
Given the nodes
| a | | c |
| | b | |

And the ways
| nodes |
| ab |
| bc |

When I route I should get
| from | to | route |
| a | c | ab,bc |
| c | a | bc,ab |

by default, tests use the bicycle speedprofile, but this can be changed for each scenario by adding:

Given the speedprofile "car"

you can also modify the speedprofile by using:

And the speedprofile settings
| obeyOneways | no |
| primary | 50 |

note that 'Given' and 'And' is interchangeable in cucumber, you should just use what creates a natural language flow.

the second flavour of tests simply lists ways and their tags:

Scenario: Simplest possible oneway
Given the defaults
Then routability should be
| highway | oneway | forw | backw |
| primary | | x | x |
| primary | yes | x | |
| primary | -1 | | x |

an osm file is constructed, containing an isolated way for each row. columns are converted to tags. forw and backw has special meaning; if an forw column is present the way is tested for forward routability. if an backw row is present, it's tested for backwards routability.

nodes are spaced on grid with origin at 1,1 and a grid size of 100m

Running tests:
You can run all tests, or specific sets or scenarios.

to run all test: cucumber (or 'rake test')
to run test with a specific tag, use: cucumber --tags @restrictions
to run a specific scenario: cucumber features/restrictions.feature:6

routes are valided by parsing the instructions hash. osrm currently often return empty routes, which is one of the main reasons a lot of test are currently failing.

during test, several log files are used. output from osrm-extract and osrm-prepare are directed to test/preprocess.log. output from osrm-routed is directed to test/osrm-routed.log.

tests results are shown in the terminal. green rows indicates a passed route. yellow is an expected, but missing, result, while grey is an unexpected result.

there's an issue with the order failing rows are displayed, which can be somewhat confusing - the failing row is not directly beneauth the expected row.

additional info is logged to fail.log, containing details about the osm data used, the speedprofile, expected and returned routes, as well as actual query and response involved.

there's currently a lot of failing tests. i'm sure some are caused by incorrect test cases or bugs in the cucumber setup. we just need to iron things out.

Rake task:
a few useful rake task are included. you can run 'rake -T' to get a list of rake commands.

for example, you can run 'rake download ' to download and crop osm data into your sandbox folder. curently only a few fixed areas work, including:

kbh: openhagen
dk: denmark

after downloading, you can run 'rake process' to process the downloaded data.

to run your server in the terminal, use 'rake run'. you must download and process data beforehand.

you can also run the server in the background with 'rake up' and 'rake down'. output is directed to osrm-routed.log. again you must download and process data beforehand. note however, that the include index.html file doesn't point to localhost, which is what you need to use the server you run locally.