DennisOSRM/Project-OSRM — Issue #95: discussion: cucumber tests
Author: emiltin
Created: 2012-01-29 11:05:53+00:00
State: open

i've looked a bit more at the cucumber tests. the idea is to enable test based on small, well-known datasets. here's what tests look like:

´
@routing @basic
Feature: Basic Routing

Scenario: Smallest possible datasat
Given the nodes
| a | b |

And the ways
| nodes |
| ab |

When I route I should get
| from | to | route |
| a | b | ab |
| b | a | ba |

Scenario: Connected ways
Given the nodes
| a | | c |
| | b | |

And the ways
| nodes |
| ab |
| bc |

When I route I should get
| from | to | route |
| a | c | abc |
| c | a | cba |
| a | b | ab |
| b | a | ba |
| b | c | bc |
| c | b | cb |

Scenario: Unconnected ways
Given the nodes
| a | b |
| c | d |

And the ways
| nodes |
| ab |
| cd |

When I route I should get
| from | to | route |
| a | b | ab |
| b | a | ba |
| c | d | cd |
| d | c | dc |
| a | c | |
| c | a | |
| b | d | |
| d | c | |
| a | d | |
| d | a | |

Scenario: Pick the fastest way type
Given the nodes
| a | s |
| p | b |

And the ways
| nodes | highway |
| apb | primary |
| asb | secondary |

When I route I should get
| from | to | route |
| a | b | apb |
| b | a | bpa |

@routing @restrictions
Feature: Turn restrictions
OSRM should handle turn restrictions as defined by http://wiki.openstreetmap.org/wiki/Relation:restriction

Scenario: No left turn at T-junction
Given the nodes
| a | j | b |
| | s | |

And the ways
| nodes	|
| aj	|
| jb	|
| sj	|

And the relations
| from	| to	| via	| restriction	|
| sj	| ja	| j	| no_left_turn	|

When I route I should get
| from	| to	| route	|
| a	| b	| ajb	|
| a	| s	| ajs	|
| b	| a	| bja	|
| b	| s	| bjs	|
| s	| a	|	|
| s	| b	| sjb	|
`

what goes on is:
1. first you define a small node map. nodes are labelled with letters.
2. then you define ways connecting the nodes. you can set any tags you want on ways.
3. cucumber then writes he corresponding osm and speedprofile files, and processes them.
4. the routing examples are them run one at a time, checking against expected behaviour.

here's an example run:

`
~/code/Project-OSRM$ cucumber --tags @basic
@routing @basic
Feature: Basic Routing

Scenario: Smallest possible datasat # features/basic.feature:4
Given the nodes # features/step_definitions/test_data.rb:124
| a | b |
And the ways # features/step_definitions/test_data.rb:139
| nodes |
| ab |
When I route I should get # features/step_definitions/routing.rb:167
| from | to | route |
| a | b | ab |
| b | a | ba |
| a | b | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| b | a | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
Tables were not identical (Cucumber::Ast::Table::Different)
./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
features/basic.feature:12:in `When I route I should get'

Scenario: Connected ways # features/basic.feature:17
Given the nodes # features/step_definitions/test_data.rb:124
| a | | c |
| | b | |
And the ways # features/step_definitions/test_data.rb:139
| nodes |
| ab |
| bc |
When I route I should get # features/step_definitions/routing.rb:167
| from | to | route |
| a | c | abc |
| c | a | cba |
| a | c | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| a | b | ab |
| c | a | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| b | a | ba |
| a | b | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| b | c | bc |
| b | a | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| c | b | cb |
| b | c | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| c | b | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
Tables were not identical (Cucumber::Ast::Table::Different)
./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
features/basic.feature:27:in `When I route I should get'

Scenario: Unconnected ways # features/basic.feature:36
Given the nodes # features/step_definitions/test_data.rb:124
| a | b |
| c | d |
And the ways # features/step_definitions/test_data.rb:139
| nodes |
| ab |
| cd |
When I route I should get # features/step_definitions/routing.rb:167
| from | to | route |
| a | b | ab |
| b | a | ba |
| a | b | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| c | d | cd |
| b | a | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| d | c | dc |
| c | d | |
| a | c | (i) "" |
| d | c | (i) nil |
| c | a | (i) "" |
| a | c | (i) nil |
| b | d | |
| c | a | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
| d | c | (i) "" |
| b | d | (i) nil |
| a | d | (i) "" |
| d | c | (i) nil |
| d | a | (i) "" |
| a | d | (i) nil |
| d | a | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
Tables were not identical (Cucumber::Ast::Table::Different)
./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
features/basic.feature:46:in `When I route I should get'

Scenario: Pick the fastest way type # features/basic.feature:59
Given the nodes # features/step_definitions/test_data.rb:124
| a | s |
| p | b |
And the ways # features/step_definitions/test_data.rb:139
| nodes | highway |
| apb | primary |
| asb | secondary |
When I route I should get # features/step_definitions/routing.rb:167
| from | to | route |
| a | b | apb |
| b | a | bpa |
| a | b | Empty route: [["Head", "", 111, 0, 266, "0", "NE", 22.5]] |
| b | a | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
Tables were not identical (Cucumber::Ast::Table::Different)
./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
features/basic.feature:69:in `When I route I should get'

Failing Scenarios:
cucumber features/basic.feature:4 # Scenario: Smallest possible datasat
cucumber features/basic.feature:17 # Scenario: Connected ways
cucumber features/basic.feature:36 # Scenario: Unconnected ways
cucumber features/basic.feature:59 # Scenario: Pick the fastest way type

4 scenarios (4 failed)
12 steps (4 failed, 8 passed)
1m56.321s
`

all tests are currently failing due to #88 (missing way names in routing instructions).

launching and quitting the binaries currently takes a while (perhaps due to large ram/disk allocations?) which makes the tests rather slow.

perhaps an option could be added to allocate small amounts when testing?

another idea would be to merge all three binaries into one, which could be kept running. it would then be instructed (perhaps by touching a file) to reprocess data. this would have the added benefit that request during reprocessing could be handled gracefully by sending back an appropriate status code.