mbedmicro/mbed — pull request #719 by PrzemekWirkus (2014-11-18 10:38:06+00:00)
Title: Host test plugin improvements, part 2: Copy plugins

# Description
This is second and last test suite modification related to reset / copy plugin architecture. Previous was done in https://github.com/mbedmicro/mbed/pull/597 pull request.

## Changes
* New copy / reset scheme introduced where plugins are implementing how to reset / flash mbed enabled device.
* Moved copy, reset and serial port management to host_test.
* added sew command line switched to pass e.g. test binary image location.
* Cleaned unnecessary host_test functionality (dead code).
* Refactored network tests to align with new reset / copy scheme.
* Function test() and run() usage in host tests is now more unified - code is a bit more readable.
* Few tests and minor defects corrected on the way (test / test suite stability).

## Testing
I checked new functionality on our CI server with many MUTs:
(Some of the results below)
```
Test summary:
+---------+----------+-----------+-----------+---------------------------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+---------+----------+-----------+-----------+---------------------------------------+--------------------+---------------+-------+
| OK | LPC11U24 | uARM | DTCT_1 | Simple detect test | 3.91 | 10 | 1/1 |
| OK | LPC11U24 | uARM | EXAMPLE_1 | /dev/null | 6.91 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_10 | Hello World | 3.74 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_11 | Ticker Int | 14.78 | 20 | 1/1 |
| OK | LPC11U24 | uARM | MBED_12 | C++ | 5.02 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_16 | RTC | 7.93 | 15 | 1/1 |
| OK | LPC11U24 | uARM | MBED_2 | stdio | 4.21 | 20 | 1/1 |
| OK | LPC11U24 | uARM | MBED_22 | Semihost | 4.1 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_23 | Ticker Int us | 14.87 | 15 | 1/1 |
| OK | LPC11U24 | uARM | MBED_24 | Timeout Int us | 14.98 | 15 | 1/1 |
| OK | LPC11U24 | uARM | MBED_25 | Time us | 14.82 | 15 | 1/1 |
| OK | LPC11U24 | uARM | MBED_26 | Integer constant division | 4.1 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_34 | Ticker Two callbacks | 14.9 | 15 | 1/1 |
| OK | LPC11U24 | uARM | MBED_A1 | Basic | 4.06 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_A2 | Semihost file system | 4.84 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_A21 | Call function before main (mbed_main) | 4.68 | 10 | 1/1 |
| OK | LPC11U24 | uARM | MBED_A9 | Serial Echo at 115200 | 9.3 | 10 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_1 | Basic thread | 15.0 | 15 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_2 | Mutex resource lock | 15.09 | 20 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_3 | Semaphore resource lock | 12.03 | 20 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_4 | Signals messaging | 9.27 | 10 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_5 | Queue messaging | 5.94 | 10 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_6 | Mail messaging | 6.26 | 10 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_7 | Timer | 15.06 | 15 | 1/1 |
| OK | LPC11U24 | uARM | RTOS_8 | ISR (Queue) | 10.05 | 10 | 1/1 |
| OK | LPC1768 | ARM | DTCT_1 | Simple detect test | 4.26 | 10 | 1/1 |
| OK | LPC1768 | ARM | EXAMPLE_1 | /dev/null | 7.32 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_10 | Hello World | 4.14 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_11 | Ticker Int | 15.17 | 20 | 1/1 |
| OK | LPC1768 | ARM | MBED_12 | C++ | 5.32 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_16 | RTC | 8.29 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_2 | stdio | 4.52 | 20 | 1/1 |
| OK | LPC1768 | ARM | MBED_22 | Semihost | 5.31 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_23 | Ticker Int us | 15.09 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_24 | Timeout Int us | 15.23 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_25 | Time us | 15.07 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_26 | Integer constant division | 4.26 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_34 | Ticker Two callbacks | 15.04 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_A1 | Basic | 4.23 | 10 | 1/1 |
| TIMEOUT | LPC1768 | ARM | MBED_A18 | Interrupt vector relocation | 20.84 | 10 | 0/1 |
| OK | LPC1768 | ARM | MBED_A2 | Semihost file system | 5.48 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_A21 | Call function before main (mbed_main) | 4.82 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_A9 | Serial Echo at 115200 | 9.61 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_1 | Basic thread | 15.38 | 15 | 1/1 |
| OK | LPC1768 | ARM | RTOS_2 | Mutex resource lock | 14.5 | 20 | 1/1 |
| OK | LPC1768 | ARM | RTOS_3 | Semaphore resource lock | 12.5 | 20 | 1/1 |
| OK | LPC1768 | ARM | RTOS_4 | Signals messaging | 9.44 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_5 | Queue messaging | 6.43 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_6 | Mail messaging | 6.52 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_7 | Timer | 15.29 | 15 | 1/1 |
| OK | LPC1768 | ARM | RTOS_8 | ISR (Queue) | 9.42 | 10 | 1/1 |
| OK | LPC1768 | uARM | DTCT_1 | Simple detect test | 4.09 | 10 | 1/1 |
| OK | LPC1768 | uARM | EXAMPLE_1 | /dev/null | 6.97 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_10 | Hello World | 3.9 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_11 | Ticker Int | 14.86 | 20 | 1/1 |
| OK | LPC1768 | uARM | MBED_12 | C++ | 4.12 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_16 | RTC | 7.93 | 15 | 1/1 |
| OK | LPC1768 | uARM | MBED_2 | stdio | 4.34 | 20 | 1/1 |
| OK | LPC1768 | uARM | MBED_22 | Semihost | 4.13 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_23 | Ticker Int us | 14.77 | 15 | 1/1 |
| OK | LPC1768 | uARM | MBED_24 | Timeout Int us | 14.84 | 15 | 1/1 |
| OK | LPC1768 | uARM | MBED_25 | Time us | 14.7 | 15 | 1/1 |
| OK | LPC1768 | uARM | MBED_26 | Integer constant division | 4.04 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_34 | Ticker Two callbacks | 14.76 | 15 | 1/1 |
| OK | LPC1768 | uARM | MBED_A1 | Basic | 3.98 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_A18 | Interrupt vector relocation | 9.19 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_A2 | Semihost file system | 5.06 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_A21 | Call function before main (mbed_main) | 4.21 | 10 | 1/1 |
| OK | LPC1768 | uARM | MBED_A9 | Serial Echo at 115200 | 9.33 | 10 | 1/1 |
| OK | LPC1768 | uARM | RTOS_1 | Basic thread | 14.81 | 15 | 1/1 |
| OK | LPC1768 | uARM | RTOS_2 | Mutex resource lock | 14.25 | 20 | 1/1 |
| OK | LPC1768 | uARM | RTOS_3 | Semaphore resource lock | 12.13 | 20 | 1/1 |
| OK | LPC1768 | uARM | RTOS_4 | Signals messaging | 9.08 | 10 | 1/1 |
| OK | LPC1768 | uARM | RTOS_5 | Queue messaging | 6.1 | 10 | 1/1 |
| OK | LPC1768 | uARM | RTOS_6 | Mail messaging | 6.15 | 10 | 1/1 |
| OK | LPC1768 | uARM | RTOS_7 | Timer | 14.78 | 15 | 1/1 |
| OK | LPC1768 | uARM | RTOS_8 | ISR (Queue) | 9.1 | 10 | 1/1 |
+---------+----------+-----------+-----------+---------------------------------------+--------------------+---------------+-------+
Result: 76 OK / 1 TIMEOUT

Completed in 716.10 sec
```

```
Test summary:
+--------+---------+-----------+-----------+---------------------------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+-----------+---------------------------------------+--------------------+---------------+-------+
| OK | K64F | ARM | DTCT_1 | Simple detect test | 6.76 | 10 | 1/1 |
| OK | K64F | ARM | EXAMPLE_1 | /dev/null | 9.81 | 10 | 1/1 |
| OK | K64F | ARM | MBED_10 | Hello World | 6.6 | 10 | 1/1 |
| OK | K64F | ARM | MBED_11 | Ticker Int | 16.68 | 20 | 1/1 |
| OK | K64F | ARM | MBED_12 | C++ | 7.71 | 10 | 1/1 |
| OK | K64F | ARM | MBED_16 | RTC | 10.91 | 15 | 1/1 |
| OK | K64F | ARM | MBED_2 | stdio | 7.08 | 20 | 1/1 |
| OK | K64F | ARM | MBED_23 | Ticker Int us | 16.62 | 15 | 1/1 |
| OK | K64F | ARM | MBED_24 | Timeout Int us | 16.75 | 15 | 1/1 |
| OK | K64F | ARM | MBED_25 | Time us | 16.59 | 15 | 1/1 |
| OK | K64F | ARM | MBED_26 | Integer constant division | 7.69 | 10 | 1/1 |
| OK | K64F | ARM | MBED_34 | Ticker Two callbacks | 16.7 | 15 | 1/1 |
| OK | K64F | ARM | MBED_A1 | Basic | 7.63 | 10 | 1/1 |
| OK | K64F | ARM | MBED_A21 | Call function before main (mbed_main) | 7.66 | 10 | 1/1 |
| OK | K64F | ARM | MBED_A9 | Serial Echo at 115200 | 12.14 | 10 | 1/1 |
| OK | K64F | ARM | RTOS_1 | Basic thread | 18.18 | 15 | 1/1 |
| OK | K64F | ARM | RTOS_2 | Mutex resource lock | 18.46 | 20 | 1/1 |
| OK | K64F | ARM | RTOS_3 | Semaphore resource lock | 15.44 | 20 | 1/1 |
| OK | K64F | ARM | RTOS_4 | Signals messaging | 13.26 | 10 | 1/1 |
| OK | K64F | ARM | RTOS_5 | Queue messaging | 9.33 | 10 | 1/1 |
| OK | K64F | ARM | RTOS_6 | Mail messaging | 9.36 | 10 | 1/1 |
| OK | K64F | ARM | RTOS_7 | Timer | 18.12 | 15 | 1/1 |
| OK | K64F | ARM | RTOS_8 | ISR (Queue) | 13.38 | 10 | 1/1 |
| OK | LPC1768 | ARM | DTCT_1 | Simple detect test | 4.38 | 10 | 1/1 |
| OK | LPC1768 | ARM | EXAMPLE_1 | /dev/null | 7.21 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_10 | Hello World | 4.34 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_11 | Ticker Int | 15.12 | 20 | 1/1 |
| OK | LPC1768 | ARM | MBED_12 | C++ | 5.24 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_16 | RTC | 8.28 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_2 | stdio | 4.51 | 20 | 1/1 |
| OK | LPC1768 | ARM | MBED_22 | Semihost | 5.29 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_23 | Ticker Int us | 15.1 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_24 | Timeout Int us | 15.19 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_25 | Time us | 15.06 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_26 | Integer constant division | 5.23 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_34 | Ticker Two callbacks | 15.15 | 15 | 1/1 |
| OK | LPC1768 | ARM | MBED_A1 | Basic | 4.24 | 10 | 1/1 |
| FAIL | LPC1768 | ARM | MBED_A18 | Interrupt vector relocation | 7.38 | 10 | 0/1 |
| OK | LPC1768 | ARM | MBED_A2 | Semihost file system | 5.49 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_A21 | Call function before main (mbed_main) | 4.21 | 10 | 1/1 |
| OK | LPC1768 | ARM | MBED_A9 | Serial Echo at 115200 | 9.55 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_1 | Basic thread | 15.46 | 15 | 1/1 |
| OK | LPC1768 | ARM | RTOS_2 | Mutex resource lock | 15.71 | 20 | 1/1 |
| OK | LPC1768 | ARM | RTOS_3 | Semaphore resource lock | 12.53 | 20 | 1/1 |
| OK | LPC1768 | ARM | RTOS_4 | Signals messaging | 9.47 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_5 | Queue messaging | 6.49 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_6 | Mail messaging | 6.46 | 10 | 1/1 |
| OK | LPC1768 | ARM | RTOS_7 | Timer | 15.26 | 15 | 1/1 |
| OK | LPC1768 | ARM | RTOS_8 | ISR (Queue) | 9.59 | 10 | 1/1 |
+--------+---------+-----------+-----------+---------------------------------------+--------------------+---------------+-------+
Result: 1 FAIL / 48 OK

Completed in 528.25 sec
```

```
Test summary:
+--------+---------+-----------+---------+----------------------------+--------------------+---------------+-------+
| Result | Target | Toolchain | Test ID | Test Description | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+----------------------------+--------------------+---------------+-------+
| OK | K64F | ARM | NET_1 | TCP client hello world | 13.8 | 15 | 1/1 |
| OK | K64F | ARM | NET_13 | TCP client echo loop | 12.67 | 15 | 1/1 |
| OK | K64F | ARM | NET_2 | NIST Internet Time Service | 14.93 | 15 | 1/1 |
| OK | K64F | ARM | NET_3 | TCP echo server | 11.98 | 10 | 1/1 |
| OK | K64F | ARM | NET_4 | TCP echo client | 12.08 | 10 | 1/1 |
| OK | K64F | ARM | NET_5 | UDP echo server | 11.73 | 10 | 1/1 |
| OK | K64F | ARM | NET_6 | UDP echo client | 12.14 | 10 | 1/1 |
| OK | K64F | ARM | NET_7 | HTTP client hello world | 23.3 | 15 | 1/1 |
| OK | K64F | ARM | NET_8 | NTP client | 13.01 | 10 | 1/1 |
| OK | LPC1768 | ARM | NET_1 | TCP client hello world | 11.72 | 15 | 1/1 |
| OK | LPC1768 | ARM | NET_13 | TCP client echo loop | 10.41 | 15 | 1/1 |
| OK | LPC1768 | ARM | NET_2 | NIST Internet Time Service | 10.24 | 15 | 1/1 |
| OK | LPC1768 | ARM | NET_3 | TCP echo server | 9.8 | 10 | 1/1 |
| OK | LPC1768 | ARM | NET_4 | TCP echo client | 9.84 | 10 | 1/1 |
| OK | LPC1768 | ARM | NET_5 | UDP echo server | 9.58 | 10 | 1/1 |
| OK | LPC1768 | ARM | NET_6 | UDP echo client | 9.86 | 10 | 1/1 |
| OK | LPC1768 | ARM | NET_7 | HTTP client hello world | 15.46 | 15 | 1/1 |
| OK | LPC1768 | ARM | NET_8 | NTP client | 10.13 | 10 | 1/1 |
+--------+---------+-----------+---------+----------------------------+--------------------+---------------+-------+
Result: 18 OK

Completed in 234.56 sec
```