Package: mbed-host-tests
Development movedThe development of Htrun has been moved into thembed-os-toolspackage. You can continue to use this module for legacy reasons, however all further development should be continued in the new package.Htrun (mbed-host-tests)htrunhas extensive command line. In most caseshtrun(or its command line avatarmbedhtrun) will be run in background:driving test binary flashing,device reset andtest execution.Default binary flashing method is one supported bymbed-enableddevices: binary file is copied on mbed-enabled DUT (Device Under Test) mounted drive (MSD). This procedure will automatically flash device with given binary file content.Default DUT reset method is one supported bymbed-enableddevices: serial port (CDC) "sendBreak" command resets target MCU on mbed-enabled platform.Test execution phase will consist of:Opening connection between host computer and DUT,DUT will send to host preamble with test runner information such as:test environment version,test timeout,preferred host test script (Python script which is used to supervise/instrument test execution),Host will spawn host test script and test execution will be instrumentedExchange data (in most cases text) between host and DUT,Command line overviewThis chapter will present few examples of how you can usembedhtruncommand line to execute tests. In most cases test automation tools such asGreenteawill executembedhtrunimplicitly. There are cases when we want to executembedhtrunindependently. Mostly in situation when we want to:debug our code and have binary + host test instrumentation on,prototype orjust want to replace test runner in another OS with one compatible with mbed-enabled devices.Allmbedhtrunneeds is name of the binary you want to flash and method of flashing!Useful command line end-to-end examplesFlashing phase operationsFlash binary file/path/to/file/binary.binusing mount pointD:. Use serial portCOM4to communicate with DUT:$ mbedhtrun -f /path/to/file/binary.bin -d D: -p COM4Flash (use shell commandcopy) binary file/path/to/file/binary.binusing mount pointD:. Use serial portCOM4to communicate with DUT:$ mbedhtrun -f /path/to/file/binary.bin -d D: -p COM4 -c copySkip flashing phase (e.g. you've already flashed this device with/path/to/file/binary.binbinary). Use serial portCOM4to communicate with DUT:$ mbedhtrun -f /path/to/file/binary.bin -d D: -p COM4 --skip-flashingDUT-host communication and reset phaseFlash binary file/path/to/file/binary.binusing mount pointD:. Use serial portCOM4with baudrate115200to communicate with DUT:$ mbedhtrun -f /path/to/file/binary.bin -d D: -p COM4:115200As above but we will skip reset phase (non so common but in some cases can be used to suppress reset phase for some reasons):$ mbedhtrun -f /path/to/file/binary.bin -d D: -p COM4:115200 --skip-resetFlash binary file/path/to/file/binary.binusing mount pointD:. Use serial portCOM4with default baudrate to communicate with DUT. Do not send__synckey-value protocol synchronization packet to DUT before preamble read:$ mbedhtrun -f /path/to/file/binary.bin -d D: -p COM4 --sync=0Note: Sync packet management allows you to manipulate the wayhtrunsends__syncpacket(s) to DUT. With current settings we can force onhtrunto send__syncpackets in this manner:--sync=0- No sync packets will be sent to DUT.--sync=-1-__syncpackets will be sent unless we will reach timeout or proper response is sent from DUT.--sync=N- Where N is integer > 0. Send up to N__syncpackets to target platform. Response is sent unless we get response from target platform or timeout occurs.Global Resource Manager connectionFlash local file/path/to/file/binary.binto remote device resource (platformK64F) provided byremote_clientGRM service available on IP address10.2.203.31and port:8000. Force serial port connection to remote device9600with baudrate:$ mbedhtrun -p :9600 -f /path/to/file/binary.bin -m K64F --grm remote_client:10.2.203.31:8000Command line switch--grmhas format:<module_name>:<IP_address>:<port_number>.<module_name>- name of Python module to load as remote resource manager.<IP_address>and<port_number>- IP address and port of remote resource manager.Note: Switch -m <platform_name> is required to tell Global Resource Management which platform to request.Note: Command line switch--grmimplicitly forces--skip-flashingand--skip-resetbecause both flags are used for locally available DUTs.Fast Model connectionThis option is designed for htrun to use Arm Fast Models.The "--fm" option only available whenmbed-fastmodel-agentmodule is installed :Load local file/path/to/file/binary.elfto onto fastmodel FVP_MPS2_m3 simulators:$ mbedhtrun -f /path/to/file/binary.elf -m FVP_MPS2_M3 --fm DEFAULTCommand line switch format--fm <config_name>.<config_name>- ether pre-defined CONFIG_NAME from mbedfm or a local config file for the Fast Models.Note: Switch -m <platform_name> is required to tell this fastmodel connection which Fastmodel to request.Note: Command line switch--fmimplicitly forces--skip-flashingand--skip-resetbecause both flags are used for locally available DUTs.MiscellaneousList available host tests names, class names and origin:$ mbedhtrun --listList available host tests names, class names and origin. Load additional host tests from/path/to/host_testsdirectory:$ mbedhtrun --list -e /path/to/host_testsList available reset and flashing plugins:$ mbedhtrun --pluginsFlash binary file/path/to/file/binary.binusing pluginstlink. Use serial portCOM4with baudrate115200to communicate with DUT:mbedhtrun -c stlink -f /path/to/file/binary.bin -p COM4:115200Installationhtrunis redistributed with sources, as Python 2.7 compatible module calledmbed-host-testsand command line tool calledmbedhtrun.Installation from PyPI (Python Package Index)mbed-host-testsmodule is redistributed via PyPI. We recommend you use theapplication pip.Note:Python 2.7.9 onwards includepipby default, so you may havepipalready.Note:mbed-host-testsmodule is redistributed withmbed-greenteamodule as a dependency. So if you've already installed Greenteambed-host-testsshould be there!To install mbed-ls fromPyPIuse command:$ pip install mbed-host-tests --upgradeInstallation from Python sourcesTo install the mbed test suite, first clone thembed-os-toolsrepository:$ git clone https://github.com/ARMmbed/mbed-os-tools.gitChange the directory to thembed-os-tools/packages/mbed-host-testsdirectory:$ cd mbed-os-tools/packages/mbed-host-testsNow you are ready to installhtrun:$ python setup.py installChecking installationTo check whether the installation was successful try running thembedgt --helpcommand and check that it returns information (you may need to restart your terminal first):$ mbedhtrun --help
Usage: mbedgt-script.py [options]

Flash, reset and perform host supervised tests on mbed platforms

Options:
  -h, --help            show this help message and exitmbed-host-testsmbed's test suite (codenamedGreentea) supports thetest supervisorconcept. This concept is realized by this module.mbed-host-testsis a collection of host tests. Host test is script written in Python, which is executed in parallel with the test suite runner (a binary running on the target hardware / device under test) to monitor the test execution's progress or to control the test flow (interaction with the mbed device under test - DUT). The host test is also responsible for grabbing the test result, or deducing it from the test runner's behavior.Key-value protocol was developed and is used to provide communication layer between DUT (device under test) and host computer. Key-value protocol defined host computer as master and DUT as slave.Slave side APIs and key-value protocol implementation is encapsulated ingreentea-clientmodule.Master side APIs and key-value protocol is encapsulated inmbed-host-tests.mbed-host-testsresponsibilities are:Flash mbed device with given binary.Reset mbed device after flashing to start test suite execution.Use key-value protocol to handshake with device and make sure correct host test script is executed to supervise test suite execution.Run key-value protocol state machine and execute event callbacks.Monitor serial port traffic to parse valid key-value protocol events.Make decision if test test suite passed / failed / returned error.Provide command line tool interface, command:mbedhtrunafter module installation (on host).Provide few basic host test implementations which can be used out of the box for test development. For example the basic host test (calleddefaultordefault_auto) just parses events from DUT and finished host test execution whenendevent is received. Other included in this module host tests can help you to test timers or RTC.Key-value protocol overviewText based protocol, format{{KEY;VALUE}}}.Master-slave mode where host is master and DUT is slave.Design draftSimple key-value protocol is introduced. It is used to communicate between DUT and host. Protocol main features:Protocol introduced is master-slave protocol, where master is host and slave is device under test.Transport layer consist of simple{{ KEY ; VALUE }} \ntext messages sent by slave (DUT). Both key and value are strings with allowed character set limitations (to simplify parsing and protocol parser itself). Message ends with required by DUT K-V parser\ncharacter.DUT always (except for handshake phase) initializes communication by sending key-value message to host.To avoid miscommunication between master and slave simple handshake protocol is introduces:Master (host) sends sync packet:{{__sync;UUID-STRING}}}with message value containing random UUID string.DUT waits for{{__sync;...}}message in input stream and replies with the same packer{{__sync;...}}.After correct sync packet is received by master, messages{{__timeout;%d}}and{{__host_test_name}}are expected.Host parses DUTs tx stream and generates events sent to host test.Each event is a tuple of(key, value, timestamp), where key and value are extracted from message andHost tests are now driven by simple async feature. Event state machine on master side is used to process events from DUT. Each host test is capable of registering callbacks, functions which will be executed when event occur. Event name is identical with KEY in key-value pair send as event from/to DUT.DUT slave side uses simple parser to parse key-value pairs from stream. All non key-value data will be ignored. Blocking wait for an event API is provided: This implies usage of master-slave exchange between DUT and host where DUT uses non-blocking send event API to send to host (master) event and can wait for response. Master implements corresponding response after receiving event and processing data.Message parsing transforms key-value string message to Python event in this order:{{key;value}}string captured on DUT output.key-value data becomes a recognizable message with key (string) and value (string) payload.Event is formed in host test, a tuple ofkey(string),value(string),timestampwheretimestampis time of message reception in Pythontime.time()format (float, time in seconds since the epoch as a floating point number.).Each host test registers callbacks for available events.Few keys' names in key-value messaging protocol are promoted to be considered "system events". Their names are used by event loop mechanism to communicate between DUT, host and various internal components. Please do not use restricted even names for your own private events. What's more:User can't register callbacks to "system events" with few exceptions.Reserved event/message keys have leading__in name:__sync- sync message, used by master and DUT to handshake.__notify_sync_failed- sent by host when sync response not received from DUT.__timeout- timeout in sec, sent by DUT after{{sync;UUID}}is received.__version-greentea-clientversion send from DUT to host.__host_test_name- host test name, sent by DUT after{{sync;UUID}}is received.__notify_prn- sent by host test to print log message.__notify_conn_lost- sent by host test's connection process to notify serial port connection lost.__notify_complete- sent by DUT, async notificaion about test case result (true, false, none).__coverage_start- sent by DUT, coverage data.__testcase_start- sent by DUT, test case start data.__testcase_finish- sent by DUT, test case result.__exit- sent by DUT, test suite execution finished.__exit_event_queue- sent by host test, indicating no more events expected.Non-Reserved event/message keys have leading__in name:__rxd_line- Event triggered when\nwas found on DUT RXD channel. It can be overridden (self.register_callback('__rxd_line', <callback_function>)) and used by user. Event is sent by host test to notify a new line of text was received on RXD channel.__rxd_lineevent payload (value) in a line of text received from DUT over RXD.Each host test (master side) has four functions used by async framework:setup()used to initialize host test and register callbacks.result()used to return test case result whennotify_complete()is not called.teardown()used to finalize and resource freeing. It is guaranteed thatteardown()will be always called after timeout or async test completion().notify_complete(result : bool)used by host test to notify test case result. This result will be read after test suiteTIMEOUTs or after DUT send__exitmessage (test suite execution finished event).self.send_kv(key : string, value : string)- send key-value message to DUT.self.log(text : string)- send event__notify_prnwith text as payload (value). Your message will be printed in log.Result returned from host test is a test suite result. Test cases results are reported by DUT, usually using modifiedutestframework.Greentea client APIDUT test API was first introduced inmbedmicro/mbedprojecthere. After refactoring this functionality was copied and improved ingreentea-clientmodule.Slave side key-value protocol API, seeherefor details.// Send key-value pairs from slave to mastervoidgreentea_send_kv(constchar*,constchar*);voidgreentea_send_kv(constchar*,constint);voidgreentea_send_kv(constchar*,constint,constint);voidgreentea_send_kv(constchar*,constchar*,constint);voidgreentea_send_kv(constchar*,constchar*,constint,constint);// Blocking, receive key-value message from masterintgreentea_parse_kv(char*,char*,constint,constint);Functions are used to send key-string or key-integer value messages to master. This functions should replace typicalprintf()calls with payload/control data to host.Blockingwait for key-value pair message in input stream:intgreentea_parse_kv(char*out_key,char*out_value,constintout_key_len,constintout_value_len);This function should replacescanf()used to check for incoming messages from master.
Function parses input and if key-value message is found load toout_key,out_valuekey-value pair. Useout_key_sizeandout_value_sizeto define out buffers max size (including trailing zero).Key-value transport protocol sequenceKey-value protocol has few parts:Handshake- synchronize master and slave.Preamble exchange- DUT informs host about test parameters such as client version, test suite timeout, requested host test name etc. After this part is finished master will create requested host test and attach callbacks to user events.Event exchange- key-value event exchange between slave and master. In this exchange in general slave (DUT) will initialize communication. This part may end with ending pair of eventsendand__exitwhereendevent carries test suite result returned by DUT and__exitevent marks test suite ended and exited. After__exitevent is received there will be no more communication between DUT and host test.HandshakeHanshake between DUT and host is a sequence of__syncevents send between host (master) and DUT (slave). This is currently only situation when master initiates communication first. Handshake should provide synchronization point where master and slave are starting the same session.After reset:DUT calls functionGREENTEA_SETUP(timeout, "host test name");whichcalls immediatelygreentea_parse_kv(blocking parse of input serial port for event{{__sync;UUID}}).When__syncpacket is parsed in the stream DUT sends back (echoes)__syncevent with the sameUUIDas payload. UUID is a random value e.g.5f8dbbd2-199a-449c-b286-343a57da7a37.DUT (slave)        host (master)
                             -----               -----
                               |                   |
                DUT reset ---> |                   |
                               |                   |
greentea_parse_kv(key,value)   |                   |
-------[ blocking ]----------->|                   |
                               |                   |
                               .                   .
                               .                   .
                               |                   |  self.send_kv("__sync", UUID)
                               |  {{__sync;UUID}}  |<-----------------------------
                               |<------------------|
                               |                   |
                               |                   |
greentea_parse_kv              |  {{__sync;UUID}}  |
echoes __sync event with       |------------------>|
the same UUID to master        |                   |
                               |                   |Example of handshake fromhtrunlog:DUT code:// GREENTEA_SETUP pseudo-codevoidGREENTEA_SETUP(constinttimeout,constchar*host_test_name){// Wait for SYNC and echo it backchar_key[8]={0};char_value[48]={0};while(1){greentea_parse_kv(_key,_value,sizeof(_key),sizeof(_value));if(strcmp(_key,GREENTEA_TEST_ENV_SYNC)==0){// Found correct __sunc messagegreentea_send_kv(_key,_value);break;}}// Send PREAMBLE: client version, test suite timeout and requested host testgreentea_send_kv(GREENTEA_TEST_ENV_HOST_TEST_VERSION,"0.1.8");greentea_send_kv(GREENTEA_TEST_ENV_TIMEOUT,timeout);greentea_send_kv(GREENTEA_TEST_ENV_HOST_TEST_NAME,host_test_name);}Corresponding log:[1458565465.35][SERI][INF] reset device using 'default' plugin...
[1458565465.60][SERI][INF] wait for it...
[1458565466.60][CONN][INF] sending preamble '2f554b1c-bbbf-4b1b-b1f0-f45493282f2c'
[1458565466.60][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1458565466.60][SERI][TXD] {{__sync;2f554b1c-bbbf-4b1b-b1f0-f45493282f2c}}
[1458565466.74][CONN][INF] found SYNC in stream: {{__sync;2f554b1c-bbbf-4b1b-b1f0-f45493282f2c}}, queued...
[1458565466.74][HTST][INF] sync KV found, uuid=2f554b1c-bbbf-4b1b-b1f0-f45493282f2c, timestamp=1458565466.743000
[1458565466.74][CONN][RXD] {{__sync;2f554b1c-bbbf-4b1b-b1f0-f45493282f2c}}Preamble exchangeThis phase comes just after handshake phase. DUT informs host about test parameters such as client version, timeout, requested host test name etc. After this part is finished master will create requested host test and attach callbacks to user events.
This phase is ended with__host_test_namebeing received by host. After__host_test_nameevent is receivedDUT (slave)              host (master)
  -----                     -----
    |                         |
    |    {{__version;%s}}     |
    |------------------------>|
    |                         |
    |    {{__timeout;%d}}     |
    |------------------------>|
    |                         |
    | {{__host_test_name;%s}} |
    |------------------------>|
    |                         |Example of handshake fromhtrunlog:DUT code:voidmain(){GREENTEA_CLIENT(5,"default_auto");// ...}Corresponding log:[1458565466.76][CONN][INF] found KV pair in stream: {{__version;0.1.8}}, queued...
[1458565466.76][CONN][RXD] {{__version;0.1.8}}
[1458565466.76][HTST][INF] DUT greentea-client version: 0.1.8
[1458565466.77][CONN][INF] found KV pair in stream: {{__timeout;5}}, queued...
[1458565466.77][HTST][INF] setting timeout to: 5 sec
[1458565466.78][CONN][RXD] {{__timeout;5}}
[1458565466.81][CONN][INF] found KV pair in stream: {{__host_test_name;default_auto}}, queued...
[1458565466.81][HTST][INF] host test setup() call...
[1458565466.81][HTST][INF] CALLBACKs updated
[1458565466.81][HTST][INF] host test detected: default_auto
[1458565466.81][CONN][RXD] {{__host_test_name;default_auto}}Event exchangeIn this phase DUT and host exchange events and host side is calling callbacks registered to each of the events sent from DUT. DUT can use functiongreentea_parse_kvto parse input stream for next incoming key-value event.
After__host_test_nameevent is received and before any event is consumed during this stage:Host state machine loads host test object by name provided in payload of__host_test_nameevent.E.g. event ```{{____host_test_name;default_auto}} will load host test named "default_auto".Host state machine loads callbacks registered by user in host test setup phase and hooks them to event machine.
Now host is ready to handle test suite test execution. From this moment each event sent from DUT will be handled by corresponding callback registered by user in host test setup. Unknown events will not be handled and warning will be printed in log.DUT (slave)      host (master)
  -----             -----
    |                 |
    |                 |        Host Test
    |                 |         -----
    |                 |  create   |
    |                 |---------->|
    |                 |           |
    |                 |           |
    | {{key1;value}}  |           |
    |---------------->|           |          ht.setup()
    |       .         |           |<---[ user register callbacks ]---
    |       .         |           |
    |       .         |           |  host.callbacks.update(ht.get_callbacks())
    |       .         |           |<---[ host state machine ]------------------
    | {{key2;value}}  |           |
    |---------------->|           |
    |                 |           |
    |                 |           |
    |                 |           | ht.callbacks[key1](key, value, timestamp)
    |                 |           |<------------------------------------------
    |                 |           | ht.callbacks[key2](key, value, timestamp)
    |                 |           |<------------------------------------------
    |                 |           |
    |                 |           |
    -  - - - - - - - -  - - - -     - -
          TEST CASE FLOW CONTINUES
    -  - - - - - - - -  - - - -     - -
    |                 |           |
    |                 |           | ht.notify_complete(true)
    |                 |           | (sets test suite 'result' to true
    |                 |           |<----------------
    |                 |           |
    |                 |           |
    | {{end;success}} |           |
    |---------------->|           |
    |                 |           |
    | {{__exit;%d}}   |           |
    |---------------->|           |
    |                 |           |
    |                 |           | result = ht.result()
    |                 |           |<----------------
    |                 |           |
    |                 |           | ht.teardown()
    |                 |           |<----------------
    |                 |           |
    |                 |           |After DUT send__exitor after timeout it is guaranteed that host testteardown()function will be called. This call is blocking, please make sure your tear down function finishes.DUT - host test case workflowDUT implementationintmain(){// 1. Handshake between DUT and host and// 2. Send test case related dataGREENTEA_SETUP(15,"gimme_auto");// __timeout, __host_test_name// ...// Send to master {{gimme_something; some_stuff}}greentea_send_kv("gimme_something","some_stuff");charkey[16]={0};charvalue[32]={0};// Blocking wait for master response for {{gimme_something; some_stuff}}greentea_parse_kv(key,value,sizeof(key),sizeof(value));// ...fprintf(stderr,"Received from master %s, %s",key,value);// ...GREENTEA_TESTSUITE_RESULT(true);// __exit}Example of corresponding host testclassGimmeAuto(BaseHostTest):""" Simple, basic host test's test runner waiting for serial portoutput from MUT, no supervision over test running in MUT is executed."""__result=Nonename="gimme_auto"def_callback_gimme_something(self,key,value,timestamp):# You've received {{gimme_something;*}}# We will send DUT some data back...# And now decide about test case resultifvalue=='some_stuff':# Message payload/value was 'some_stuff'# We can for example return true from testself.send_kv("print_this","This is what I wanted%s"%value)self.notify_complete(True)else:self.send_kv("print_this","This not what I wanted :(")self.notify_complete(False)defsetup(self):# Register callback for message 'gimme_something' from DUTself.register_callback("gimme_something",self._callback_gimme_something)# Initialize your host test here# ...defresult(self):# Define your test result here# Or use self.notify_complete(bool) to pass result anytime!returnself.__resultdefteardown(self):# Release resources here after test is completedpassLog:[1454926794.22][HTST][INF] copy image onto target...
        1 file(s) copied.
[1454926801.48][HTST][INF] starting host test process...
[1454926802.01][CONN][INF] starting connection process...
[1454926802.01][CONN][INF] initializing serial port listener...
[1454926802.01][SERI][INF] serial(port=COM188, baudrate=9600)
[1454926802.02][SERI][INF] reset device using 'default' plugin...
[1454926802.27][SERI][INF] wait for it...
[1454926803.27][CONN][INF] sending preamble '9caa42a0-28a0-4b80-ba1d-befb4e43a4c1'...
[1454926803.27][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1454926803.27][SERI][TXD] {{__sync;9caa42a0-28a0-4b80-ba1d-befb4e43a4c1}}
[1454926803.40][CONN][RXD] {{__sync;9caa42a0-28a0-4b80-ba1d-befb4e43a4c1}}
[1454926803.40][CONN][INF] found SYNC in stream: {{__sync;9caa42a0-28a0-4b80-ba1d-befb4e43a4c1}}, queued...
[1454926803.40][HTST][INF] sync KV found, uuid=9caa42a0-28a0-4b80-ba1d-befb4e43a4c1, timestamp=1454926803.405000
[1454926803.42][CONN][RXD] {{__timeout;15}}
[1454926803.42][CONN][INF] found KV pair in stream: {{__timeout;15}}, queued...
[1454926803.42][HTST][INF] setting timeout to: 15 sec
[1454926803.45][CONN][RXD] {{__host_test_name;gimme_auto}}
[1454926803.45][CONN][INF] found KV pair in stream: {{__host_test_name;gimme_auto}}, queued...
[1454926803.45][HTST][INF] host test setup() call...
[1454926803.45][HTST][INF] CALLBACKs updated
[1454926803.45][HTST][INF] host test detected: gimme_auto
[1454926803.48][CONN][RXD] {{gimme_something;some_stuff}}
[1454926803.48][CONN][INF] found KV pair in stream: {{gimme_something;some_stuff}}, queued...
[1454926803.48][SERI][TXD] {{print_this;This is what I wanted some_stuff}}
[1454926803.48][HTST][INF] __notify_complete(True)
[1454926803.62][CONN][RXD] Received from master print_this, This is what I wanted some_stuf
[1454926803.62][CONN][RXD] {{end;success}}
[1454926803.62][CONN][INF] found KV pair in stream: {{end;success}}, queued...
[1454926803.62][HTST][ERR] orphan event in main phase: {{end;success}}, timestamp=1454926803.625000
[1454926803.63][CONN][RXD] {{__exit;0}}
[1454926803.63][CONN][INF] found KV pair in stream: {{__exit;0}}, queued...
[1454926803.63][HTST][INF] __exit(0)
[1454926803.63][HTST][INF] test suite run finished after 0.21 sec...
[1454926803.63][HTST][INF] exited with code: None
[1454926803.63][HTST][INF] 0 events in queue
[1454926803.63][HTST][INF] stopped consuming events
[1454926803.63][HTST][INF] host test result() skipped, received: True
[1454926803.63][HTST][INF] calling blocking teardown()
[1454926803.63][HTST][INF] teardown() finished
[1454926803.63][HTST][INF] {{result;success}}
mbedgt: mbed-host-test-runner: stopped
mbedgt: mbed-host-test-runner: returned 'OK'
mbedgt: test on hardware with target id: 02400226d94b0e770000000000000000000000002492f3cf
mbedgt: test suite 'mbed-drivers-test-gimme' ......................................................... OK in 10.02 sec
mbedgt: shuffle seed: 0.3631708941
mbedgt: test suite report:
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| target        | platform_name | test suite              | result | elapsed_time (sec) | copy_method |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| frdm-k64f-gcc | K64F          | mbed-drivers-test-gimme | OK     | 10.02              | shell       |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
mbedgt: test suite results: 1 OKHost test examplesReturn result after __exitclassGimmeAuto(BaseHostTest):""" Simple, basic host test's test runner waiting for serial portoutput from MUT, no supervision over test running in MUT is executed."""__result=Nonename="gimme_auto"def_callback_gimme_something(self,key,value,timestamp):# You've received {{gimme_something;*}}# We will send DUT some data back...# And now decide about test case resultifvalue=='some_stuff':# Message payload/value was 'some_stuff'# We can for example return true from testself.send_kv("print_this","This is what I wanted%s"%value)self.__result=Trueelse:self.send_kv("print_this","This not what I wanted :(")self.__result=Falsedefsetup(self):# Register callback for message 'gimme_something' from DUTself.register_callback("gimme_something",self._callback_gimme_something)# Initialize your host test here# ...defresult(self):# Define your test result here# Or use self.notify_complete(bool) to pass result anytime!returnself.__resultdefteardown(self):# Release resources here after test is completedpassCorresponding log:[1454926627.11][HTST][INF] copy image onto target...
        1 file(s) copied.
[1454926634.38][HTST][INF] starting host test process...
[1454926634.93][CONN][INF] starting connection process...
[1454926634.93][CONN][INF] initializing serial port listener...
[1454926634.93][SERI][INF] serial(port=COM188, baudrate=9600)
[1454926634.94][SERI][INF] reset device using 'default' plugin...
[1454926635.19][SERI][INF] wait for it...
[1454926636.19][CONN][INF] sending preamble '9a743ff3-45e6-44cf-9e2a-9a83e6205184'...
[1454926636.19][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1454926636.19][SERI][TXD] {{__sync;9a743ff3-45e6-44cf-9e2a-9a83e6205184}}
[1454926636.33][CONN][RXD] {{__sync;9a743ff3-45e6-44cf-9e2a-9a83e6205184}}
[1454926636.33][CONN][INF] found SYNC in stream: {{__sync;9a743ff3-45e6-44cf-9e2a-9a83e6205184}}, queued...
[1454926636.33][HTST][INF] sync KV found, uuid=9a743ff3-45e6-44cf-9e2a-9a83e6205184, timestamp=1454926636.331000
[1454926636.34][CONN][RXD] {{__timeout;15}}
[1454926636.34][CONN][INF] found KV pair in stream: {{__timeout;15}}, queued...
[1454926636.34][HTST][INF] setting timeout to: 15 sec
[1454926636.38][CONN][RXD] {{__host_test_name;gimme_auto}}
[1454926636.38][CONN][INF] found KV pair in stream: {{__host_test_name;gimme_auto}}, queued...
[1454926636.38][HTST][INF] host test setup() call...
[1454926636.38][HTST][INF] CALLBACKs updated
[1454926636.38][HTST][INF] host test detected: gimme_auto
[1454926636.41][CONN][RXD] {{gimme_something;some_stuff}}
[1454926636.41][CONN][INF] found KV pair in stream: {{gimme_something;some_stuff}}, queued...
[1454926636.41][SERI][TXD] {{print_this;This is what I wanted some_stuff}}
[1454926636.54][CONN][RXD] Received from master print_this, This is what I wanted some_stuf
[1454926636.54][CONN][RXD] {{end;success}}
[1454926636.54][CONN][INF] found KV pair in stream: {{end;success}}, queued...
[1454926636.55][HTST][ERR] orphan event in main phase: {{end;success}}, timestamp=1454926636.541000
[1454926636.56][CONN][RXD] {{__exit;0}}
[1454926636.56][CONN][INF] found KV pair in stream: {{__exit;0}}, queued...
[1454926636.56][HTST][INF] __exit(0)
[1454926636.56][HTST][INF] test suite run finished after 0.22 sec...
[1454926636.56][HTST][INF] exited with code: None
[1454926636.56][HTST][INF] 0 events in queue
[1454926636.56][HTST][INF] stopped consuming events
[1454926636.56][HTST][INF] host test result(): True
[1454926636.56][HTST][INF] calling blocking teardown()
[1454926636.56][HTST][INF] teardown() finished
[1454926636.56][HTST][INF] {{result;success}}
mbedgt: mbed-host-test-runner: stopped
mbedgt: mbed-host-test-runner: returned 'OK'
mbedgt: test on hardware with target id: 02400226d94b0e770000000000000000000000002492f3cf
mbedgt: test suite 'mbed-drivers-test-gimme' ......................................................... OK in 10.04 sec
mbedgt: shuffle seed: 0.3866075474
mbedgt: test suite report:
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| target        | platform_name | test suite              | result | elapsed_time (sec) | copy_method |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| frdm-k64f-gcc | K64F          | mbed-drivers-test-gimme | OK     | 10.04              | shell       |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
mbedgt: test suite results: 1 OKWriting DUT test suite (slave side)DUT test suite with single test caseWe can use few methods to structure out test suite and test cases. Simpliest would be to usegreentea-clientAPI and wrap one test case inside out test suite. This way of creating test suite is useful when you want to:write only one test case inside test suite,make example application (example as a test) orwhen your test suite is calling blocking forever function. For example all types of UDP/TCP servers which run in forever loop are in this category. In this case we do not expect from DUT__exitevent at all and host test should be designed in such a way that it always return result.DUT always finishes executionIn this example DUT code usesgreentea-clientto sync (GREENTEA_SETUP) and pass result (GREENTEA_TESTSUITE_RESULT) toGreentea. This is very simple example of how you can write tests. Note that in this example test suite only implements one test case. Actually test suite is test case at the same time. Result passed toGREENTEA_TESTSUITE_RESULTwill be at the same time test case result.DUT implementation:#include"greentea-client/test_env.h"#include"unity/unity.h"// Optional: unity ASSERTsintapp_start(int,char*[]){boolresult=true;GREENTEA_SETUP(15,"default_auto");// test case execution and assertionsGREENTEA_TESTSUITE_RESULT(result);return0;}DUT test suite never finishes executionTest suite is implemented so that it will never exit / finish its execution. For examplemain()orapp_start()functions are implemented using infinite (endless) loop. This property have for example UDP/TCP servers (listening forever), all sorts of echo servers etc.In this example DUT code usesgreentea-clientto sync (GREENTEA_SETUP) withGreentea. We are not callingGREENTEA_TESTSUITE_RESULT(result)at any time. In this example host test is responsible for providing test suite result usingself.notify_complete()API orself.result()function.You need to write and specify by name your custom host test:DUT side uses second argument ofGREENTEA_SETUP(timeout, host_test_name)function:GREENTEA_SETUP(15,"wait_us_auto");You need to place your custom host test in<module>/test/host_testsdirectory.Do not forget to name host test accordingly. See below example host testnameclass member.DUT implementation usingmy_host_testcustom host test:#include"greentea-client/test_env.h"#include"unity/unity.h"voidrecv(){// receive from client}intapp_start(int,char*[]){Etherneteth(TCP_SERVER,PORT,recv);GREENTEA_SETUP(15,"my_host_test");eth.listen();// Blocking foreverreturn0;}Example host test template:frommbed_host_testsimportBaseHostTestclassYourCustomHostTest(BaseHostTest):name="my_host_test"# Host test names used by GREENTEA_CLIENT(..., host_test_name)__result=False# Result in case of timeout!def_callback_for_event(self,key,value,timestamp):## Host test API:## self.notify_complete(result : bool)## """! Notify main even loop that host test finished processing#      @param result True for success, False failure. If None - no action in main even loop# """## self.send_kv(key : string, value : string)## """! Send Key-Value data to DUT#      @param key Event key#      @param value Event payload# """## self.log(text : string)## """! Send log message to main event loop#      @param text log message# """passdefsetup(self):# TODO:# * Initialize your resources# * Register callbacks:## Host test API:## self.register_callback(event_name, callable, force=False)## """! Register callback for a specific event (key: event name)#     @param key String with name of the event#     @param callback Callable which will be registered for event "key"#     @param force God mode, if set to True you can add callback on any system event# """passdefteardown(self):# Destroy all resources used by host test.# For example open sockets, open files, auxiliary threads and processes.passdefresult(self):# Returns host test result (True, False or None)# This function will be called when test suite ends (also timeout).# Use when you want to pass result after host state machine stops.return__resultDUT test suite withutestharnessutestharness allows you to define multiple test cases inside your test suite. This feature is supported byGreenteatest tools.DUT implementation:#include"greentea-client/test_env.h"#include"unity/unity.h"#include"utest/utest.h"status_tgreentea_failure_handler(constCase*constsource,constfailure_treason){// Continue with next test case if it failsgreentea_case_failure_abort_handler(source,reason);returnSTATUS_CONTINUE;}voidtest_uninitialised_array(){// TEst case code...}voidtest_repeated_init(){// TEst case code...}voidtest_data_types(){// TEst case code...}constCasecases[]={Case("Test uninitialised array",test_uninitialised_array,greentea_failure_handler),Case("Test repeated array initialisation",test_repeated_init,greentea_failure_handler),Case("Test basic data type arrays",test_data_types,greentea_failure_handler)// ...};status_tgreentea_setup(constsize_tnumber_of_cases){GREENTEA_SETUP(5,"default_auto");returngreentea_test_setup_handler(number_of_cases);}intapp_start(int,char*[]){// Run the test casesHarness::run(specification);}Writing host tests (master side)When writing a new host test for your module please bear in mind that:You own the host test and you should write it the way so it can coexist with the same host tests ran by other processes such as Continuous Integration systems or other host users.Note: If you work in isolation and your test environment if fully controlled by you (for example you queue all tasks calling host tests, or use global host unique socket port numbers) this rule doesn’t apply to you.When writing host test using OS resources such as sockets, files, serial ports, peripheral devices like for example multi-meters / scopes. remember that those resources are indivisible!For example if you hardcode in your host test UDP port 32123 and use it for UDP server implementation  of your host test bear in mind that this port may be already used. It is your responsibility to react for this event and implement means to overcome it (if possible).CallbacksYou can register callbacks insetup()phase or decorate callback functions using@event_callbackdecorator.Callback registration in setup() methodfrommbed_host_testsimportBaseHostTestclassDetectRuntimeError(BaseHostTest):__result=Falsedefcallback_some_event(self,key,value,timeout):# Do something with 'some_event'passdefsetup(self):# Reagister call back for 'some_event' eventself.register_callback('some_event',self.callback_some_event)defresult(self):# Do some return calculationsreturnself.__resultBelow the same callback registered using decorator:Callback decorator definitionfrommbed_host_tests.host_testsimportBaseHostTest,event_callbackclassDetectRuntimeError(BaseHostTest):__result=False@event_callback('some_event')defcallback_some_event(self,key,value,timeout):# Do something with 'some_event'passdefsetup(self):# Do some extra setup if required# You can also register here callbacks using self.register_callback(...) methodpassdefresult(self):# Do some return calculationsreturnself.__resultParsing text received from DUT (line by line)Example of host test expectingRuntime error ... CallbackNode ...string in DUT output.
We will use allowed to override__rxd_lineevent to hook to DUT RXD channel lines of text.Before Greentea v0.2.0fromsysimportstdoutfrommbed_host_testsimportBaseHostTestclassDetectRuntimeError(BaseHostTest):name='detect_runtime_error'deftest(self,selftest):result=selftest.RESULT_FAILUREtry:whileTrue:line=selftest.mbed.serial_readline()iflineisNone:returnselftest.RESULT_IO_SERIALstdout.write(line)stdout.flush()line=line.strip()ifline.startswith("Runtime error")andline.find("CallbackNode")!=-1:result=selftest.RESULT_SUCCESSbreakexceptKeyboardInterrupt,_:selftest.notify("\r\n[CTRL+C] exit")result=selftest.RESULT_ERRORreturnresultUsing __rdx_line eventfrommbed_host_testsimportBaseHostTestclassDetectRuntimeError(BaseHostTest):"""! We _expect_ to detect 'Runtime error' """__result=Falsedefcallback__rxd_line(self,key,value,timeout):## Parse line of text received over e.g. serial from DUT#line=value.strip()ifline.startswith("Runtime error")and"CallbackNode"inline:# We've found exepcted "Runtime error" string in DUTs output streamself.notify_complete(True)defsetup(self):# Force, we force callback registration even it is a restricted one (starts with '__')self.register_callback('__rxd_line',self.callback__rxd_line,force=True)defresult(self):# We will return here (False) when we reach timeout of the testreturnself.__resultdefteardown(self):passhtrunnew log format:[timestamp][source][level]- new log format, where:timestamp- returned by Python'stime.time().source- log source.CONN- connection process (pooling for connection source e.g. serial port),SERI- serial port wrapper with standard read, write, flush interface,HTST- host test object, HostTestBase derived object,PLGN- host test plugins, typeBasePluginof the plugin,COPY- host test plugins, typeCopyMethodof the plugin,REST- host test plugins, typeResetMethodof the plugin,level- logging level:INF(info),WRN(warning),ERR(error).TXD(host's TX channel, to DUT).RXD(host's RX channel, from DUT).Log example[1455218713.87][CONN][RXD] {{__sync;a7ace3a2-4025-4950-b9fc-a3671103387a}}:Logged fromCONN(connection process).RXDchannel emitted{{__sync;a7ace3a2-4025-4950-b9fc-a3671103387a}}.Time stamp:2016-02-11 19:53:27, see below:PluginsIn order to work with platforms for which the hardware is still under development, and hence may not have an mbed interface chip, some "hook" files are required.  Operation with these platforms is a matter for the platform development teams involved and is not, in general, supported by ARM.SARA NBIOT EVKThe SARA NBIOT EVK board must be connected to a Windows PC using a Segger JLink box, which is used for downloading code and resetting the board. The USB port on the EVK must also be connected to the same PC.  To make use of these hooks you will also require access to some proprietary tools that can be requested from u-blox.Testing mbed-os examplesmbed-os examples are essentially sample apps written as inspirational code for developers to understand the mbed-os APIs and coding paradigms. Before every mbed-os release all examples are tested across all supported configs and platforms. There is already a large set examples available and as they grow it is important to automate them. Hence automating examples make sense. Although it is important not to pollute them with test like instrumentation. As that will defeat the purpose of examples being simple and specific.Hence the strategy for testing examples is based on observation instead of interaction. An example's serial logging is captured and converted into a templated log. All successive executions of this example should match this log.Templated log simply means a log with text that does not change or regular expressions replacing original text. Below is an example of the templated log:>	Using Ethernet LWIP

							      >	Client IP Address is 10.2.203.139

							      >	Connecting with developer.mbed.org

Starting the TLS handshake...								Starting the TLS handshake...

							      >	TLS connection to developer.mbed.org established

Server certificate:								Server certificate:

							      >
								    cert. version     : 3
							      >
								    serial number     : 11:21:B8:47:9B:21:6C:B1:C6:AF:BC:5D:0
							      >
								    issuer name       : C=BE, O=GlobalSign nv-sa, CN=GlobalSi
							      >
								    subject name      : C=GB, ST=Cambridgeshire, L=Cambridge,
							      >
								    issued  on        : 2016-03-03 12:26:08
							      >
								    expires on        : 2017-04-05 10:31:02
							      >
								    signed using      : RSA with SHA-256
							      >
								    RSA key size      : 2048 bits
							      >
								    basic constraints : CA=false
							      >
								    subject alt name  : *.mbed.com, mbed.org, *.mbed.org, mbe
							      >
								    key usage         : Digital Signature, Key Encipherment
							      >
								    ext key usage     : TLS Web Server Authentication, TLS We

Certificate verification passed								Certificate verification passed




							      >	HTTPS: Received 439 chars from server

							      >	HTTPS: Received 200 OK status ... [OK]

HTTPS: Received 'Hello world!' status ... [OK]								HTTPS: Received 'Hello world!' status ... [OK]

HTTPS: Received message:								HTTPS: Received message:




							      >	HTTP/1.1 200 OK

							      >	Server: nginx/1.7.10

							      >	Date: Thu, 01 Dec 2016 13:56:32 GMT

							      >	Content-Type: text/plain

							      >	Content-Length: 14

							      >	Connection: keep-alive

							      >	Last-Modified: Fri, 27 Jul 2012 13:30:34 GMT

							      >	Accept-Ranges: bytes

							      >	Cache-Control: max-age=36000

							      >	Expires: Thu, 01 Dec 2016 23:56:32 GMT

							      >	X-Upstream-L3: 172.17.0.3:80

							      >	X-Upstream-L2: developer-sjc-indigo-2-nginx

							      >	Strict-Transport-Security: max-age=31536000; includeSubdomain




Hello world!								Hello world!Please observe above that all the lines that have data that changes from execution to execution (on right) have been removed. It makes it possible htrun to compare these logs. htrun matches lines from the compare log (on left) one by one. It keeps on looking for a line until it matches. Once matched it moves on to match the next line. If it finds all lines from the compare log in the target serial output stream. Then it halts and passes the examples.Another example with regular examples is shown below:SHA-256                  :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  SHA-256                  :       1922 Kb/s,         61 cycl

  SHA-512                  :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  SHA-512                  :        614 Kb/s,        191 cycl

  AES-CBC-128              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-CBC-128              :       1401 Kb/s,         83 cycl

  AES-CBC-192              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-CBC-192              :       1231 Kb/s,         95 cycl

  AES-CBC-256              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-CBC-256              :       1097 Kb/s,        106 cycl

  AES-GCM-128              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-GCM-128              :        429 Kb/s,        273 cycl

  AES-GCM-192              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-GCM-192              :        412 Kb/s,        285 cycl

  AES-GCM-256              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-GCM-256              :        395 Kb/s,        297 cycl

  AES-CCM-128              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-CCM-128              :        604 Kb/s,        194 cycl

  AES-CCM-192              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-CCM-192              :        539 Kb/s,        217 cycl

  AES-CCM-256              :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  AES-CCM-256              :        487 Kb/s,        241 cycl

  CTR_DRBG \(NOPR\)          :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  CTR_DRBG (NOPR)          :       1145 Kb/s,        102 cycl

  CTR_DRBG \(PR\)            :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  CTR_DRBG (PR)            :        821 Kb/s,        142 cycl

  HMAC_DRBG SHA-256 \(NOPR\) :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  HMAC_DRBG SHA-256 (NOPR) :        219 Kb/s,        537 cycl

  HMAC_DRBG SHA-256 \(PR\)   :\s*\d+ Kb/s,\s*\d+ cycles/byte							      |	  HMAC_DRBG SHA-256 (PR)   :        193 Kb/s,        612 cycl

  RSA-2048                 :\s*\d+ ms/ public							      |	  RSA-2048                 :      30 ms/ public

  RSA-2048                 :\s*\d+ ms/private							      |	  RSA-2048                 :    1054 ms/private

  RSA-4096                 :\s*\d+ ms/ public							      |	  RSA-4096                 :     101 ms/ public

  RSA-4096                 :\s*\d+ ms/private							      |	  RSA-4096                 :    5790 ms/private

  ECDHE-secp384r1          :\s*\d+ ms/handshake							      |	  ECDHE-secp384r1          :    1023 ms/handshake

  ECDHE-secp256r1          :\s*\d+ ms/handshake							      |	  ECDHE-secp256r1          :     678 ms/handshake

  ECDHE-Curve25519         :\s*\d+ ms/handshake							      |	  ECDHE-Curve25519         :     580 ms/handshake

  ECDH-secp384r1           :\s*\d+ ms/handshake							      |	  ECDH-secp384r1           :     503 ms/handshake

  ECDH-secp256r1           :\s*\d+ ms/handshake							      |	  ECDH-secp256r1           :     336 ms/handshake

  ECDH-Curve25519          :\s*\d+ ms/handshake							      |	  ECDH-Curve25519          :     300 ms/handshakeTo capture a log use following option:mbedhtrun -d D: -p COM46 -m K64F -f .\BUILD\K64F\GCC_ARM\benchmark.bin --serial-output-file compare.logOption--serial-output-filetakes file name as argument and writes the target serial output to the file. Edit the file to remove lines that will change in successive executions. Put regular expressions if needed at places like benchmark numbers in above log. With these edits you are left with a template good for comparison.Use following command to test the example and the comparison log:mbedhtrun -d D: -p COM46 -m K64F -f .\BUILD\K64F\GCC_ARM\benchmark.bin --compare-log compare.logIn case an application requires more time to process data and generate results, you can use the option--polling-timeoutto override the default timeout setting.A tested comparison log can be checked into GitHub with the examples and can be used in the CI for example verification.