BitShares/bitshares — pull request #1085 by drltc (2014-12-03 17:14:26+00:00)
Title: tscript:  Simple but flexible JSON-based scripting

I've been steadily working on improving our testing framework. This pull request implements a simple but flexible JSON-based scripting environment which I call `tscript`. For an example of a tscript, see https://github.com/drltc/bitshares_toolkit/blob/f802de5e06bbe16a1b8866628701829d6ce91bec/tests/drltc_tests/tscript/simple.tscript which creates two clients, has both of them wait for 3 blocks, and ensures that both clients report `blockchain_get_block_count` is equal to three.

This framework most resembles the regression tests, however the regression tests are very dependent on client-specific formatting code for both input and output. Thus we have commits like 3475bcc997b13c3d8ee19847054551dbdd93a947, 499cf6c9705db5063ac2dd5749a877917ad64e69, many others which just change spacing, column headers, etc. Besides wasting developer time and filling our commit history with garbage commits, when creating such a commit it would be very easy for a maintainer copy-pasting dozens or hundreds of lines of new logs to miss a semantically important change in the output, like a different number somewhere, in all the "noise" of changed whitespace.

Since we already have framework in place to have the client execute input commands formatted as JSON, and generate the output of those commands as JSON, I simply leveraged that capability to implement testing scripts as JSON. Thus tscripts don't have to worry about doing complicated parsing with ``'s everywhere, heavy dependence on formatting code, or filtering the output to make it easier to parse.

Also, I added code to log each RPC's input and output as JSON (which may be `#defined` out for performance and security reasons in production builds). Currently this is only active in the wallet_tests, where a file called `glog.out` is now produced in the output directory. My plan is to eventually use the logged output to automatically create `tscript` tests which do the same sequence of RPC's as the regression / acceptance tests and expect the same output.

In other words, rather than starting multiple client instances and talking to them via the RPC interface (like the acceptance tests currently do), or running multiple C++ code by directly interfacing to multiple client objects in-process (like the regression tests and dev tests currently do), our testing frameworks should talk to the core by producing tscripts.

I'll probably eventually copy-paste most of the text of this pull request onto a wiki page or something, once the tscript framework is a little more mature. The basic functionality works, but it needs to be thoroughly exercised by writing several more complicated tests. Which may of course lead to improvements to the framework if functionality needed for a test is missing or awkward.