Package: trapit
The Math Function Unit Testing design pattern, implemented in PythonThis module supports a new design pattern for unit testing that can be applied in any language, and is here implemented in Python. The module name is derived from 'TRansactional API Testing' (TRAPIT), and the 'unit' should be considered to be a transactional unit (this is not micro-testing).The module supplies a simple utility for unit testing Python programs based on the 'Math Function Unit Testing design pattern'. The utility provides a generic driver program for unit testing, with test data read from an input JSON file, results written to an output JSON file, and all specific test code contained in a callback function passed to the generic driver function. A separate JavaScript module is used to parse the results JSON and format them in HTML and plain text.There is a blog post on scenario selection in unit testing that may be of interest:Unit Testing, Scenarios and Categories: The SCAN MethodIn This README...↓ Background↓ Usage↓ API↓ Installation↓ Unit Testing↓ See Also↓ LicenseBackground↑ In This README...On March 23, 2018 I made the following presentation at the Oracle User Group conference in Dublin:Database API Viewed As A Mathematical Function: Insights into TestingThe first section was summarised as:Developing a universal design pattern for testing APIs using the concept of a 'pure' function as a wrapper to manage the 'impurity' inherent in database APIsAlthough the presentation focussed on database testing the design pattern is clearly quite general.The main features of the design pattern are:The unit under test is viewed from the perspective of a mathematical function having anextended signature, comprising any actual parameters and return value, together with other inputs and outputs of any kindA wrapper function is constructed based on this conceptual function, and the wrapper function isexternally pure, while internally handling impurities such as file I/OThe wrapper function performs the steps necessary to test the UUT in a single scenarioIt takes all inputs of the extended signature as a parameter, creates any test data needed from them, effects a transaction with the UUT, and returns all outputs as a return valueAny test data, and any data changes made by the UUT, are reverted before returnThe wrapper function specific to the UUT is called within a loop over scenarios by a library test driver moduleThe library test driver module reads data for all scenarios in JSON format, with both inputs to the UUT and the expected outputs, and metadata records describing the specific data structureThe module takes the actual outputs from the wrapper function and merges them in alongside the expected outputs to create an output results objectThis output results object is processed by a JavaScript module to generate the results formatted as a summary page, with a detail page for each scenario, in both HTML and text versionsAt a high level the design pattern:takes an input file containing all test scenarios with input data and expected output data for each scenariocreates a results object based on the input file, but with actual outputs merged inuses the results object to generate unit test results files formatted in HTML and/or textThe Math Function Unit Testing design pattern is centred around the idea of apurewrapper function that maps fromextendedinput parameters to anextendedreturn value, with both sides using a generic nested object structure.Here is a diagram illustrating the concept of theexternally purewrapper function:The JavaScript Trapit module supports the full process for testing JavaScript programs, and, for non-JavaScript programs, performs the formatting step by reading in the results object from a JSON file materialized by the external program.The current Python project illustrates how this works in unit testing external programs, and there are also examples using Oracle and Powershell.Advantages of the design pattern include:Writing the unit test wrapper function is the only programming required for the specific unit test, with unit test driver, assertion and formatting all centralized in library packagesOnce the unit test wrapper function is written for one scenario, no further programming is required to handle additional scenarios, facilitating good scenario coverageThe formatted results show exactly what the program does in terms of data inputs and outputsAll unit test programs can follow a single, straightforward pattern with minimal programmingThe JavaScript Trapit module can be used to process results files generated from any language as JSON files, as in the current Python projectUsage↑ In This README...↓ General Usage↓ Example 1 - colgroup↓ Example 2 - hello_worldIn this section we show how to use the package for unit testing, first in general terms, then via two examples.General Usage↑ Usage↓ Preliminary Steps↓ Unit Testing Process (General)↓ Unit Test DocumentationPreliminary Steps↑ General UsageIn order to use the design pattern for unit testing, the following preliminary steps are required:Create a JSON file containing the input test data including expected return values in the required format. The input JSON file essentially consists of two objects:meta: inp and out objects each containing group objects with arrays of field namesscenarios: scenario objects containing inp and out objects, with inp and out objects containing, for each group defined in meta, an array of input records and an array of expected output records, respectively, records being in delimited fields formatCreate a unit test script containing the wrapper function and a 1-line main section calling the Trapit library function, passing in the wrapper as a callback function. The wrapper function should call the unit under test passing the appropriate parameters and return its outputs, with the following signature:Input parameter: 3-level list with test inputs as an object with groups as properties having 2-level arrays of record/field as values: {GROUP: [[String]], ...}Return Value:    2-level list with test outputs as an object with groups as properties having an array of records as delimited fields strings as value: {GROUP: [String], ...}This wrapper function may need to write inputs to, and read outputs from, files or tables, but should beexternally purein the sense that any changes made are rolled back before returning, including any made by the unit under test, and should beessentiallydeterministic.The diagram shows the flows between input and output files:Input JSON file (yellow)Output JSON file (yellow)Formatted unit test reports (blue)and the four code components, where the design pattern centralizes as much code as possible in the library packages:JavaScript Trapit library package for formatting results (dark green)External (Python) library package for unit testing (light green)Specific (Python) test package (tan)Unit under test (Python) (rose)Unit Testing Process (General)↑ General UsageOnce the preliminary steps are executed, the script (testuut.py, say) can be executed as follows:$py[path]/testuut.pyThe output results files are processed by a JavaScript program that has to be installed separately, as described in theInstallationsection. The JavaScript program produces listings of the results in HTML and/or text format in a subfolder named from the unit test title.To run the processor, go to the npm trapit package folder after placing the output JSON files, trapit_py_out.json, in a new (or existing) folder, python, within the subfolder externals and run:$ node externals/format-externals pythonThis outputs to screen the following summary level report (for both examples described below), as well as writing the formatted results files to the subfolders indicated:Unit Test Results Summary for Folder ./externals/python
=======================================================
 File                 Title        Inp Groups  Out Groups  Tests  Fails  Folder     
--------------------  -----------  ----------  ----------  -----  -----  -----------
*colgroup_out.json    Col Group             3           4      5      1  col-group  
 helloworld_out.json  Hello World           0           1      1      0  hello-world

1 externals failed, see ./externals/python for scenario listings
colgroup_out.jsonThe running of the python unit test, and its Javascript formatting can easily be automated, as in the following Powershell script in the examples folder:$ ./Run-Examples.ps1This example script runs both example unit tests and then the Javascript formatter, assuming a hard-coded npm root folder, and writes the summary to a file python.log.Unit Test Documentation↑ General UsageIn documenting our unit testing it may be helpful to divide into four sections: The unit testing process; wrapper function design and code; scenario category analysis; unit test results. The heading structure might look as follows:Unit Testing ProcessUnit Test Wrapper FunctionWrapper Function Signature DiagramInput JSON FileWrapper Function CodeScenario Category ANalysis (SCAN)Simple Category SetsComposite Category SetsScenario Category MappingUnit Test ResultsResults SummaryUnit Test Report: TitleExample 1 - colgroup↑ Usage↓ Unit Testing Process - colgroup↓ Unit Test Wrapper Function - colgroup↓ Scenario Category ANalysis (SCAN) - colgroup↓ Unit Test Results - colgroupThis example is a python class with a constructor function that reads in a CSV file and counts instances of distinct values in a given column. The constructor function appends a timestamp and call details to a log file. The class has methods to list the value/count pairs in several orderings.There is a main script that shows how the class might be called outside of unit testing, run from the module root folder:$pyexamples/colgroup/maincolgroup.pywith output to console:Counts sorted by (as is)
========================
Team         #apps
-----------  -----
team_name_2      1
team_name_1      1
West Brom     1219
Swansea       1180
Blackburn       33
Bolton          37
Chelsea       1147
Arsenal        534
Everton       1147
Tottenham     1288
Fulham        1209
QPR           1517
Liverpool     1227
Sunderland    1162
Man City      1099
Man Utd       1231
Newcastle     1247
Stoke City    1170
Wolves          31
Aston Villa    685
Wigan         1036
Norwich       1229
West Ham      1126
Reading       1167
...and to log file, fantasy_premier_league_player_stats.csv.log:Sun Sep 23 2018 13:29:07: File ./examples/colgroup/fantasy_premier_league_player_stats.csv, delimiter ',', column 6The example illustrates how a wrapper function can handleimpurefeatures of the unit under test:Reading input from fileWriting output to file...and also how the JSON input file can allow for nondeterministic outputs giving rise to deterministic test outcomes:By using regex matching for strings including timestampsBy using number range matching and converting timestamps to epochal offsets (number of units of time since a fixed time)Unit Testing Process - colgroup↑ Example 1 - colgroupTo run the unit test program from the module root folder:$pyexamples/colgroup/testcolgroup.pyThe output result file is processed by a JavaScript program as explained in theGeneral Usagesection above. It outputs to screen a summary level report (for both examples), as well as writing the listings of the results in HTML and/or text format in a subfolder named from the unit test title, as specified in the input JSON file.The sectionUnit Testing Process (General), above, shows how to combine the running of the python script and the JavaScript formatter in a single powershell script.Unit Test Wrapper Function - colgroup↑ Example 1 - colgroup↓ WF Signature Diagram - colgroup↓ Input JSON File - colgroup↓ Wrapper Function Code - colgroupWF Signature Diagram - colgroup↑ Unit Test Wrapper Function - colgroupThe JSON input file containsmetaandscenariosproperties, as mentioned above, with structure reflecting the (extended) inputs and outputs of the unit under test. I like to make a diagram of the input and output groups, which for this example is:Input JSON File - colgroup↑ Unit Test Wrapper Function - colgroupAn easy way to generate a starting point for the input JSON file is to use a powershell utilityPowershell Utilites moduleto generate a template file with a single scenario with placeholder records from simple CSV files. The CSV files,colgroup_inp.csv, containing input group, field pairs, and the second,colgroup_out.csv, the same for output for the JSON structure diagram above would look like this:The powershell utility can be run from a powershell window like this:Import-ModuleTrapitUtilsWrite-UT_Template'colgroup''|'This generates a JSON template file, colgroup_temp.json.The template is then updated with test data for 5 scenarios (showing just the first one here):{"meta":{"title":"Col Group","inp":{"Log":["Line"],"Scalars":["Delimiter","Column#"],"Lines":["Line"]},"out":{"Log":["#Lines","Date Offset","Text"],"listAsIs":["#Instances"],"sortByKey":["Key","Value"],"sortByValue":["Key","Value"]}},"scenarios":{"Col 1/3; 2 duplicate lines; double-delimiter; 1-line log":{"active_yn":"Y","inp":{"Log":[],"Scalars":[",|2"],"Lines":["0,1,Cc,3","00,1,A,9","000,1,B,27","0000,1,A,81"]},"out":{"Log":["1|IN [0, 2000]|LIKE /.*: File ./examples/colgroup/ut_group.csv, delimiter ',', column 2/"],"listAsIs":["3"],"sortByKey":["A|2","Bx|1","Cc|1"],"sortByValue":["B|1","Cc|1","A|2"]}},...3morescenarios}}Notice the syntax for the expected values for the second and third fields in the 3-field output record for the log group. This specifies matching against a numeric range and a regular expression, respectively, as follows:Date Offset: "IN [0, 2000]" - the datetime offset in microseconds must be between 0 and 2000 microseconds from the datetime at the start of executionText: "LIKE /.*: File ./examples/colgroup/ut_group.csv, delimiter ',', column 2/" - the line of text written must match the regular expression betwen the '/' delimiters, allowing us to ignore the precise timestamp for testing purposes, but still to display it for informationWrapper Function Code - colgroup↑ Unit Test Wrapper Function - colgroupThe text box below shows the entire specific unit test code for this example (short isn't it? 😁) containing the pure wrapper function, purely_wrap_unit, and the one line main section calling the library function, trapit.test_unit.importsys,osfromdatetimeimportdatetimesys.path.insert(0,os.path.abspath(os.path.join(os.path.dirname(__file__),'../..')))importtrapit,colgroupascgROOT=os.path.dirname(__file__)+'/'DELIM='|'INPUT_JSON,OUTPUT_JSON,INPUT_FILE,LOG_FILE=\ROOT+'colgroup.json',ROOT+'colgroup_out.json',ROOT+'ut_group.csv',ROOT+'ut_group.csv.log'GRP_LOG,GRP_SCA,GRP_LIN,GRP_LAI,GRP_SBK,GRP_SBV=\'Log','Scalars','Lines','listAsIs','sortByKey','sortByValue'deffrom_CSV(csv,col):returncsv.split(DELIM)[col]defjoin_tuple(t):returnt[0]+DELIM+str(t[1])defsetup(inp):withopen(INPUT_FILE,'w')asinfile:infile.write('\n'.join(inp[GRP_LIN]))if(len(inp[GRP_LOG])>0):withopen(LOG_FILE,'w')aslogfile:logfile.write('\n'.join(inp[GRP_LOG])+'\n')returncg.ColGroup(INPUT_FILE,from_CSV(inp[GRP_SCA][0],0),from_CSV(inp[GRP_SCA][0],1))defteardown():os.remove(INPUT_FILE)os.remove(LOG_FILE)defpurely_wrap_unit(inp_groups):col_group=setup(inp_groups)withopen(LOG_FILE,'r')aslogfile:logstr=logfile.read()lines_array=logstr.split('\n')lastLine=lines_array[len(lines_array)-2]text=lastLinedate=lastLine[0:19]logDate=datetime.strptime(date,'%Y-%m-%d%H:%M:%S')now=datetime.now()diffDate=(now-logDate).microseconds/1000teardown()return{GRP_LOG:[str((len(lines_array)-1))+DELIM+str(diffDate)+DELIM+text],GRP_LAI:[str(len(col_group.list_as_is()))],GRP_SBK:list(map(join_tuple,col_group.sort_by_key())),GRP_SBV:list(map(join_tuple,col_group.sort_by_value_lambda()))}trapit.test_unit(INPUT_JSON,OUTPUT_JSON,purely_wrap_unit)Scenario Category ANalysis (SCAN) - colgroup↑ Example 1 - colgroup↓ Simple Category Sets - colgroup↓ Composite Category Sets - colgroup↓ Scenario Category Mapping - colgroupThis article,Unit Testing, Scenarios and Categories: The SCAN Method, explains how to derive unit test scenarios using a new approach called the SCAN method.Following the method, in this section we identify the category sets for the problem, and tabulate the corresponding categories. We need to consider which category sets can be tested independently of each other, and which need to be considered in combination. We can then obtain a set of scenarios to cover all relevant combinations of categories.Simple Category Sets - colgroup↑ Scenario Category ANalysis (SCAN) - colgroupMUL-LIN - Multiplicity of linesCheck works correctly with 0, 1 and multiple lines.CodeDescription0None1OnemMultiplePOS-KEY - Position of key columnCheck works correctly when the key column is first, last or in the middle.CodeDescriptionFFirstLLastMMiddleMUL-KEY - Multiplicity of key instancesCheck works correctly with 1 and multiple key instances.CodeDescription1OnemMultipleMUL-COL - Multiplicity of file columnsCheck works correctly with 1 and multiple columns in file.CodeDescription1OnemMultipleMUL-DEL - Multiplicity of delimiter characterCheck works correctly with 1 and multiple delimiter character.CodeDescription1OnemMultipleSIZ - Size of keyCheck works correctly with short and long key values.CodeDescriptionSShortLLongLOG  - Log file existenceCheck works correctly when there is already a log file and when there isn't.CodeDescriptionNNo - file does not exist at time of callYNo - file exists at time of callORD-SAM  - Ordering same by key and by value?Check ordering methods work when order of output records same by key and value and when differs.CodeDescriptionNOrder by key differs from order by valueYOrder by key same as order by valueComposite Category Sets - colgroup↑ Scenario Category ANalysis (SCAN) - colgroupIn this section we need to consider which simple category sets need to be considered in combination with others. In fact, in this case, all the category sets other thanMultiplicity of linesare independent of each other, and have a very simple dependence onMultiplicity of lines: There has to be at least one line in the file to test the other category sets, and multiple lines to test a couple of them.AsPosition of key columnhas three categories, we can test these with multiple lines, as well as testing the zero-lines edge case and the 1-line case. The possible combinations of these two category sets can then be the basis for our scenarios, and we can just enumerate the other categories within them.MUL-LK - Multiplicity of lines and key instancesCheck works correctly with 0, 1 and multiple lines, and for multiple lines all three categories ofPosition of key column.MUL-LINPOS-KEYDescription0-Lines: None; Key column: NA1FLines: 1; Key column: FirstmFLines: Multiple; Key column: FirstmLLines: Multiple; Key column: LastmMLines: Multiple; Key column: MiddleScenario Category Mapping - colgroup↑ Scenario Category ANalysis (SCAN) - colgroupWe now want to construct a set of scenarios based on the category sets identified, covering each individual category, and also covering combinations of categories that may interact. As discussed in the previous section, the possible combinations of the first two category sets form the basis for the category-level scenario set, with the remaining category sets enumerated in the additional columns.#MUL-LINPOS-KEYMUL-KEYMUL-COLMUL-DELSIZLOGORD-SAMDescription10-------Lines: None; Key column: NA21F111SN-Lines: 1; Key column: First3mFmmmLYNLines: Multiple; Key column: First4mLmmmLYYLines: Multiple; Key column: Last5mMmmmLYNLines: Multiple; Key column: MiddleUnit Test Results - colgroup↑ Example 1 - colgroup↓ Results Summary - colgroup↓ Unit Test Report: Col GroupResults Summary - colgroup↑ Unit Test Results - colgroupThe results summary from the JavaScript test formatter was (for both examples):Unit Test Results Summary for Folder ./externals/python
=======================================================
 File                 Title        Inp Groups  Out Groups  Tests  Fails  Folder     
--------------------  -----------  ----------  ----------  -----  -----  -----------
*colgroup_out.json    Col Group             3           4      5      1  col-group  
 helloworld_out.json  Hello World           0           1      1      0  hello-world

1 externals failed, see ./externals/python for scenario listings
colgroup_out.jsonYou can review the HTML formatted unit test results for the program here:Unit Test Report: Col GroupThe formatted results files, both text and HTML, are available in thecol-groupsubfolder. The summary report showing scenarios tested, in text format, along with the detailed report for scenario 5, are copied below:Unit Test Report: Col Group↑ Unit Test Results - colgroupUnit Test Report: Col Group
===========================

      #    Scenario                             Fails (of 4)  Status 
      ---  -----------------------------------  ------------  -------
      1    Lines: None; Key column: NA          0             SUCCESS
      2    Lines: 1; Key column: First          0             SUCCESS
      3    Lines: Multiple; Key column: First   0             SUCCESS
      4    Lines: Multiple; Key column: Last    0             SUCCESS
      5*   Lines: Multiple; Key column: Middle  1             FAILURE

Test scenarios: 1 failed of 5: FAILURE
======================================Note the record #5 above marked with a '*' indicating failure status. The detailed report for the fifth scenario, in text format, is copied below:SCENARIO 5: Lines: Multiple; Key column: Middle {
=================================================
   INPUTS
   ======
      GROUP 1: Log {
      ==============
            #  Line      
            -  ----------
            1  Log line 1
      }
      =
      GROUP 2: Scalars {
      ==================
            #  Delimiter  Column#
            -  ---------  -------
            1  ;;         5      
      }
      =
      GROUP 3: Lines {
      ================
            #  Line                                                                         
            -  -----------------------------------------------------------------------------
            1  0;;1;;2;;3;;4;;12345678901234567890123456789012345678901234567890;;5;;6;;7;;8
            2  0;;1;;2;;3;;4;;abc;;5;;6;;7;;8                                               
            3  0;;1;;2;;3;;4;;12345678901234567890123456789012345678901234567890;;5;;6;;7;;8
      }
      =
   OUTPUTS
   =======
      GROUP 1: Log {
      ==============
            #  #Lines  Date Offset           Text                                                                                                                                                                                                           
            -  ------  --------------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
            1  2       IN [0,2000]: 480.149  LIKE /.*: File .*examples-colgroup-ut_group.csv, delimiter ';;', column 5/: 2021-11-20 16:14:04: File C:-Users-Brend-OneDrive-Script-pip-trapit-trapit-examples-colgroup-ut_group.csv, delimiter ';;', column 5
      } 0 failed of 1: SUCCESS
      ========================
      GROUP 2: listAsIs {
      ===================
            #   #Instances
            --  ----------
            1   3         
            1*  2         
      } 1 failed of 1: FAILURE
      ========================
      GROUP 3: sortByKey {
      ====================
            #  Key                                                 Value
            -  --------------------------------------------------  -----
            1  12345678901234567890123456789012345678901234567890  2    
            2  abc                                                 1    
      } 0 failed of 2: SUCCESS
      ========================
      GROUP 4: sortByValue {
      ======================
            #  Key                                                 Value
            -  --------------------------------------------------  -----
            1  abc                                                 1    
            2  12345678901234567890123456789012345678901234567890  2    
      } 0 failed of 2: SUCCESS
      ========================
} 1 failed of 4: FAILURE
========================Note the record #1 above marked with a '*' in 'GROUP 2: listAsIs', indicating a mismatch between expected and actual values. This is a deliberate error to illustrate the format when mismatches occur. Where the actual value differs from expected the actual record is listed below the expected, with the '*' marker against the record number, and in the HTML report the record is coloured red. In fact the value '2' is correct and the expected value has been incorrectly set to '3'.Example 2 - hello_world↑ Usage↓ Unit Testing Process - helloworld↓ Unit Test Wrapper Function - helloworld↓ Scenario Category ANalysis (SCAN) - helloworld↓ Unit Test Results - helloworlddefhello_world():return'Hello World!'This is a pure function form of Hello World program, returning a value rather than writing to screen itself. It is of course trivial, but has some interest as an edge case with no inputs and extremely simple JSON input structure and test code.There is a main script that shows how the function might be called outside of unit testing, run from the module root folder:$pyexamples/helloworld/mainhelloworld.pywith output to console:Hello World!Unit Testing Process - helloworld↑ Example 2 - hello_worldTo run the unit test program from the module root folder:$pyexamples/helloworld/testhelloworld.pyThe output result file is processed by a JavaScript program as explained in theGeneral Usagesection above. It outputs to screen a summary level report (for both examples), as well as writing the listings of the results in HTML and/or text format in a subfolder named from the unit test title, as specified in the input JSON file.The sectionUnit Testing Process (General), above, shows how to combine the running of the python script and the JavaScript formatter in a single powershell script.Unit Test Wrapper Function - helloworld↑ Example 2 - hello_world↓ WF Signature Diagram - helloworld↓ Input JSON File - helloworld↓ Wrapper Function Code - helloworldWF Signature Diagram - helloworld↑ Unit Test Wrapper Function - helloworldThe JSON structure diagram for this trivial example is:Input JSON File - helloworld↑ Unit Test Wrapper Function - helloworldThe input JSON file, showing empty input property in the meta and scenarios objects, is:↑ Input JSON File{"meta":{"title":"Hello World","inp":{},"out":{"Group":["Greeting"]}},"scenarios":{"Scenario":{"inp":{},"out":{"Group":["Hello World!"]}}}}Wrapper Function Code - helloworld↑ Unit Test Wrapper Function - helloworldThe text box below shows the entire specific unit test code for this example. In this trivial case, we can pass the pure wrapper function as a lambda expression.importsys,ossys.path.insert(0,os.path.abspath(os.path.join(os.path.dirname(__file__),'../..')))importtrapit,helloworldROOT=os.path.dirname(__file__)+'/'INPUT_JSON=ROOT+'helloworld.json'OUTPUT_JSON=ROOT+'helloworld_out.json'trapit.test_unit(INPUT_JSON,OUTPUT_JSON,lambdainp_groups:{'Group':[helloworld.hello_world()]})Scenario Category ANalysis (SCAN) - helloworld↑ Example 2 - hello_worldWith no input data, the set of input data category sets is of course empty ðŸ™‚.Unit Test Results - helloworld↑ Example 2 - hello_world↓ Results Summary - helloworld↓ Unit Test Report: Hello WorldResults Summary - helloworld↑ Unit Test Results - helloworldThe results summary from the JavaScript test formatter was (for both examples):Unit Test Results Summary for Folder ./externals/python
=======================================================
 File                 Title        Inp Groups  Out Groups  Tests  Fails  Folder     
--------------------  -----------  ----------  ----------  -----  -----  -----------
*colgroup_out.json    Col Group             3           4      5      1  col-group  
 helloworld_out.json  Hello World           0           1      1      0  hello-world

1 externals failed, see ./externals/python for scenario listings
colgroup_out.jsonYou can review the HTML formatted unit test results for the program here:Unit Test Report: Hello WorldThe formatted results files, both text and HTML, are available in thehello-worldsubfolder. Here is the full set of results in text format:Unit Test Report: Hello World↑ Unit Test Results - helloworldUnit Test Report: Hello World
=============================
      #    Scenario  Fails (of 1)  Status 
      ---  --------  ------------  -------
      1    Scenario  0             SUCCESS
Test scenarios: 0 failed of 1: SUCCESS
======================================
SCENARIO 1: Scenario {
======================
   INPUTS
   ======
   OUTPUTS
   =======
      GROUP 1: Group {
      ================
            #  Greeting    
            -  ------------
            1  Hello World!
      } 0 failed of 1: SUCCESS
      ========================
} 0 failed of 1: SUCCESS
========================API↑ In This README...↓ trapit.test_unit(inp_file, out_file, purely_wrap_unit)importtrapittrapit.test_unit(inp_file, out_file, purely_wrap_unit)↑ APIThe unit test driver utility function is called as effectively the main function of any specific unit test script. It reads the input JSON scenarios file, then loops over the scenarios making calls to a function passed in as a parameter from the calling script. The function acts as apurewrapper around calls to the unit under test. It isexternally purein the sense that it is deterministic, and interacts externally only via parameters and return value. Where the unit under test reads inputs from file the wrapper writes them based on its parameters, and where the unit under test writes outputs to file the wrapper reads them and passes them out in its return value. Any file writing is reverted before exit.It has the following parameters:inp_file: JSON input file nameout_file: JSON output file namepurely_wrap_unit: wrapper function, which calls the unit under test passing the appropriate parameters and returning its outputs, with the following signature:inp_groups: input groups object, a 3-level list with test inputs as an object with groups as properties having 2-level arrays of record/field as values: {GROUP: [[String]], ...}Return Value: output groups object, a 2-level list with test outputs as an object with groups as properties having an array of records as delimited fields strings as value: {GROUP: [String], ...}Installation↑ In This README...↓ Python Installation - pip↓ Javascript Installation - npmPython Installation - pip↑ InstallationWithpythoninstalled, run in a powershell or command window:$py-mpipinstalltrapitJavascript Installation - npm↑ InstallationTrapit JavaScript Tester/Formatter - GitHub moduleWithnpminstalled, run from your npm installation folder:$npminstalltrapitUnit Testing↑ In This README...↓ Unit Testing Process↓ Wrapper Function↓ Scenario Category ANalysis (SCAN)↓ Unit Test ResultsIn this section the unit testing API function trapit.test_unit is itself tested using the Math Function Unit Testing design pattern.Unit Testing Process↑ Unit TestingThe unit test utility can be used to test itself following the same 'Math Function Unit Testing design pattern' that it facilitates for testing of general programs. The challenge in this case is in determining a suitable signature and specification for the wrapper function that has to represent unit testing of any program.Unit testing is data-driven from the input file trapit_py.json and produces an output results file, trapit_py_out.json. This contains arrays of expected and actual records by group and scenario.To run the unit test program from the module root folder:$pyunit_test/testtrapit.pyThe output result file is processed by a JavaScript program as explained in theGeneral Usagesection above. It outputs to screen a summary level report (including for both of the earlier examples), as well as writing the listings of the results in HTML and/or text format in a subfolder named from the unit test title, as specified in the input JSON file.To run the processor, go to the npm trapit package folder after placing the output JSON files, trapit_py_out.json, in a new (or existing) folder, python, within the subfolder externals and run:$nodeexternals/format-externalspythonThis outputs to screen the following summary level report, as well as writing the formatted results files to the subfolders indicated:Unit Test Results Summary for Folder ./externals/python
=======================================================
 File                 Title               Inp Groups  Out Groups  Tests  Fails  Folder            
--------------------  ------------------  ----------  ----------  -----  -----  ------------------
*colgroup_out.json    Col Group                    3           4      5      1  col-group         
 helloworld_out.json  Hello World                  0           1      1      0  hello-world       
 trapit_py_out.json   Python Unit Tester           7           6      3      0  python-unit-tester

1 externals failed, see ./externals/python for scenario listings
colgroup_out.jsonThe running of the python unit test, and its Javascript formatting can easily be automated, as in the following Powershell script in the unit_test folder:$ ./Run-Ut.ps1This script runs the unit test and then the Javascript formatter, assuming a hard-coded npm root folder, and writes the summary to a file, python.log.Wrapper Function↑ Unit Testing↓ Wrapper Function Signature Diagram↓ Input JSON File↓ Wrapper Function CodeThe signature of the unit under test is:trapit.test_unit(inp_file, out_file, purely_wrap_unit)The parameters are input and output file names, and a function. Theextendedinputs and outputs required for the wrapper function include the contents of the input and output files.Wrapper Function Signature Diagram↑ Wrapper FunctionAs noted above, the inputs to the unit under test here include a function. This raises the interesting question as to how we can model a function in our test data. In fact the best way to do this seems to be to regard the function as a kind of black box, where we don't care about the interior of the function, but only its behaviour in terms of returning an output from an input. This is why we have theActual Valuesgroup in the input side of the diagram above, as as well as on the output side. We can model any deterministic function in our test data simply by specifying input and output sets of values.As we are using the trapit.test_unit API to test itself, we will have inner and outer levels for the calls and their parameters. The inner-level wrapper function passed in in the call to the unit under test by the outer-level wrapper function therefore needs simply to return the set ofActual Valuesrecords for the given scenario. In order for it to know which set to return, the scenarios need to be within readable scope, and we need to know which scenario to use. This is achieved by maintaining arrays containing a list of inner scenarios and a list of inner output groups, along with a nonlocal variable with an index to the current inner scenario that the inner wrapper increments each time it's called. This allows the output array to be extracted from the input parameter from the outer wrapper function.Input JSON File↑ Wrapper FunctionAn easy way to generate a starting point for the input JSON file is to use a powershell utilityPowershell Utilites moduleto generate a template file with a single scenario with placeholder records from simple CSV files (see the script test_unit.ps1 in thetestsubfolder). The CSV files,test_unit_inp.csv, containing input group, field pairs, and the second,test_unit_out.csv, the same for output for the JSON structure diagram above would look like this:The powershell utility can be run from a powershell window like this:Import-ModuleTrapitUtilsWrite-UT_Template'test_unit''|'This generates a JSON template file, test_unit_temp.json. The template is then updated with test data for the four scenarios identified in theScenario Category Analysissection.Wrapper Function Code↑ Wrapper Function↓ testtrapit.py↓ purely_wrap_unit↓ write_input_json↓ get_actuals↓ Small functionsThe wrapper function has the structure shown in the diagram below, being defined in a driver script followed by a single line calling the test_unit API.testtrapit.py↑ Wrapper Function CodeThe text box below shows the code for the driving script, with the wrapper function def line as a placeholder for later expansion.sys.path.insert(0,os.path.abspath(os.path.join(os.path.dirname(__file__),'..')))importtrapitROOT=os.path.dirname(__file__)+'\\'DELIM='|'INP_JSON,OUT_JSON,INP_JSON_INNER,OUT_JSON_INNER=\ROOT+'trapit_py.json',ROOT+'trapit_py_out.json',ROOT+'trapit_py_inner.json',ROOT+'trapit_py_out_inner.json'TITLE,DELIMITER,ACTIVE_YN,UNIT_TEST,META,SCENARIOS,INP,OUT,EXP,ACT=\'title','delimiter','active_yn','Unit Test','meta','scenarios','inp','out','exp','act'INP_FIELDS,OUT_FIELDS,INP_VALUES,EXP_VALUES,ACT_VALUES=\'Input Fields','Output Fields','Input Values','Expected Values','Actual Values'defpurely_wrap_unit(inp_groups,# input groups objectscenario):# scenario keytrapit.test_unit(INP_JSON,OUT_JSON,purely_wrap_unit)purely_wrap_unit↑ Wrapper Function CodeThis is the outer level unit test wrapper function, returning an object with the output group objects actual values from the unit under test for a single scenario. It defines several local functions, including an inner level wrapper function, purely_wrap_unit_inner (whose
bodies are shown later).defpurely_wrap_unit(inp_groups):# input groups objectdefgroups_from_group_field_pairs(group_field_lis):# group/field pairs listdefgroups_obj_from_gf_pairs(group_lis,# groups listgroup_field_lis):# group/field pairs listdefgroups_obj_from_sgf_triples(sce,# scenariogroup_lis,# groups listsgf_triple_lis):# scenario/group/field triples listdefpurely_wrap_unit_inner(inp_groups_inner)# input groups object (inner level)defwrite_input_json():defget_actuals():out_group_lis,sce_inp_lis=write_input_json()sce_inp_ind=0trapit.test_unit(INP_JSON_INNER,OUT_JSON_INNER,purely_wrap_unit_inner)returnget_actuals()write_input_json↑ Wrapper Function CodeThis function writes out the inner level JSON file. It returns two objects: a list of (inner) output groups, and a list of (inner) scenarios; these are referenced in purely_wrap_unit_inner.defwrite_input_json():inp_group_field_lis=inp_groups[INP_FIELDS]inp_group_lis=groups_from_group_field_pairs(inp_group_field_lis)out_group_field_lis=inp_groups[OUT_FIELDS]out_group_lis=groups_from_group_field_pairs(out_group_field_lis)title,delimiter=inp_groups[UNIT_TEST][0].split(DELIM)meta={TITLE:title,DELIMITER:delimiter,INP:groups_obj_from_gf_pairs(inp_group_lis,inp_group_field_lis),OUT:groups_obj_from_gf_pairs(out_group_lis,out_group_field_lis)}scenarios={}sce_inp_lis=[]fors_rowininp_groups['Scenario']:sce,active_yn=s_row.split(DELIM)ifactive_yn=='Y':sce_inp_lis.append(sce)sce_inp=groups_obj_from_sgf_triples(sce,inp_group_lis,inp_groups[INP_VALUES])sce_out=groups_obj_from_sgf_triples(sce,out_group_lis,inp_groups[EXP_VALUES])scenarios[sce]={ACTIVE_YN:active_yn,INP:sce_inp,OUT:sce_out}inp_json_obj={META:meta,SCENARIOS:scenarios}withopen(INP_JSON_INNER,'w')asinp_f:json.dump(inp_json_obj,inp_f,indent=4)return[out_group_lis,sce_inp_lis]get_actuals↑ Wrapper Function CodeThis function extract the actual results from the JSON output file created by the inner level call to trapit.test_unit. It returns an object with output groups as keys and actual values lists as values for given scenario.defget_actuals():withopen(OUT_JSON_INNER,encoding='utf-8')asout_f:out_json_obj=json.loads(out_f.read())meta,scenarios=out_json_obj[META],out_json_obj[SCENARIOS]g_unit_test=[meta[TITLE]+DELIM+meta[DELIMITER]]g_inp_fields,g_out_fields,g_inp_values,g_exp_values,g_act_values=[],[],[],[],[]forginmeta[INP]:foriinmeta[INP][g]:g_inp_fields.append(g+DELIM+i)forginmeta[OUT]:foriinmeta[OUT][g]:g_out_fields.append(g+DELIM+i)forsinscenarios:forginscenarios[s][INP]:foriinscenarios[s][INP][g]:g_inp_values.append(s+DELIM+g+DELIM+i)forginscenarios[s][OUT]:foriinscenarios[s][OUT][g][EXP]:g_exp_values.append(s+DELIM+g+DELIM+i)foriinscenarios[s][OUT][g][ACT]:g_act_values.append(s+DELIM+g+DELIM+i)os.remove(INP_JSON_INNER)os.remove(OUT_JSON_INNER)return{UNIT_TEST:g_unit_test,INP_FIELDS:g_inp_fields,OUT_FIELDS:g_out_fields,INP_VALUES:g_inp_values,EXP_VALUES:g_exp_values,ACT_VALUES:g_act_values}Small functions↑ Wrapper Function Codegroups_from_group_field_pairsThis function returns a list of distinct groups from an input list of group/field pairs.defgroups_from_group_field_pairs(group_field_lis):# group/field pairs listreturnlist(dict.fromkeys([gf.split(DELIM)[0]forgfingroup_field_lis]))groups_obj_from_gf_pairsThis function returns an object with groups as keys and field lists as values, based on input lists of groups and group/field pairs.defgroups_obj_from_gf_pairs(group_lis,# groups listgroup_field_lis):# group/field pairs listobj={}forgingroup_lis:gf_pairs=filter(lambdagf:gf[:len(g)]==g,group_field_lis)obj[g]=[gf[len(g)+1:]forgfingf_pairs]returnobjgroups_obj_from_sgf_triplesThis function returns an object with groups as keys and field lists as values for given scenario, based on input scenario and lists of groups and scenario/group/field triplesdefgroups_obj_from_sgf_triples(sce,# scenariogroup_lis,# groups listsgf_triple_lis):# scenario/group/field triples listthis_sce_pairs=list(filter(lambdag:g[:len(sce)]==sce,sgf_triple_lis))group_field_lis=[p[len(sce)+1:]forpinthis_sce_pairs]returngroups_obj_from_gf_pairs(group_lis,group_field_lis)purely_wrap_unit_innerThis function is the inner level unit test wrapper function, returning an object with the output group objects 'actual' values from unit under test, which is here trapit.test_unit. It returns the 'Actual Values' group values specified in the outer level for the given scenario, ignoring the (required) input groups parameter in this special case. It references two arrays held in the scope of the outer level wrapper function, and also an index into the scenarios list that has the same outer level scope.defpurely_wrap_unit_inner(inp_groups_inner):# input groups object (inner level)nonlocalsce_inp_indscenario_inner=sce_inp_lis[sce_inp_ind]sce_inp_ind+=1returngroups_obj_from_sgf_triples(scenario_inner,out_group_lis,inp_groups[ACT_VALUES])Scenario Category ANalysis (SCAN)↑ Unit Testing↓ Simple Category Sets↓ Composite Category Sets↓ Scenario Category MappingThe art of unit testing lies in choosing a set of scenarios that will produce a high degree of confidence in the functioning of the unit under test across the often very large range of possible inputs.A useful approach to this is to think in terms of categories of inputs, where we reduce large ranges to representative categories. Categories are chosen to explore the full range of potential behaviours of the unit under test.In this section we identify the category sets for the problem, and tabulate the corresponding categories. We need to consider which category sets can be tested independently of each other, and which need to be considered in combination. We can then obtain a set of scenarios to cover all relevant combinations of categories.Simple Category Sets↑ Scenario Category ANalysis (SCAN)↓ SAF - Scenario active flag↓ MUL-0 - Multiplicity including zero↓ MUL-1 - Multiplicity excluding zero↓ INV - Invalidity TypeIn this section we identify some simple category sets to apply.SAF - Scenario active flag↑ Simple Category SetsWe want to check that active scenarios are processed while inactive ones are ignored.CodeDescriptionYScenario activeNScenario inactiveMUL-0 - Multiplicity including zero↑ Simple Category SetsWe want to check behaviour when there are 0, 1, or more than 1, records for each entity, for those entities where zero multiplicity makes sense.CodeDescription0Zero values11 valuemMultiple valuesThis category set is applied to the following entities:Category SetDescriptionIGMInput group multiplicityOGMOutput group multiplicityIVMInput value multiplicityOVMOutput value multiplicityMUL-1 - Multiplicity excluding zero↑ Simple Category SetsWe want to check behaviour when there are 1, or more than 1, records for each entity, for those entities where zero multiplicity does not make sense.CodeDescription11 valuemMultiple valuesThis category set is applied to the following entities:Category SetDescriptionSCMScenario multiplicityIFMInput field multiplicityOFMOutput field multiplicityDCMDelimiter characters multiplicityINV - Invalidity Type↑ Simple Category SetsA unit test returns a status of Failure if any output group returns a status of Failure, which happens when the actual output set of records differs from the expected output set.We can categorise types of invalidity by set cardinality differences (with E for expected set cardinality, and A for actual set cardinality). We want to check behaviour for each type of invalidity as well as the valid case.CodeDescriptionVALSame cardinalities and all records the sameE=ASame cardinalities but at least one record differs in valueE>AMore records in expected set than in actual setA>EMore records in actual set than in expected setComposite Category Sets↑ Scenario Category ANalysis (SCAN)↓ IGM / OGM↓ SCM / SAFThe category sets considered can be treated as largely independent, with the exception that having zero multiplicity for both input and output groups doesn't make sense, and making a scenario inactive means nothing else can be tested within that scenario. Therefore we can take the following combinations of the category sets IGM, OGM as the basis of our scenario category mapping, and ensure that the combinations noted below of SCM and SAF are handled.The Invalidity Type category set could be tested within the third scenario, but we'll add a fourth scenario for greater clarity.IGM / OGM↑ Composite Category SetsEnsure the zero edge case for input groups and output groups are handled separately, and note that 0 for either group implies no fields are possible.IGMOGMIFMOFM01-10-mmSCM / SAF↑ Composite Category SetsEnsure that the inactive scenario category occurs within a multi-scenario situation, so that other categories can be simultaneously tested.SCMSAF1YmNScenario Category Mapping↑ Scenario Category ANalysis (SCAN)We now want to construct a set of scenarios based on the category sets identified, covering each individual category, and also covering combinations of categories that may interact.In this case, the first four category sets may be considered as a single composite set with the combinations listed below forming the scenario keys, while the two SIZ categories are covered in the first two scenarios.#IGMOGMIVMOVMSCMIFMOFMDCMSAFINVDescription101-01-11YVALZero input groups, 1 of other entities where possible; active scenario; valid2101-11-1YVALZero output groups, 1 of other entities where possible; active scenario; valid3mmmmmmmmNVALMultiple entities; one inactive scenario; all valid411mmmmmmY*One input and output groups; multiple other entities; active scenarios; each type of invalid scenarioUnit Test Results↑ Unit Testing↓ Results Summary↓ Unit Test Report: Python Unit TesterResults Summary↑ Unit Test ResultsUnit Test Results Summary for Folder ./externals/python
=======================================================
 File                 Title               Inp Groups  Out Groups  Tests  Fails  Folder            
--------------------  ------------------  ----------  ----------  -----  -----  ------------------
*colgroup_out.json    Col Group                    3           4      5      1  col-group         
 helloworld_out.json  Hello World                  0           1      1      0  hello-world       
 trapit_py_out.json   Python Unit Tester           7           6      4      0  python-unit-tester

1 externals failed, see ./externals/python for scenario listings
colgroup_out.jsonYou can review the HTML formatted unit test results here:Unit Test Report: Python Unit TesterUnit Test Report: Python Unit Tester↑ Unit Test ResultsUnit Test Report: Python Unit Tester
====================================

      #    Scenario                                                                                               Fails (of 6)  Status 
      ---  -----------------------------------------------------------------------------------------------------  ------------  -------
      1    Zero input groups, 1 of other entities where possible; active scenario; valid                          0             SUCCESS
      2    Zero output groups, 1 of other entities where possible; active scenario; valid                         0             SUCCESS
      3    Multiple entities; one inactive scenario; all valid                                                    0             SUCCESS
      4    One input and output groups; multiple other entities; active scenarios; each type of invalid scenario  0             SUCCESS

Test scenarios: 0 failed of 4: SUCCESS
======================================Here are the output results for the first scenario (slightly edited for brevity):SCENARIO 1: Zero input groups, 1 of other entities where possible; active scenario; valid {
===========================================================================================
   INPUTS
   ======
      GROUP 1: Unit Test {
      ====================
            #  Title        Delimiter
            -  -----------  ---------
            1  Inner title  ;        
      }
      GROUP 2: Input Fields: Empty
      ============================
      GROUP 3: Output Fields {
      ========================
            #  Group           Field         
            -  --------------  --------------
            1  Output Group 1  Output Field 1
      }
      GROUP 4: Scenario {
      ===================
            #  Scenario          Active Y/N
            -  ----------------  ----------
            1  Inner scenario 1  Y         
      }
      GROUP 5: Input Values: Empty
      ============================
      GROUP 6: Expected Values {
      ==========================
            #  Scenario          Group           Row CSV         
            -  ----------------  --------------  ----------------
            1  Inner scenario 1  Output Group 1  Expected value 1
      }
      GROUP 7: Actual Values {
      ========================
            #  Scenario          Group           Row CSV       
            -  ----------------  --------------  --------------
            1  Inner scenario 1  Output Group 1  Actual value 1
      }
   OUTPUTS
   =======
      GROUP 1: Unit Test {
      ====================
            #  Title        Delimiter
            -  -----------  ---------
            1  Inner title  ;        
      } 0 failed of 1: SUCCESS
      ========================
      GROUP 2: Input Fields: Empty as expected: SUCCESS
      =================================================
      GROUP 3: Output Fields {
      ========================
            #  Group           Field         
            -  --------------  --------------
            1  Output Group 1  Output Field 1
      } 0 failed of 1: SUCCESS
      ========================
      GROUP 4: Input Values: Empty as expected: SUCCESS
      =================================================
      GROUP 5: Expected Values {
      ==========================
            #  Scenario          Group           Row CSV         
            -  ----------------  --------------  ----------------
            1  Inner scenario 1  Output Group 1  Expected value 1
      } 0 failed of 1: SUCCESS
      ========================
      GROUP 6: Actual Values {
      ========================
            #  Scenario          Group           Row CSV       
            -  ----------------  --------------  --------------
            1  Inner scenario 1  Output Group 1  Actual value 1
      } 0 failed of 1: SUCCESS
      ========================
} 0 failed of 6: SUCCESS
========================See Also↑ In This README...Database API Viewed As A Mathematical Function: Insights into TestingUnit Testing, Scenarios and Categories: The SCAN MethodTrapit JavaScript Tester/Formatter - GitHub moduleTrapit Python Tester - GitHub moduleTrapit Oracle Tester - GitHub modulePowershell Utilities - GitHub moduleTrapit Python Tester - Python Package Index moduleTimer Set Python Code Timer - GitHub moduleLicense↑ In This README...MIT