Package: test-measure-process-lib
Test, Measure, Process library (TMPL)TMPL is a library for writing lab test measurement code in a modular and reusable way.Lab tests are broken up into lightweight classes that represent setup conditions and measurements. These conditions and measurements can be combined together to make test sequences. TMPL provides infrastructure to execute the conditions and measurements in a specific order and log the data into a convenient format.Although written for use in lab measurements TMPL makes no assumptions about test equipment setup and is very generic. It could be used to make test sequences based on models or data analysis instead of actual measurements.TMPL is built around storing data in thexarrayDatasetclass. This provides a convenient data structure for storing multi-dimensional data that can be easily visualised using libraries likeHoloviewsor its offshoothvplot.Dependencies and installationTMPL depends on these libraries.xarraypandasxlsxwriterTMPL can be installed viapippipinstalltest-measure-process-libNote that this does not install the dependencies. This is in case another package manager, e.g. Anaconda, is being used. So they have to be manually installed.DocumentationThis file gives basic descriptions of how to use TMPL, for more details consultthe full documentationCore classesTMPL is built on a set of core classes. These are built by inheriting from the Abstract classes defined intmpl_core.py. The classes are:TestManagerclasses: Based onAbstractTestManagerclass. These classes run sequences of measurements over multiple setup conditions e.g. temperature, humidity etc. TheTestManagersare ultimately responsible for gathering up all data recorded during the sequence and packaging it into one xarray Dataset object.Measurementclasses: Based onAbstractMeasurementclass. These are the classes that actually perform the measurements. They are classes that can be run independently or in a sequence from aTestManager. The data that they collect and process is stored in an internal xarray Dataset object. When run from aTestManagerthis dataset will be scooped up at the end of the sequence and added into the overall Dataset maintained by theTestManager.SetupConditionclasses: Based onAbstractSetupCondition. These classes are responsible for setting up experimental conditions e.g. temperature, pressure, supply voltage etc. They are small classes that have only one purpose, to set a specific condition.TestManagersuseSetupConditionclasses to set conditions during a sequence before running measurements.Example usageBefore explaining the inner workings of TMPL this section runs through a hypothetical example to show how the classes are used at the top level.Suppose aTestManagerhas been defined that has objects for settingtemperature and pressure. It also has three measurement objects defined calledcalibrate_scales, MeasureVolume and MeasureMass.TheTestManageris initialised with anyresourcesthat the measurement classes require, e.g. instruments.# Define test equipment objects for measurements to use# - can be anything that Measurement classes requireresources={'chamber':chamber_object,'test_sample':test_sample_object,'instrument':instr_object}# Create test manager objecttest=TestManager_mymeas(resources)Setup conditions under which test is to be run can be defined by accessing theSetupConditionsobjects directly through theconditionsproperty.test.conditions.temperature.values=[25,40,75]test.conditions.pressure.values=[12,15,65]The measurements to run during the sequence can be enabled/disabled by direct access to theMeasurementobjects through themeasproperty.test.meas.calibrate_scales.enable=Falsetest.meas.MeasureVolume.enable=Truetest.meas.MeasureMass.enable=TrueNow the sequence has been configured we can run the test over all setup conditionstest.run()Once the test sequence is finished we can get the results as anxarray Dataset.test.ds_resultsCan also get individual results from aMeasurementobject directly.test.meas.MeasureVolume.ds_resultsIndividual measurements can be run with specific conditions independent of theTestManager.conditions=dict(temperature_degC=34,pressure_nm=15)test.meas.MeasureVolume.run(conditions)Or measurements can be run without specifiying conditionstest.meas.MeasureVolume.run()Note in the last two cases where the measurement is run individually, specifiying the conditions merely includes the conditions as coordinates in the results Dataset.Measurementclassesdo notset their own conditions, that is only done bySetupConditionsclasses.Creating a measurement sequenceA measurement sequence consists of aTestManagerclass to run the overall sequence, any number ofSetupConditionsclasses and any number ofMeasurementclasses.Let's take a simple example, the measurement of resistance of a resistor.+--------------+
|   voltage    |
|   source     +-----+
|              |     |
+--------------+     |
                     |
                     |
                   +-+-+
                   |   |
                   |   |
                   | R |    Resistor to measure
                   |   |
                   |   |
                   |   |
                   +-+-+
                     |
                     |
                +----+-----+
                |          |
                | Ammeter  |
                |          |
                +----+-----+
                     |
                     |
                 --------- Ground
                   -----
                    ---We have two pieces of test equipment in this measurement: the voltage source and the ammeter. The measurement is simply to set a voltage, measure the current, and calculate the resistance from Ohm's law :Voltage = Resistance x CurrentIn this measurement we have one setup condition, voltage, one measurement, current and one processing step, resistance.Let's assume that the voltage source and ammeter are controlled through the objectsvoltage_sourceandammeter. These two objects will be supplied asresourcesto theTestManager,SetupConditionsandMeasurementclasses e.g.resources={'voltage_source':voltage_source,'ammeter':ammeter}All classes will automatically have thevoltage_sourceandammeterobjects available as properties.Setup conditionsFirst we'll setup the voltage source. This is our only setup condition and it will be done using aSetupConditionsclass.SetupConditionsclasses inherit from the abstract classAbstractSetupConditions. They require one method and two properties to be defined:initialise: Perform any initialisation, usually setting defaults for the propertyvalues.setpoint: Property that is used to set/get the condition set point valueactual: Property that returns the actual value of the condition, e.g. the actual voltage rather than the setpointThe complete class definition is shown here:classVoltage(tmpl.AbstractSetupConditions):definitialise(self):"""Initialise default values and any other setup"""# Set default valuesself.values=[3.0]@propertydefactual(self):"""Return actual measured voltage"""returnself.voltage_source.actual_voltage_V@propertydefsetpoint(self):"""Get/Set the output voltage of the voltage source"""returnself.voltage_source.voltage_set_V@setpoint.setterdefsetpoint(self,value):self.log(f'Set Voltage source to{value}V')# printoutself.voltage_source.voltage_set_V=valueMeasurementsNext the central measurement class is defined. Measurement classes inherit from theAbstractMeasurementclass. The only method thatneedsto be defined ismeas_sequence(). This is generally the top level function of a specific measurement procedure. Any number of extra methods can be added to the class to supportmeas_sequence(), but when a measurement is executed it basically calls themeas_sequence()method.In this case the measurement is simply to read the ammeter and store the reading, which can be done in themeas_sequence(). The resistance, however, is derived from the ammeter reading and the setpoint of the voltage source. Since this is "processing" rather than measurement it is good practice to do this in another method. This ensures that the real measurement, the ammeter reading, is done even if the processing step crashes. In this case the processing function could be re-run later to debug it, without re-running the measurement.classCurrentMeasure(tmpl.AbstractMeasurement):defmeas_sequence(self):"""Mandatory method for Measurement classesPerforms the actual measurement and stores data."""#  Measure current with ammetercurrent=self.ammeter.current_A# Store the dataself.store_data_var('current_A',current)@tmpl.with_results(data_vars=['current_A'])defprocess(self):"""Calculate resistance using measured current and voltage sourcesetting."""# Get voltage from current conditionsVoltage=self.current_conditions['Voltage']# Get current measured at the last conditionscurrent_A=self.current_results.current_Aresistance_ohms=Voltage/current_Aself.store_data_var('resistance_ohms',[resistance_ohms])Theprocess()method is called automatically after themeas_setup()method if it is present.Theprocess()method uses thetmpl.with_resultsdecoratorto ensure that there is always an entry stored calledcurrent_A. Ifprocess_results()were to be executed beforemeas_sequence()then an error would be thrown becausecurrent_Ahad not been created. Thetmpl.with_resultsdecorator is not mandatory it is a convenience that avoids having to add boilerplate code such as :assert'current_A'inself.ds_results# use decorator instead: @tmpl.with_results(data_vars=['current_A'])Note also thattmpl.with_results()can have a list of names passed to it if more than one value has been measured and stored.Test managerNow that the setup conditions and measurement have been defined, all that remains is to assemble the top level test sequence class. Again this is inherited from an abstract class:AbstractTestManagerclassSimpleResistanceMeasurement(tmpl.AbstractTestManager):defdefine_setup_conditions(self):"""Add the setup conditions here in the order that they should be set"""# Add setup conditions using class nameself.add_setup_condition(Voltage)defdefine_measurements(self):"""Add measurements here in the order of execution"""# Setup links to all the measurements using class nameself.add_measurement(CurrentMeasure)The test manager requires two methods to be defined:define_setup_conditions(): This method is a list of calls toself.add_setup_condition(<class name>). This method takes the name of the SetupConditions class defined previously. In this case there is only one setup condition, Voltage, but if there are multiple SetupConditions classes they are all added herein the order that they should be set.define_measurements(): Similarly measurement classes are added using their class names using theself.add_measurement(<class name>). Again the order here dictates the order in which the measurements will be executed.Running the testWith the classes defined the test can be run by supplying the required resources to the test manager class:importtmpl# Make the instrument objectsR=tmpl.examples.ResistorModel(10e3)vs=tmpl.examples.VoltageSupply(R)am=tmpl.examples.Ammeter(R)# Make resourcesresources={'voltage_source':vs,'ammeter':am}# Create test managertest=SimpleResistanceMeasurement(resources)# Run the testtest.run()the output should look like this:@ SimpleResistanceMeasurement | <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
@ SimpleResistanceMeasurement | Running SimpleResistanceMeasurement
@ SimpleResistanceMeasurement | Generating the sequence running order
@ SimpleResistanceMeasurement | 	Running order done
------------------------------------------------------------
@ Voltage                   | Set Voltage source to 3.0V
@ CurrentMeasure            | <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
@ CurrentMeasure            | Running CurrentMeasure
@ CurrentMeasure            | CurrentMeasure	Time taken: 0.003s 
@ CurrentMeasure            | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
@ SimpleResistanceMeasurement | ========================================
@ SimpleResistanceMeasurement | SimpleResistanceMeasurement	Time taken: 0.006s 
@ SimpleResistanceMeasurement | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>Running orderInternally TMPL generates a list of functions to call when therun()method is called. It can be useful to view this running order before actually running the test sequence. The propertydf_running_orderdisplays this in apandasDataFrame for a convenient tabular printout.Here's the running order of the resistance measurement example:>>>test.df_running_orderOperationLabelVoltage0CONDITIONVoltage3.01MEASUREMENTCurrentMeasure3.0It shows that the test sequence consists of two steps, the first step is aCONDITIONoperation, i.e. setting the voltage. The second step is aMEASUREMENT, i.e. reading the Ammeter.Results dataThe whole point of the TMPL library is to get experimental data intoxarrayDataset format. Once a test sequence has been run, all the data collected will be available from the test manager object in the propertyds_results.ds_resultsis anxarray Datasetobject. Here's the result of the simple resistance measurement:>>>test.ds_results# Display results from test sequence<xarray.Dataset>Dimensions:(Voltage:1)Coordinates:*Voltage(Voltage)float643.0Datavariables:current_A(Voltage)float640.0002963resistance_ohms(Voltage)float641.013e+04The data can be stored and re-loaded in JSON format# Save to JSONtest.save('my_data.json')# Load from JSONtest.load('my_data.json')This stores theds_resultsDataset into JSON format, which can be loaded back in later. Loading previously measured data can be useful for testing new processing functions.Individual Measurement dataTheds_resultsproperty of a test manager class, e.g.test, scoops up all the data measured in individual measurement class object and puts it into one Dataset. However the individual measurement data can be accessed in the same way. All TMPL class objects have ads_resultsproperty and all can be saved and loaded in the same way.So for the resistor measurement example we can access the data from the measurement class,CurrentMeasurelike this:>>>test.meas.CurrentMeasure.ds_results<xarray.Dataset>Dimensions:(Voltage:1)Coordinates:*Voltage(Voltage)float643.0Datavariables:current_A(Voltage)float640.0002963resistance_ohms(Voltage)float641.013e+04It looks exactly the same astest.ds_results, becauseCurrentMeasureis the only measurement class in this test sequence. It can also be saved and loaded in the same manner.# Save to JSONtest.meas.CurrentMeasure.save('my_data.json')# Load from JSONtest.meas.CurrentMeasure.load('my_data.json')Dataset extra featuresTMPL adds some extra features to Datasets for easy storing of the data. It registers adataset_accessor, which adds thesaveproperty to the Dataset. Thesaveproperty has several functions for saving the Dataset into different formats as shown here:# Save Dataset to JSONtest.ds_results.save.to_json(filename)# Save Dataset to JSON stringjstr=test.ds_results.save.to_json_str()# Save Dataset to Excel spreadsheettest.ds_results.save.to_excel(filename)More advanced exampleThe simple resistance measurement was good for demonstrating the basic operation of TMPL. Now we will look at a more advanced example. It is still based on measuring the resistance of a resistor but this time we will make the measurement more sophisticated in the following ways.Instead of using the setting of the voltage source for the voltage value, we will use a dedicated voltmeter across the resistor.Rather than calculating resistance from single values of voltage and resistance we will sweep the voltage and measure the current. We can then fit a line to these measurements and obtain resistance from the slope of the line.We also want to measure the resistance variation against environmental conditions so we will put it in a chamber that can vary the temperature and humidity.Here's a diagram of the new setup:+--------------+
|   voltage    |
|   source     +-----+
|              |     |
+--------------+     |
                     +----------------------+
                     |                      |
           +---------------+         +------+------+
           |         |     |         |             |
 Chamber   |       +-+-+   |         |             |
           |       |   |   |         |             |
    +------+       | R |   |         | Voltmeter   |
    | Temp |       |   |   |         |             |
    +------+       |   |   |         |             |
    | Hum  |       +-+-+   |         +-------+-----+
    +------+         |     |                 |
           +---------------+                 |
                     |                       |
                     +-----------------------+
                     |
                +----+-----+
                | Ammeter  |
                |          |
                +----+-----+
                     |
                     |
                 +-------+ Ground
                   +---+
                    +-+Now our representation in TMPL will be:Setup condition:TemperatureHumidityVoltage source setpointMeasurements:Current (from Ammeter)Voltage across resistor (from Voltmeter)First we'll need more resources for the new equipment: an environmental chamber for setting temperature and humidity, plus a voltmeter.resources={'chamber':chamber_object,'voltage_source':voltage_source_object,'ammeter':ammeter_object,'voltmeter':voltmeter_object}These will be given to the test manager class.Setup conditionsWe now need some new setup condition classes for temperature and humidity. These will make use of the chambertemperature_degCandhumidity_pcproperties like this:classTemperature(tmpl.AbstractSetupConditions):definitialise(self):"""Initialise default values and any other setup"""# Set default valuesself.values=[25,35,45]@propertydefactual(self):returnself.chamber.temperature_degC@propertydefsetpoint(self):returnself.chamber.temperature_setpoint_degC@setpoint.setterdefsetpoint(self,value):self.chamber.temperature_setpoint_degC=valueclassHumidity(tmpl.AbstractSetupConditions):definitialise(self):"""Initialise default values and any other setup"""# Set default valuesself.values=[55,85]@propertydefactual(self):returnself.chamber.humidity_degC@propertydefsetpoint(self):returnself.chamber.humidity_setpoint_degC@setpoint.setterdefsetpoint(self,value):self.chamber.humidity_setpoint_degC=valueThis time we are not going to use the voltage source as a setup condition because we want to sweep the voltage.MeasurementsThe main measurement in this example will be a sweep of the voltage source setpoint. During this sweep the current from the ammeter and the voltage from the voltmeter will be measured. This requires a new measurement class to be created.classVoltageSweeper(tmpl.AbstractMeasurement):definitialise(self):# Set up the voltage values to sweep overself.config.voltage_sweep=np.linspace(0,1,10)defmeas_sequence(self):#  Do the measurementcurrent=np.zeros(self.config.voltage_sweep.shape)voltage=np.zeros(self.config.voltage_sweep.shape)forindex,Vinenumerate(self.config.voltage_sweep):# Set voltageself.voltage_source.voltage_set_V=V# Measure currentcurrent[index]=self.ammeter.current_A# Measure voltage across resistorvoltage[index]=self.voltmeter.voltage_V# Store the dataself.store_coords('swp_voltage',self.config.voltage_sweep)self.store_data_var('current_A',current,coords=['swp_voltage'])self.store_data_var('voltage_diff_V',voltage,coords=['swp_voltage'])@tmpl.with_results(data_vars=['current_A','voltage_diff_V'])defprocess(self):volts=self.current_results.voltage_diff_V.valuesamps=self.current_results.current_A.values# Fit line to amps vs volts, get resistance from slopefit_coefficients=np.polyfit(amps,volts,1)resistance_ohms=fit_coefficients[0]# slopeself.store_data_var('resistance_ohms',[resistance_ohms])This measurement is more detailed than the previous example. Measurement classes can have aninitialise()method where configuration parameters can be defined. In this case we are defining the voltage values that going to be swept over in the line:self.config.voltage_sweep=np.linspace(0,1,10)Every TMPL class has aconfigdictionary that can be used to store any kind of data. It is a special dictionary defined in TMPL called anObjDict, where elements can be added by using the dot notation to assign values. The standarddictway can also be used. We could equally have used:self.config['voltage_sweep']=np.linspace(0,1,10)Measurements are just normal classes so you can define your own properties as well. Using theconfigdict just organises the data under one roof.The mandatorymeas_setup()is now a loop that follows the sequence:Set voltageMeasure currentMeasure voltage across resistorWhere the two measurements are stored in 1 dimensional arrays.The last part of themeas_setup()method stores the data. The current and measured voltage are functions of thevoltage_sweepvalues. To capture this relationship we need to makevoltage_sweepinto a coordinate in addition to the setup conditions of temperature and humidity. This is done by explicitly storingvoltage_sweepas a coordinate and indicating instore_data_vars()that it is a coordinate.# Store the dataself.store_coords('swp_voltage',self.config.voltage_sweep)self.store_data_var('current_A',current,coords=['swp_voltage'])self.store_data_var('voltage_diff_V',voltage,coords=['swp_voltage'])In order to do this the data being stored must be the same shaped array as the coordinate. In this case everything is a 1D array.Theprocess()method operates just as before. It requires that the current and voltage have been measured. If they have then it will fit a line to the data and get the resistance from the slope of that line.Test managerNow we can assemble the setup condition and measurement into aTestManagerclass. This follows the same pattern as before:classAdvancedResistanceMeasurement(tmpl.AbstractTestManager):defdefine_setup_conditions(self):"""Add the setup conditions here in the order that they should be set"""# Add setup conditions using class nameself.add_setup_condition(Temperature)self.add_setup_condition(Humidity)defdefine_measurements(self):"""Add measurements here in the order of execution"""# Setup links to all the measurements using class nameself.add_measurement(VoltageSweeper)Set up theTestManagerobject:importtmpl# Setup resourcesR=tmpl.examples.ResistorModel(10e3)vs=tmpl.examples.VoltageSupply(R)am=tmpl.examples.Ammeter(R)vm=tmpl.examples.Voltmeter(R)chamber=tmpl.examples.EnvironmentalChamber(R)resources={'voltage_source':vs,'ammeter':am,'voltmeter':vm,'chamber':chamber}# Create test managertest=AdvancedResistanceMeasurement(resources)We can see the running order again:>>>test.df_running_order@AdvancedResistanceMeasurement|Generatingthesequencerunningorder@AdvancedResistanceMeasurement|RunningorderdoneOperationLabelTemperatureHumidity0CONDITIONTemperature25.0NaN1CONDITIONHumidityNaN55.02MEASUREMENTVoltageSweeper25.055.03CONDITIONHumidityNaN85.04MEASUREMENTVoltageSweeper25.085.05CONDITIONTemperature35.0NaN6CONDITIONHumidityNaN55.07MEASUREMENTVoltageSweeper35.055.08CONDITIONHumidityNaN85.09MEASUREMENTVoltageSweeper35.085.010CONDITIONTemperature45.0NaN11CONDITIONHumidityNaN55.012MEASUREMENTVoltageSweeper45.055.013CONDITIONHumidityNaN85.014MEASUREMENTVoltageSweeper45.085.0and run the test:>>>test.run()Run@AdvancedResistanceMeasurement|Generatingthesequencerunningorder@AdvancedResistanceMeasurement|Runningorderdone@AdvancedResistanceMeasurement|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<@AdvancedResistanceMeasurement|RunningAdvancedResistanceMeasurement@AdvancedResistanceMeasurement|Generatingthesequencerunningorder@AdvancedResistanceMeasurement|Runningorderdone------------------------------------------------------------------------------------------------------------------------@VoltageSweeper|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<@VoltageSweeper|RunningVoltageSweeper@VoltageSweeper|finishedsweep@VoltageSweeper|VoltageSweeperTimetaken:0.008s@VoltageSweeper|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>------------------------------------------------------------@VoltageSweeper|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<@VoltageSweeper|RunningVoltageSweeper@VoltageSweeper|finishedsweep@VoltageSweeper|VoltageSweeperTimetaken:0.008s@VoltageSweeper|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>------------------------------------------------------------------------------------------------------------------------@VoltageSweeper|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<@VoltageSweeper|RunningVoltageSweeper@VoltageSweeper|finishedsweep@VoltageSweeper|VoltageSweeperTimetaken:0.007s@VoltageSweeper|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>------------------------------------------------------------@VoltageSweeper|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<@VoltageSweeper|RunningVoltageSweeper@VoltageSweeper|finishedsweep@VoltageSweeper|VoltageSweeperTimetaken:0.008s@VoltageSweeper|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>------------------------------------------------------------------------------------------------------------------------@VoltageSweeper|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<@VoltageSweeper|RunningVoltageSweeper@VoltageSweeper|finishedsweep@VoltageSweeper|VoltageSweeperTimetaken:0.008s@VoltageSweeper|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>------------------------------------------------------------@VoltageSweeper|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<@VoltageSweeper|RunningVoltageSweeper@VoltageSweeper|finishedsweep@VoltageSweeper|VoltageSweeperTimetaken:0.007s@VoltageSweeper|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>@AdvancedResistanceMeasurement|========================================@AdvancedResistanceMeasurement|AdvancedResistanceMeasurementTimetaken:0.049s@AdvancedResistanceMeasurement|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>Installing for developmentTMPL can be installed locally by cloning the repository to a local folder and then installing with pipcd<local_path>
gitclonehttps://github.com/redlegjed/test_measure_process_lib.git
pipinstall-e<local_path>/test_measure_process_libThe code can then be edited from<local_path>/test_measure_process_lib. Changes will be included when importingtmplinto a new python instance.