Automated testing of HVAC devices

Architectures or techniques are presented that can facilitate automated function testing (AFT) in connection with an HVAC system or component thereof. The architectures detailed herein can facilitate creation, modification, or duplication of quiz data (e.g., a test). This quiz data can be executed in order to automatically test the function of the HVAC system or component. Additionally, prior to execution, a verification procedure can be performed to ensure that an expected state indicated in the quiz data can in fact be exhibited by the device. Further, execution of the quiz data can include an exit condition that, when satisfied can cause termination of the execution prior to completion. Such can be useful to avoid potentially dangerous situations or to avoid undue disruption to a service provided by the HVAC system.

TECHNICAL FIELD

The present disclosure is directed to systems, apparatuses, and methods for testing HVAC devices within an HVAC system, and more particularly to automated testing of the functionality and/or states of the HVAC devices.

BACKGROUND

Modern heating, ventilation, and air conditioning (HVAC) systems and associated building automation systems (BAS) or control elements can benefit from periodic testing to ensure the systems are functioning properly. As one example, a functional test can ensure that one or more components are operating properly, which can include testing that these components activate or enter the proper state in response to designated stimuli. Traditionally, sequences of operations are tested for accuracy by either manual procedures or testing programs that are substantially hardcoded. Generally, these other testing techniques include manually overriding individual points to “trick” sensors or other equipment. Some systems use partially automated functions such as variable air volume (VAV) auto-commissioning in order to force the system into a different state for which data is collected and interpreted by the tester.

SUMMARY

According to an embodiment of the present disclosure, a testing device can comprise a processor and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations. The computer executable instructions can comprise receiving quiz data that defines a test of a device of a heating, ventilation, and air conditioning (HVAC) system. The quiz data can comprise a command that applies a point to the device. The quiz data can further comprise a predicate that is satisfied if the device exhibits an expected state in response to application of the point.

According to an embodiment of this disclosure, the testing device can perform a verification procedure. The verification procedure can be satisfied in response to determining that the device is capable of exhibiting the expected state. In response to the verification procedure being satisfied, the testing device can perform an execution procedure. The execution procedure can comprise various elements. For example, the execution procedure can comprise initiating execution of the quiz data with respect to the device. The execution procedure can further comprise determining an exit condition. The exit condition can be such that, when satisfied, will cause the execution of the quiz data to terminate prior to completion.

In some embodiments, elements described in connection with the systems above can be embodied in different forms such as a computer-implemented method, a computer-readable medium, or another form.

DETAILED DESCRIPTION

Overview

As used herein a functional test can be a deliberate series of commands/operations on a piece of equipment or system meant to exercise and verify or prove proper operation. When discrete portions (e.g., quizzes) of a test are completed, a report is made to provide feedback (e.g., pass/fail) based on the results of the testing operations. Enhancements in that regard or described herein with respect to testing device that can provide an improved automated function test (AFT).

As outlined above, other systems for functional testing of heating, ventilation, and air conditioning (HVAC) systems tend to rely on manual procedures, which can be time-consuming and error-prone to implement, or hardcoded into testing software, which can be inflexible and difficult to update or improve. In other systems, processes for proper functional testing can be time-consuming as most of a given process is manual and is prone to error or misinterpretation of the written sequence of operation. Misinterpretation of the results and data can be costly when a formal commissioning agent requires changes be made to the programming prior to retesting.

The disclosed subject matter relates to automating the processes used in functional testing of HVAC systems. The disclosed techniques can further collect and collate information from the testing and interpret the results with precision. Furthermore, if the testing procedures are verified by a formal commissioning agent, there will be little or no questions of different interpretations of the specified sequence of operation.

The disclosed testing device can provide a way to test the functionality and sequences of operation of a piece of equipment with or without the full context of other pieces of equipment within the HVAC system. The tests and tools can externally command and change individual points for the purposes of determining a pass/fail result based on some predetermined logical condition. The tests can be editable to account for alterations in the field and/or changes to the original sequence of operation.

In some embodiments, the disclosed systems can interface with a variety of different networks and can be network protocol-agnostic. A given HVAC network protocol (e.g., BACnet, etc) can be interfaced via protocol plugins that can provide useful extensibility to the disclosed system. Significantly, in some embodiments, the disclosed techniques can be more easily tailored to real-world situations. For example, automated testing can be performed in view of safety or operational checks, which are often not extant in a laboratory setting. Hence, exit conditions can be integrated into the automated testing that can, e.g., exit the AFT if a dangerous condition or too much disruption to HVAC services occurs.

In some embodiments, customers or other users can create or choose from a library tests based on their own particular equipment selection. These users are provided the ability to define expected outcomes or device states, test steps, and criteria for the result (e.g., pass/fail) of a test or portion thereof. In addition, the disclosed systems can, in some embodiments, detect that certain predicate conditions (e.g., expected states) can in fact be satisfied by the HVAC device being tested.

Results of a given test can be objective (e.g., Boolean pass/fail) and system granularity can be selected such that a given test can operate on a single HVAC device or component, a specified section of the HVAC system, or even a physical area (e.g., a specified floor of a building), or the like. Significantly, a single test or a collection of individual tests can verify the sequential operation of a building as a whole and/or the entire HVAC system. Many different components can be tested according to individual constraints or in operation in the aggregate. Such can be done in a sequential manner that results in testing of individual predicates to determine whether any one of them fail during testing.

Example Systems

Referring now to the drawings, with initial reference toFIG.1, a block diagram of an example non-limiting system100is depicted that can provide automated function testing in accordance with one or more embodiments of the disclosed subject matter. In some embodiments, system100can comprise testing device102that can employed to make various determinations or perform various procedures detailed herein. Testing device102can comprise a processor104and a memory106that stores executable instructions that, when executed by the processor, facilitate performance of operations. Additional examples of said processor104and memory106, as well as other suitable computer or computing-based elements, can be found with reference toFIG.9, and can be used in connection with implementing one or more of the devices or components shown and described in connection withFIG.1or other figures disclosed herein. It should be understood that in the discussion of the present embodiment and of embodiments to follow, repetitive description of like elements employed in the various embodiments described herein is omitted for sake of brevity.

System100can further comprise a heating, ventilation, and air conditioning (HVAC) device108. HVAC system108can comprise an HVAC device110. HVAC device110can be a physical piece of equipment of HVAC system108or in some embodiments can be a software or logic circuit or component that controls a portion of HVAC system108. HVAC system108can further comprise HVAC network112, which can operate according to any suitable networking protocol. By way of illustration purposes and not limitation, HVAC network112can operate according to one or more of a building automation control network (BACnet) protocol, a LonTalk protocol, a KNX protocol, a Modbus protocol, a ZigBee protocol, a Z-Wave protocol, an open source protocol for building automation, and a standardized protocol that is standardized by the American society of heating, refrigeration, and air conditioning engineers (ASHRAE).

Testing device102can receive quiz data114. Quiz data114can define a test of an HVAC system. Quiz data114can be similar to what is referred to herein as an automated function test (AFT). Quiz data114can comprise one or more command(s)116and one or more predicate(s)120. Command116can apply a point118to HVAC device110. In some embodiments, point118can represent a set point for HVAC device110. In some embodiments, point118can represent an analog value, a binary value, or a multistate value. Predicate120can represent, for example, a Boolean logic based comparator of two expressions. An expression can be a low level object that contains a mathematical expression. The expression can contain one or more points118, numbers or other expressions, which can potentially be recursive. For example, predicate120can be satisfied if HVAC device110exhibits expected state122in response to application of point118. As an example, consider a fan (e.g., HVAC device110) that is expected to activate at a particular setting (e.g., expected state122) in response to some stimuli (e.g., emulated by command116that applies point118).

In addition, testing device102can perform verification procedure124. Verification procedure124can be satisfied (e.g., verification is satisfied) in response to determining that HVAC device110is capable of exhibiting expected state122. Hence, testing device102can potentially pinpoint errors in quiz data114generated by users or others even prior to executing quiz data114. For instance, if HVAC device110cannot exhibit expected state122, it is already known that particular sequence will fail.

In response to verification procedure124being satisfied, testing device102can perform execution procedure126. Execution procedure126can include initiating execution of quiz data114(e.g., executing the AFT) with respect to HVAC device110. It is understood that execution procedure126can include recursive elements as well as multiple quizzes114that can be sequentially executed. Multiple quizzes114are referred to herein as an exam. In other words, exam data can comprise multiple, distinct instances of quiz data114, each potentially referring to difference HVAC devices110, with different commands116and predicates120.

Execution procedure126can further include determine exit condition128. Exit condition128can represent a condition that, when satisfied, will cause the execution of quiz data114(e.g., execution procedure126) to terminate prior to completion. Additional detail with reference to exit condition128, execution procedure126, and additional aspects or elements is further discussed in connection withFIG.2.

Turning now toFIG.2, a block diagram of system200is presented depicting non-limiting examples of system200. System200illustrates additional aspect or elements in connection with automated function testing in accordance with one or more embodiments of the disclosed subject matter. For example, execution procedure126, can include logging procedure202. Logging procedure202can record pretest point204of HVAC device110. Pretest point204can represent a setting or point exhibited by HVAC device110prior to execution procedure126or a relevant portion thereof.

In some embodiments, in addition to pretest point204, logging procedure202can further record one or more state(s)206of HVAC device110that are exhibited during execution procedure126or other times. During execution procedure126, state206can be compared to expected state122to determine whether predicate120is or is not satisfied. In some embodiments, a determination of whether expected state122is exhibited by HVAC device110and/or predicate120is satisfied can comprise determining that predicate120is satisfied within a defined time period208or that predicate120is not satisfied if expected state122is not exhibited within defined time period208. In other words, if expected state122is not exhibited within defined time period208, the associated predicate120fails.

It is understood that logging procedure202can save settings, state data, or other information such that, following termination of execution procedure126, HVAC device110can be reverted to its pretest point(s)204and/or previous state(s)206. Furthermore, the information recorded during logging procedure202can be used to generate report210. Report210can represent raw data in any format or a human-readable illustration of point(s)118that were applied to HVAC device110during execution procedure126, the resultant state(s)206exhibited by HVAC device110in response, whether the resultant state206exhibited corresponds to expected state122, and, potentially, timing information such as an amount of time after point118was applied that HVAC device110exhibited state expected state122.

In some embodiments, testing device102can perform cleanup procedure212. In operation, cleanup procedure212can revert HVAC device110to a pretest state such as, for example, by applying pretest point204to HVAC device110. Cleanup procedure212can be a portion of execution procedure126(e.g., a final portion), or initiate upon execution procedure126terminating depending on implementation. For example, in some embodiments, cleanup procedure212can initiate in response to exit condition128being satisfied (e.g., execution procedure126terminated prior to completion). In other embodiments, cleanup procedure212can initiate in response to execution procedure126completing. It is understood that certain operations can be recursive or involve multiple sub-quizzes or routines. Accordingly, depending on testing goals, HVAC device110can be reverted to a previous operational state upon exit or termination of a given sub-portion of the AFT or only after the entire AFT has exited or terminated.

Hence, in some embodiments, during performance of execution procedure126, testing device can monitor exit condition128. As discussed, if exit condition128is satisfied, execution procedure126can terminate prior to completion. In some embodiments, exit condition128can be satisfied in response to a determination that execution procedure126can violate safety protocol220. For example, consider again the case in which HVAC device110is a fan device having (in response to application of point118) an expected state122of activating. However, further suppose this fan is undergoing maintenance, or is proximal to a different device that is undergoing maintenance, or otherwise maintenance personnel might be in the vicinity. In these or other cases, it can be understood that executing a function test on the fan at this time can violate safety protocol220.

In some embodiments, exit condition128can be satisfied in response to a determination that execution procedure126can disrupt service222. For example, consider the case in which HVAC device110is a heating device. During function testing, a temperature set point of a particular space increases beyond a comfort barrier or other defined threshold. In these or other cases, it can be understood that executing a function test on the heating device at this time can disrupt service222of HVAC system108expected by occupants. It is understood that safety protocol220and disrupt service22are merely two non-limiting examples of potential exit condition128and other examples can be used in addition or alternatively.

As discussed previously, testing device102can be configured to operation in conjunction with many different types of HVAC network112. In some embodiments, testing device102can abstract a protocol-agnostic interface to HVAC network112that can interact with any suitable type of network, e.g., based on protocol plugins or the like. Prior systems tend to be network-specific without the capability to interface to many different types of HVAC network112. Such a capability is not trivial because it is very common that elements of device control are typically integrated into various network protocols. Because testing device102operates to change points (e.g., pretest point204) to other values or otherwise take control of HVAC device110to some degree, such often must be done within the context of the particular HVAC network112.

For example, BACnet as one example maintains priorities in connection with points. Thus, if signals are received to change a point of HVAC device110, the signal with the higher priority will take precedence. Further, in some cases with BACnet or other similar protocols, in order to input a point value to a given HVAC device, sometimes it might be necessary to set an out-of-service flag to true. Other protocols on the other hand might operate differently, with a key take away that the type of protocol employed by HVAC network112can affect not only communication but also the operation of how an AFT might work to generate suitable results.

Consider an example in which HVAC network112operates according to a BACnet protocol. One goal of a given AFT might be to test HVAC device110by applying point118, which overrides pretest point204, as detailed above. However, pretest point204will likely be associated with a priority value. Hence, in order to override pretest point204, point118should have a priority that is higher. However, the priority of point118should have an upper limit because it is not desirable to override pretest point204when such could jeopardize safety or cause damage.

In some embodiments, testing device can determine native protocol214. Native protocol214can be representative of a protocol by which HVAC network112operates, such as, for example BACnet or Lontalk. As noted, testing device102can be network-agnostic and can interface to HVAC network112according to native protocol214. In some embodiments, testing device102can determine a priority216of point118based at least in part on native protocol214. For example, based on native protocol216(e.g., BACnet) priority216can be selected to be higher than ordinary priorities, but lower than safety priorities. Thus, in ordinary cases, priority216will be higher than a priority of pretest point204allowing one to be overwritten with the other, but denying the overwrite in the event that priority216is not higher than that for pretest point204(e.g., in cases where pretest point has a priority that is ensures a safety protocol or the like).

In some embodiments, testing device102can further comprise generating marriage certificate218. Marriage certificate218can be, e.g., a translation object that connects devices an points118of quiz data114(e.g., a data model) to actual HVAC device(s)110and pretest points204on a live network. Marriage certificate218can handle ‘on-the-wire’ translations between various systems of measurement, for instance, as execution procedure126is performed.

With reference now toFIG.3, a block diagram300is presented depicting non-limiting examples of components of the testing device in accordance with one or more embodiments of the disclosed subject matter. In some embodiments, testing device102can comprise crafting engine302. Crafting engine302can perform various sets of offline operations and online operations. As example offline operations, crafting engine302can facilitate creation and editing of quiz data114or suitable portions of quiz data114. Such can be augmented with access to test library storage such as AFT506that is further discussed in connection withFIG.5. As example online operations, crafting engine302can facilitate device discovery on link such as detecting or discovering HVAC device110when connected to HVAC network112. Crafting engine might also provide resolution or application of various tests to devices and provide graphical support during execution of tests.

In some embodiments, testing device102can comprise execution engine304. Execution engine304can perform the execution of commands116and is typically invoked during execution procedure126. Execution engine304can further collect data such as elements detailed in connection with logging procedure202. Further, in some instances at conclusion of execution procedure126, execution engine304can interpret results and associate those results with corresponding HVAC device(s)110. In some embodiments, execution engine304can further perform all or a portion of cleanup procedure212or other similar tasks.

In some embodiments, testing device102can comprise reporting engine306. Reporting engine306can, for instance, receive data from execution engine306and generate a human-readable document that describes the AFT and associated format and indicates results of the AFT.

It is understood that components of testing device102can have numerous advantageous characteristics consistent with concepts described herein. For example, execution engine304can be portable to many different platforms such as for example a Symbio 800 platform, a Tracer SC+ platform or other suitable platforms employed for building automation or HVAC systems. Execution engine304can be designed with some network protocol independence. Hence, execution engine304can be interfaced to different network protocols. Further, execution engine304can run multiple AFTs (e.g., quizzes) or portions thereof in parallel, which can be asynchronous in execution. Execution engine304can also be configured to provide certain live status feedback that can be useful as execution procedure126is performed.

Furthermore, reporting engine306can also be portable to other platforms such as a TIS platform as well as a Symbio 800 platform, a Tracer SC+ platform or other suitable platforms. In some embodiments, individual components of testing device102can be serialized to be used within the context of a suitable database.

Referring now toFIG.4, a block diagram400is presented depicting non-limiting examples of an automated function test in execution in accordance with one or more embodiments of the disclosed subject matter. Points118can be representative of a fundamental element to a given AFT. As discussed, commands116execute on points118and resultant data (e.g., states206) can be collected. Data collection402can occur on points118. Values from points118that are assigned to a device during the test can be collected and prepared for inclusion in a report (e.g., report210). This collected data can further be used by execution engine304or a separate evaluation engine to determine results.

Commands116can be considered a core component of an AFT representative of actions that are executed on points118. Results404can relate to commands116and can represent elements such as, for example, an override of a device, an override release, an in service or an out of service indicator or flag, setting values, and so forth. In some embodiments, delay406can be determined or recorded.

Next to be further described is quiz(s)408. A quiz408can represent a collection of commands116and data collection402members, and can be representative of quiz data114. Each quiz408can constitute a result (e.g., a pass/fail result). In some embodiments, a quiz408can have one and only one result. A quiz408can have one expected outcome (e.g., expected state122). It is appreciated that quizzes408can be modular in design and can be catalogued in a data store (e.g., AFT library506ofFIG.5) for later access or recall. A quiz408can be used to test a single instance of functionality as described in an associated sequence of operation.

Expected state(s)122can reflect an outcome of a quiz408, which can be true or false, pass or fail, or the like, typically a binary value that can represent evaluation of some piece of logic. In some embodiments, this binary value can be a result of comparing points118to other points or constants. Predicate(s)410can represent a prerequisite such as an enable/disable condition that will allow or disallow quiz408to execute. Predicates410can be a single point118or another piece of logic.

Test412can represent a collection of quiz408. A complete test412(e.g., defined by quiz data114) can evaluate all or a portion of a sequence of operation for a system (e.g., HVAC system108) or a piece of equipment (HVAC device110). In some embodiments, multiple tests412can be aggregated into an exam.

It is understood that because many tests might be running concurrently, it can be advantageous to provide a centralized ‘tick’ timer. For example, a network ‘tick’ can be set to 12 seconds, for example, or another suitable duration. This tick timer can be utilized to synchronize network traffic where BACnet (or another network protocol) functions like “read property multiple” (or other protocol equivalent functions) can be utilized. In addition, any protocol interface can be designed to store a snapshot of the state of each pretest point204that will be changed by a command116. Hence, the pretest point204can be returned to its exact original state upon exit of any particular quiz or, if desired, potentially upon exit of a given portion of a quiz.

Turning now toFIG.5, a block diagram of system500is presented depicting non-limiting examples of potential use cases in accordance with one or more embodiments of the disclosed subject matter. User device502can interface to testing device102, for example in order to craft an AFT indicated at reference numeral504. The user can generate the AFT from scratch or download suitable tests or quizzes from AFT library506. If desired, the user can modify tests or quizzes from AFT library to tailor to a particular implementation or event.

At reference numeral508, the AFT can be applied to an HVAC device and a marriage certificate can be generated. At reference numeral510, the AFT can be initiated, and at reference numeral514, the AFT can be executed. In some embodiments, the AFT can alternatively be initiated by HVAC system controller512. As show at reference numeral516points can be executed and, at reference numeral518, data collection can occur. Subsequently, as illustrated at reference numeral520, cleanup on commands can commence. At reference numeral522, results can be analyzed and at reference numeral524, a report can be generated.

As one example use case consider the case in which a user or technician chooses a series of functional tests to be performed on one or more pieces of equipment. An auto-commissioning function can utilize this information to initiate various functional tests. The auto-commissioning function can display functional testing results and generate standardized reports or user-defined reports. Auto-commissioning function can return equipment to an original state including, e.g., releasing overrides, putting points back in service, and so forth.

As another example use case, consider the case in which CSET (computer system engineering technology) output or standards define equipment, control devices, and functional tests. Auto-commissioning function can consume CSET data and initiate function tests. The auto-commissioning function can display functional testing results and generate standardized reports or user-defined reports. Auto-commissioning function can return equipment to an original state including, e.g., releasing overrides, putting points back in service, and so forth.

In another example use case, consider the case in which a user has a piece of equipment that does not currently have any function test defined. This user can create one or more function test(s). The user can define actions to be take, define expected result(s), define pass/fail criteria for each expected result, and also define what data is to be included in a report. The auto-commissioning function can provide alerts of missing elements or possible hazards to equipment function tests. The user can save function tests for later use or to share with other users.

In yet another example use case, consider the case in which a user has a piece of equipment that is similar to equipment for which an existing AFT was developed. The auto-commissioning function can provide a method of duplicating and editing existing function tests. Auto-commissioning function can further provide alerts of missing elements or possible dangers to persons or equipment.

In still another example use case, consider the case in which a certified commissioning agent provides function tests to be performed on specified equipment. The auto-commissioning function can have a standard format for new criteria to be automatically consumed and incorporated into a library of function tests.

Referring now toFIGS.6A-6C, various block diagrams600A-600C of example architectural implementations are illustrated in accordance with one or more embodiments of the disclosed subject matter.

For example, block diagram600A depicts an example architectural design in which testing device102is situated in a remote system such as a cloud system602. Testing device102can be representative of a device that executes an AFT and/or quiz data114as illustrated in connection withFIG.1. In other words, in some embodiments, testing device102can be remote from HVAC system108.

Block diagram600B depicts an example architectural design in which AFT library506is in a remote system such as a cloud system602. In this embodiment, testing device102can be situated at a user site and/or local to the HVAC system108and communicate with the cloud to make various determinations. In other embodiments, both testing device102and AFT library506can be situated in cloud system602and communicate with HVAC system108.

Block diagram600C depicts an example architectural design in which one or both user testing device102and AFT library506are components of HVAC system108, which can be situated at the user site.

Example Methods

FIGS.7and8illustrate various methodologies in accordance with the disclosed subject matter. While, for purposes of simplicity of explanation, the methodologies are shown and described as a series of acts, it is to be understood and appreciated that the disclosed subject matter is not limited by the order of acts, as some acts can occur in different orders and/or concurrently with other acts from that shown and described herein. For example, those skilled in the art will understand and appreciate that a methodology could alternatively be represented as a series of interrelated states or events, such as in a state diagram. Moreover, not all illustrated acts can be required to implement a methodology in accordance with the disclosed subject matter. Additionally, it should be further appreciated that the methodologies disclosed hereinafter and throughout this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methodologies to computers.

FIG.7illustrates a flow diagram700of an example, non-limiting computer-implemented method that can perform operations directed to automated function testing of an HVAC device or component in accordance with one or more embodiments of the disclosed subject matter. For example, at reference numeral702, a device (e.g., testing device102) comprising a processor can receive quiz data that defines a test of a device of a heating, ventilation, and air conditioning (HVAC) system, wherein the quiz data comprises a command that applies a point to the device and a predicate that is satisfied if the device exhibits an expected state in response to application of the point.

At reference numeral704, the device can perform a verification procedure that is satisfied in response to determining that the device is capable of exhibiting the expected state. In other words, the verification procedure can verify certain elements prior to initiating the testing. As an example, the verification procedure can indicate a malformed test element and/or that the test cannot be passed (e.g., the expected outcome cannot occur).

At reference numeral706, provided that the verification procedure is passed and/or satisfied, the device can perform an execution procedure. This execution procedure can comprise various elements including elements detailed at reference numerals708and710as well as other suitable elements as detailed herein, some of which are further discussed in connection withFIG.8.

At reference numeral708, the device can initiate execution of the quiz data with respect to the device. At reference numeral710, the device can determine an exit condition that, when satisfied, will cause the execution of the quiz data to terminate prior to completion. Non-limiting examples of the exit condition can relate to determinations that executing the quiz data can cause an unsafe condition, unduly disrupt a service (e.g., heating, cooling, etc.) being provided by the HVAC system, and so forth. Method700can proceed to insert A, which is further detailed in connection withFIG.8, or terminate.

Turning now toFIG.8, illustrated is a flow diagram800of an example, non-limiting computer-implemented method that can provide additional aspects or elements in connection with automated function testing of an HVAC device or component in accordance with one or more embodiments of the disclosed subject matter.

At reference numeral802, the device can perform a timing procedure that determines whether the expected state is achieved within a defined time period. In some embodiments, if the expected state is not achieved within the defined timer period, the associated command can be determined to fail. It is further appreciated that timing procedure can include the concepts of a centralized ‘tick’ introduced above, which can be used to synchronize certain network traffic or the like.

At reference numeral804, the device can perform a logging procedure. The logging procedure can record a pretest point of the device that is exhibited prior to the point being applied to the device. In some embodiments, the logging procedure can further record resultant states exhibited by the device during the execution procedure. For example, states that are exhibited in response to application of the points of the quiz.

At reference numeral806, the device can perform a cleanup procedure that, upon termination of the execution procedure, reverts the device to the pretest point.

Example Operating Environments

With reference again toFIG.9, the example environment900for implementing various embodiments of the aspects described herein includes a computer902, the computer902including a processing unit904, a system memory906and a system bus908. The system bus908couples system components including, but not limited to, the system memory906to the processing unit904. The processing unit904can be any of various commercially available processors. Dual microprocessors and other multi-processor architectures can also be employed as the processing unit904.

The system bus908can be any of several types of bus structure that can further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory906includes ROM910and RAM912. A basic input/output system (BIOS) can be stored in a non-volatile memory such as ROM, erasable programmable read only memory (EPROM), EEPROM, which BIOS contains the basic routines that help to transfer information between elements within the computer902, such as during startup. The RAM912can also include a high-speed RAM such as static RAM for caching data.

The computer902further includes an internal hard disk drive (HDD)914(e.g., EIDE, SATA), one or more external storage devices916(e.g., a magnetic floppy disk drive (FDD)916, a memory stick or flash drive reader, a memory card reader, etc.) and an optical disk drive920(e.g., which can read or write from a CD-ROM disc, a DVD, a BD, etc.). While the internal HDD914is illustrated as located within the computer902, the internal HDD914can also be configured for external use in a suitable chassis (not shown). Additionally, while not shown in environment900, a solid state drive (SSD) could be used in addition to, or in place of, an HDD914. The HDD914, external storage device(s)916and optical disk drive920can be connected to the system bus908by an HDD interface924, an external storage interface926and an optical drive interface928, respectively. The interface924for external drive implementations can include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 994 interface technologies. Other external drive connection technologies are within contemplation of the embodiments described herein.

A number of program modules can be stored in the drives and RAM912, including an operating system930, one or more application programs932, other program modules934and program data936. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM912. The systems and methods described herein can be implemented utilizing various commercially available operating systems or combinations of operating systems.

Computer902can optionally comprise emulation technologies. For example, a hypervisor (not shown) or other intermediary can emulate a hardware environment for operating system930, and the emulated hardware can optionally be different from the hardware illustrated inFIG.9. In such an embodiment, operating system930can comprise one virtual machine (VM) of multiple VMs hosted at computer902. Furthermore, operating system930can provide runtime environments, such as the Java runtime environment or the .NET framework, for applications932. Runtime environments are consistent execution environments that allow applications932to run on any operating system that includes the runtime environment. Similarly, operating system930can support containers, and applications932can be in the form of containers, which are lightweight, standalone, executable packages of software that include, e.g., code, runtime, system tools, system libraries and settings for an application.

A monitor946or other type of display device can be also connected to the system bus908via an interface, such as a video adapter948. In addition to the monitor946, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

The computer902can operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer(s)950. The remote computer(s)950can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer902, although, for purposes of brevity, only a memory/storage device952is illustrated. The logical connections depicted include wired/wireless connectivity to a local area network (LAN)954and/or larger networks, e.g., a wide area network (WAN)956. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to a global communications network, e.g., the Internet.

When used in a LAN networking environment, the computer902can be connected to the local network954through a wired and/or wireless communication network interface or adapter958. The adapter958can facilitate wired or wireless communication to the LAN954, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter958in a wireless mode.

When used in a WAN networking environment, the computer902can include a modem960or can be connected to a communications server on the WAN956via other means for establishing communications over the WAN956, such as by way of the Internet. The modem960, which can be internal or external and a wired or wireless device, can be connected to the system bus908via the input device interface944. In a networked environment, program modules depicted relative to the computer902or portions thereof, can be stored in the remote memory/storage device952. It will be appreciated that the network connections shown are example and other means of establishing a communications link between the computers can be used.

When used in either a LAN or WAN networking environment, the computer902can access cloud storage systems or other network-based storage systems in addition to, or in place of, external storage devices916as described above. Generally, a connection between the computer902and a cloud storage system can be established over a LAN954or WAN956e.g., by the adapter958or modem960, respectively. Upon connecting the computer902to an associated cloud storage system, the external storage interface926can, with the aid of the adapter958and/or modem960, manage storage provided by the cloud storage system as it would other types of external storage. For instance, the external storage interface926can be configured to provide access to cloud storage sources as if those sources were physically connected to the computer902.