Patent ID: 12253938

The drawings have not necessarily been drawn to scale. Similarly, some components or operations may not be separated into different blocks or combined into a single block for the purposes of discussion of some of the embodiments of the present technology. Moreover, while the technology is amendable to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are described in detail below. The intention, however, is not to limit the technology to the particular embodiments described. On the contrary, the technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology as defined by the appended claims.

DETAILED DESCRIPTION

The following description and associated figures teach the best mode of the invention. For the purpose of teaching inventive principles, some conventional aspects of the best mode may be simplified or omitted. The following claims specify the scope of the invention. Note that some aspects of the best mode may not fall within the scope of the invention as specified by the claims. Thus, those skilled in the art will appreciate variations from the best mode that fall within the scope of the invention. Those skilled in the art will appreciate that the features described below can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific examples described below, but only by the claims and their equivalents.

Various embodiments will be described in detail with reference to the drawings wherein like reference numerals represent like parts and assemblies throughout the several views. Reference to various embodiments does not limit the scope of the claims attached hereto. Additionally, any examples set forth in this specification are not intended to be limiting and merely set forth some of the many possible embodiments for the appended claims. The various embodiments and examples described herein are provided by way of illustration only and should not be construed to limit the claims attached hereto. Those skilled in the art will readily recognize various modifications and changes that may be made without following the example embodiments and applications illustrated and described herein, and without departing from the true spirit and scope of the claims.

Generally, the present disclosure is directed to systems, methods and devices for testing custom application software. In an example, a method is provided, the method comprising: receiving an input that modifies a step associated with an application action; receiving a request to execute a test action corresponding to the application action; identifying one or more test steps for executing the test action; executing each of the one or more test steps in a test framework utilizing test metadata; determining whether an unexpected result occurred during the execution of the one or more steps; and causing, if an unexpected result is determined to have occurred, a graphical representation of the unexpected result to be displayed.

In another example, a system for testing custom application software is provided. The system may comprise: at least one processor; and a memory operatively connected to the at least one processor, the memory comprising computer-executable instructions that, when executed by the at least one processor, are operative to: receive an input that adds at least one step associated with an application action; receive a request to execute a test action corresponding to the application action; identify one or more test steps for executing the test action; execute each of the one or more test steps in a test framework utilizing test metadata; determine whether an unexpected result occurred during the execution of the one or more steps; and cause, if an unexpected result is determined to have occurred, a graphical representation of the unexpected result to be displayed.

In still another example, a computer-readable storage device is provided. The computer-readable storage device may comprise executable instructions that, when executed by one or more processors, assist with testing custom application software, the computer-readable storage device including instructions executable by the one or more processors for: receiving an input that modifies an application action; receiving a request to execute a test action corresponding to the application action; identifying one or more test steps for executing the test action; executing each of the one or more test steps in a test framework utilizing test metadata; determining whether an unexpected result occurred during the execution of the one or more steps; and causing, if an unexpected result is determined to have occurred, a graphical representation of the unexpected result to be displayed.

The systems, methods, and devices described herein provide technical advantages for efficiently testing software that is customized by end users. Allowing end users rather than developers to customize their own enterprise resource planning (ERP) software applications provides a high degree of flexibility on the consumer side, but is typically too cost-intensive from a developer standpoint due to the necessity of having the developer modify and test any desired modifications to the software. Utilizing the mechanisms described herein, users can customize their ERP software applications by modifying chunks of base, tenant, and lifecycle metadata, and test those modifications using a no-code testing environment that allows users to quickly see whether the modifications they would like to make are workable, or if they produce unexpected errors, and different modifications should be made. Users of the no-code testing environment do not have to have any coding experience and can easily test modifications to existing ERP applications via the no-code test framework. The ability of the consuming users to modify and test their own software in a no-code environment reduces processing costs (i.e., computer processing unit (CPU) cycles) and storage costs that would otherwise be required for those consuming users to write their own code to modify the applications rather than having the software distributors test that code and produce individualized updates for that code at the distributor level for every modifying consumer.

FIG.1is an exemplary distributed computing architecture100for testing custom application software utilizing a no-code consumer platform. Computing architecture100comprises computing device102A, which is displaying a test application user interface112for no-code testing of custom ERP applications, metadata storage sub-environment120, network and processing sub-environment114, and computing device102B, which is displaying error user interface128for highlighting errors detected with relation to corresponding custom ERP application modifications. Computing device102A and computing device102B may be the same computing device or different computing devices, and any of the computing devices depicted inFIG.1may communicate with one another and with other computing devices (e.g., one or more computing devices shown inFIG.5) via network116.

The test application user interface112displayed on computing device102A comprises no-code elements for modifying an existing ERP application framework, and testing elements of the ERP application framework that correspond to the modification. In some examples, the test application and its corresponding test application user interface112may be utilized to add, remove, and/or modify actions and various elements in an ERP application. For example, a user may utilize the test application and its corresponding test application user interface112to add journal entries, modify display objects, edit table value, modify calculations, and add, modify or delete other ERP application elements. In other examples, the test application and its corresponding test application user interface112may be utilized to add, remove, and/or modify individual steps that makeup actions in an ERP application (e.g., create item, set values for all the data items, and save action, for an add journal entry example). In some examples, when a user selects a new step to add to an existing action or set of steps, the user may select where to input the new step in the series of steps that makeup a corresponding action. While the test application user interface112illustrates one example of no-code user interface elements that may be interacted with to modify an ERP application framework for testing, other user interface configurations are contemplated according to this disclosure.

Metadata storage sub-environment120includes base metadata storage122, lifecycle metadata storage124, and tenant metadata storage126. One or more metadata chunks in tenant metadata storage126may comprise information that may be utilized to provide a customized user interface and/or user experience element associated with an ERP application to a user associated with a particular tenancy credential. In some examples, one or more metadata chunks in tenant metadata storage126may comprise information that may be utilized to provide a customized workflow and/or customized workflow elements for one or more ERP applications. Likewise, one or more metadata chunks in lifecycle metadata storage124may comprise information that may be utilized to provide a customized user interface and/or user experience element associated with an ERP application to a user associated with a lifecycle credential. In some examples, one or more metadata chunks in lifecycle metadata storage124may comprise information that may be utilized to provide a customized workflow and/or customized workflow elements for one or more ERP applications. According to examples, base metadata storage122may include metadata chunks for providing non-customized user interface and/or user experience elements for an ERP application. That is, while lifecycle metadata storage124and tenant metadata storage126include metadata chunks for one or more customized elements of an ERP application, base metadata storage122includes metadata chunks which can be utilized to generate the entirety of an ERP application, in a non-customized manner. Furthermore, metadata chunks from base metadata storage122may be combined with metadata chunks from one or both of tenant metadata storage126and/or lifecycle metadata storage124to provide a semi-customized application experience.

In some examples, users may modify one or more base metadata elements when customized application elements are created via tenant metadata and/or lifecycle metadata in relation to the creation and/or modification of customized ERP applications. One or more metadata chunks may be utilized by an ERP test application to generate test frameworks for testing modifications to existing ERP applications. According to some aspects, a change handling engine associated with network and processing sub-environment114may limit one or more elements associated with base metadata that can be modified by users. For example, the change handling engine may disallow users from making customized changes to one or more applications that would cause one or more application elements to function improperly and/or hinder administrative updating of base metadata and its subsequent use in combination with tenant metadata and/or lifecycle metadata.

According to some examples, each metadata chunk in each of tenant metadata storage126, lifecycle metadata storage124, and base metadata storage122may include a description of each other metadata chunk that it affects and/or each metadata chunk that affects it. In other examples, a separate index comprising that information may be associated with metadata storage sub-environment120and/or a custom ERP application service associated with network and processing sub-environment114. Thus, if a modification is made to one chunk of metadata, the effects of that modification on other metadata chunks may be determined from analyzing the index and/or the descriptions included in the corresponding metadata stores.

A user may wish to test modifications to an ERP application that the user has made prior to fully implementing those modifications in a full version of the ERP application. For example, even if a user may know the expected effects of modifying one or more metadata chunks corresponding to a modification, addition, and/or deletion of an action or action step because that information is associated in an index and/or with the metadata itself, unexpected results may nonetheless occur when certain ERP application modifications are implemented. When a user would like to test a modification to an ERP application prior to rolling it out to the full version of the application, the user may utilize the test application described herein.

In the specific example shown regardingFIG.1, a user has opened the no-code test application, selected the “add tax” user interface element108, and confirmed that selection via fly-out window110. According to some examples, the user may have interacted with a specific column in the ERP application that the user would like to add the tax too, and the “add tax” user interface element108may have then been presented to the user. Once the user affirmatively selects to add the tax in the test application, the test application, which may reside on computing device102A/102B and/or all or in part on server computing device118(e.g., in a cloud-based ERP application service), the test application may identify the metadata chunks in metadata storage sub-environment120corresponding to the add tax modification, generate a test version of the ERP application for implementing the newly added tax from corresponding metadata chunks in metadata storage sub-environment120, and determine, based on expected effects of the new tax identified from an index or the metadata itself, whether adding the tax to the ERP application produces expected or unexpected results. In this example, the values in the sales column of the ERP application were affected in a manner that was not expected, and the test application may thus indicate via error user interface128, the fields/application elements where the unexpected results occurred. In this example, sales column130has been highlighted by the test application as including unexpected results, and newly added tax column132has also been highlighted by the test application, thereby indicating the reason for the unexpected results in sales column130.

In this example, object values (i.e., the sales values) in the ERP application are the application elements to which unexpected results have occurred due to the application modification, and they are therefore highlighted in error user interface128. Other ERP application elements (e.g., display objects, table elements, user experience elements) may be affected in other manners by modifications to certain metadata chunks (e.g., via modifying, adding, or deleting steps or actions in an ERP application). If one of those application elements is unexpectedly impacted by a modification to an ERP application, the error user interface128may similarly highlight a corresponding error when the test flow corresponding to those elements is executed. In examples where there are no unexpected application elements that are impacted by a modification to one or more chunks of metadata, the test application may indicate that result to a user after the corresponding test metadata has been tested by the test application.

FIG.2illustrates information contained in metadata and/or an index associated with a test application service, which describes the effects that each ERP application element has on each other ERP application element through three layers of metadata. The metadata chunks shown in relation to each of: base metadata storage122in base layer206, lifecycle metadata storage124in lifecycle layer204, and tenant metadata storage126in tenant layer202, as well as the ERP application elements and their expected effects, are provided for exemplary purposes only. Other application steps, actions, objects, and their corresponding effects may be included with corresponding metadata and/or an index associated with an ERP application service.

When a user utilizes a no-code ERP test application to add an application element, modify an application element, and/or delete an application element, the test application may identify each metadata chunk corresponding to the modifications, and replicate the necessary metadata chunks for running a test scenario through the modified application elements. In running the test scenario through the modified application elements, the test application can analyze, via metadata chunks and/or an index, each step and/or action associated with the modification and identify all of the expected results related to the modification. If an unexpected result is identified by the test application after running the test, the test application can provide an indication to the testing user of what the unexpected result was, as well as an indication of what caused it.

In this specific example, a user may have modified, added, or deleted “action 1”208, corresponding to one or more metadata chunks in tenant layer202. In some examples, each of the metadata chunks corresponding to “action 1”208in tenant metadata storage126may include information210about each other chunk of metadata that the “action 1”208metadata chunks affect. In other examples, a separate index comprised of effects that each metadata chunk has in relation to each other metadata chunk may be associated with an ERP application service.

In this example, in tenant layer202, information210indicates that “action 1”208affects “user interface element 2”212and “user interface element 3”228in the ERP application. Lifecycle layer204includes metadata chunks that are also affected in some manner by the modification to “action 1”208. Specifically, information214associated with metadata chunks corresponding to “user interface element 2”212indicates that “user interface element 2”212affects metadata chunks corresponding to “user interface element 4”220; and information218associated with metadata chunks corresponding to “object value 1”216indicates that “object value 1”216affects metadata chunks corresponding to “object value 2” (not shown). Base layer206includes metadata chunks that are also affected in some manner by the modification to “action 1”208. Specifically, although information222associated with metadata chunks corresponding to “user interface element 4”220indicates that “user interface element 4”220does not affect any other object values, user interface elements, and/or user experience elements in the ERP application; and information226associated with metadata chunks corresponding to “object value 2”224indicates that “object value 2”224similarly does not affect any other object values, user interface elements, and/or user experience elements in the ERP application; information230associated with metadata chunks corresponding to “user interface element 3”228indicates that “user interface element 3”228affects metadata chunks corresponding to “object value 1”216in lifecycle layer204.

When the ERP test application executes one or more steps associated with modifications to the existing ERP application (e.g., a modification to “action 1”208) it can determine what the actual results were via its execution in test metadata, compared with the results it expects based on information210,214,218,222,226, and230.

FIG.3illustrates one exemplary mechanism300described in test metadata by which modifying a single ERP application action can affect various other ERP application elements. Although the effects of adding an entire application action are shown for illustrative purposes, steps within application actions may similarly be mapped to effects in metadata and/or an index that may be utilized in identifying expected and unexpected results of modifying those application steps via an ERP test application.

In this example, a user has utilized an ERP test application to add “tax action”302to an existing ERP application. The test application may reference an index or information associated with “tax action”302and its corresponding metadata chunks to determine whether unexpected results occur when the tax action is added to the ERP application. In this example, the index and/or information associated with “tax action”302metadata chunks, indicates that adding “tax action”302to the ERP application is expected to affect “user interface element 2”304(i.e., table 1 is expected to be expanded), and that the subsequent modification of “user interface element 2”304is expected to affect “user interface element 4” (i.e., table 1 is expected to have a tax fluctuation user interface element added to it). Adding “tax action 302” to the ERP application is also expected to affect “user interface element 3”306(i.e., table 2 is expected to be expanded), the modification of “user interface element 3”306is expected to affect “object value 1”310(i.e., sales values are expected to be modified), and the modification of “object value 1”310is expected to affect “object value 2”312(i.e., gross revenue value is expected to be modified).

FIG.4illustrates an exemplary method400for testing custom application software utilizing a no-code consumer platform. The method begins at a start operation and flow continues to operation402.

At operation402a no-code input that modifies a step associated with an application action is received. The no-code input may be received via an ERP test application user interface. In some examples, a user may select the step from a menu of pre-populated steps that can be modified in an ERP application. Examples of steps that may be selected include: create item, set values, and save, although other selectable steps may be provided depending on a specific action and/or test flow that a user is modifying. According to some examples, upon selecting the step that is to be modified, a user may provide an indication of where the step should be positioned within an action and/or a test flow that includes that step.

From operation402flow continues to operation404where a request to execute a test action corresponding to the application action is received. The request may be received via an ERP test application user interface. The test application may identify metadata chunks from one or more of a base metadata layer, a lifecycle metadata layer, and/or a tenant metadata layer, corresponding to each step in the action that is to be tested. In some examples, the test application may also identify expected results associated with modifying the step that was selected for modification.

From operation404flow continues to operation406where each of one or more steps for executing the test action in a test framework utilizing test metadata are executed. The test application may build a test action framework from existing metadata for the application, augmented by the new step modification, for executing each step in the test action framework. The test application may then execute each step for the step action.

From operation406flow continues to operation408where a determination is made as to whether an unexpected result occurred during the execution of the one or more steps. According to some examples, the test application may analyze an index that includes expected results associated with each step in the test action that is being tested. In other examples, the test application may analyze the metadata corresponding to each metadata chunk that is needed to execute the test action, and the metadata itself may contain an indication of what application elements are expected to be affected by the step modification, and in what manner those application elements are expected to be affected.

From operation408flow continues to operation410where, if an unexpected result is determined to have occurred during the execution of the one or more steps, a graphical representation of the unexpected result is caused to be displayed. In some examples, one or more of the executed steps may be highlighted to indicate to a testing user which steps resulted in unexpected results. In additional examples, the test application may generate a user interface corresponding to one or more application elements where the unexpected results occurred and highlight the area/portion of the user interface where there is an unexpected result.

From operation410flow continues to an end operation, and the method400ends.

FIG.5is a simplified diagram of a distributed computing system in which aspects of the present invention may be practiced. According to examples, any of computing devices502A (a modem),502B (a laptop computer),502C (a tablet),502D (a personal computer),502E (a smart phone), and502F (a server) may contain modules, components, engines, etc. for testing custom application software. Additionally, according to aspects discussed herein, any of computing devices502A-F may contain necessary hardware for implementing aspects of the disclosure for testing custom application software as described herein. Any and all of the operations of the methods described herein for testing custom application software may be performed, by way of example, at network servers506and/or server502F when computing devices502A-F request or receive data from external data provider518by way of network520. In examples, network servers506and/or server502F may comprise one or more of a test application engine and/or one or more application modules and/or components for receiving an input that modifies an application action; receiving a request to execute a test action corresponding to the application action; identifying one or more steps for executing the test action; executing each of the one or more test steps in a test framework utilizing test metadata; determining whether an unexpected result occurred during the execution of the one or more steps; and causing, if an unexpected result is determined to have occurred, a graphical representation of the unexpected result to be displayed.

Turning toFIG.6, one embodiment of the architecture of a system for testing custom application software is provided. Content and/or data interacted with, requested, or edited in association with one or computing devices may be stored in different communication channels or other storage types. For example, data may be stored using a directory service, a web portal, a mailbox service, an instant messaging store, or a compiled networking service for managing, interacting with, and processing metadata layers corresponding to modified ERP application steps/actions. The system for testing custom application software and executing the operations associated with the methods described herein may use any of these types of systems or the like for enabling data utilization, as described herein. A computing device618A,618B, and/or618C may provide a request to a cloud/network, which is then processed by a server620in communication with an external data provider617. As one example, the external data provider617may provide a data stream in response to a processed request to test modifications to ERP applications over the web to the computing device618A,618B, and/or618C through a network615. By way of example, a client computing device may be implemented as any of the systems described herein, and embodied in a personal computing device, a tablet computing device, and/or a mobile computing device (e.g., a smart phone, a tablet, etc.), as shown with computing devices618A,618B, and618C. Any of these aspects of the systems described herein may obtain content from the external data provider617.

In various embodiments, the types of networks used for communication between the computing devices that makeup the present invention include, but are not limited to, an Internet, an intranet, wide area networks (WAN), local area networks (LAN), virtual private networks (VPN), global positioning system (GPS) devices, cellular networks, and additional satellite based data providers such as the Iridium satellite constellation which provides voice and data coverage to satellite phones, pagers and integrated transceivers, etc. According to aspects of the present disclosure, the networks may include an enterprise network and a network through which a client computing device may access an enterprise network. According to additional aspects, a client network is a separate network accessing an enterprise network through externally available entry points, such as a gateway, a remote access protocol, or a public or private Internet address.

Additionally, the logical operations may be implemented as algorithms in software, firmware, analog/digital circuitry, and/or any combination thereof, without deviating from the scope of the present disclosure. The software, firmware, or similar sequence of computer instructions may be encoded and stored upon a computer readable storage medium. The software, firmware, or similar sequence of computer instructions may also be encoded within a carrier-wave signal for transmission between computing devices.

Operating environment600typically includes at least some form of computer readable media. Computer readable media can be any available media that can be accessed by a processor such as processing device780depicted inFIG.7and processor802shown inFIG.8or other devices comprising the operating environment. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, RAM, read-only memory (ROM), electrically erasable programmable read-only memory, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium which can be used to store the desired information. Computer storage media does not include communication media.

Communication media embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio-frequency (RF), infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

The operating environment600may be a single computer operating in a networked environment using logical connections to one or more remote computers. The one or more remote computers may comprise a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above as well as others not so mentioned. The logical connections may include any method supported by available communications media. Such networking environments are commonplace in enterprise-wide computer networks, intranets and the Internet.

FIG.7illustrates one aspect in which an exemplary architecture of a computing device710that can be used to implement one or more aspects of the present disclosure, including any of the plurality of computing devices described herein with reference to the various figures and their corresponding descriptions. The computing device710illustrated inFIG.7may be used to execute the operating system, application programs, and software modules (including the software engines) described herein. For example, with respect toFIG.8any of the program modules and engines814and corresponding operations, including auto-test creation engine816, test metadata analysis engine818, object effects engine820, value effects engine822, base metadata analysis engine824, tenant metadata analysis engine826, and/or lifecycle metadata analysis engine828may be executed utilizing the computing device710illustrated inFIG.7. To avoid undue repetition, this description of the computing device will not be separately repeated herein for each of the other computing devices described with regard to the other figures, including computing devices102A,102B, and server computing device118(depicted inFIG.1), computing device302(depicted inFIG.3), computing devices502A-F and network servers506(depicted inFIG.5), computing devices618A-618C and server620(depicted inFIG.6), and computing device800(depicted inFIG.8), but such devices can also be configured as illustrated and described with reference toFIG.7.

The computing device710includes, in some embodiments, at least one processing device, such as a central processing unit (CPU). A variety of processing devices are available from a variety of manufacturers, for example, Intel, Advanced Micro Devices, and/or ARM microprocessors. In this example, the computing device710also includes a system memory782, and a system bus784that couples various system components including the system memory782to the processing device780. The system bus784is one of any number of types of bus structures including a memory bus, or memory controller; a peripheral bus; and a local bus using any of a variety of bus architectures.

Examples of computing devices suitable for the computing device710include a server computer, a desktop computer, a laptop computer, a tablet computer, a mobile computing device (such as a smart phone, a tablet, a mobile digital device, or other mobile devices), or other devices configured to process digital instructions.

The system memory782includes read only memory786and random access memory788. A basic input/output system790containing the basic routines that act to transfer information within computing device710, such as during start up, is typically stored in the read only memory786.

The computing device710also includes a secondary storage device792in some embodiments, such as a hard disk drive, for storing digital data. The secondary storage device792is connected to the system bus784by a secondary storage interface794. The secondary storage devices792and their associated computer readable media provide nonvolatile storage of computer readable instructions (including application programs and program modules), data structures, and other data for the computing device710. Details regarding the secondary storage devices792and their associated computer readable media, as well as their associated nonvolatile storage of computer readable instructions (including application programs and program modules) will be more fully described below with reference toFIG.8.

Although the exemplary environment described herein employs a hard disk drive as a secondary storage device, other types of computer readable storage media are used in other aspects according to the disclosure. Examples of these other types of computer readable storage media include magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, compact disc read only memories, digital versatile disk read only memories, random access memories, or read only memories. Additional aspects may include non-transitory media. Additionally, such computer readable storage media can include local storage or cloud-based storage.

A number of program modules and/or program engines can be stored in secondary storage device792or system memory782, including an operating system796, one or more application programs798, other program modules700(such as the software engines described herein), and program data702. The computing device710can utilize any suitable operating system, such as Linux, Microsoft Windows™, Google Chrome™, Apple OS, and any other operating system suitable for a computing device.

According to examples, a user provides inputs to the computing device710through one or more input devices704. Examples of input devices704include a keyboard706, mouse708, microphone709, and touch sensor712(such as a touchpad or touch sensitive display). Additional examples may include other input devices704. The input devices are often connected to the processing device780through an input/output interface714that is coupled to the system bus784. These input devices704can be connected by any number of input/output interfaces, such as a parallel port, serial port, game port, or a universal serial bus. Wireless communication between input devices and input/output interface714is possible as well, and includes infrared, BLUETOOTH® wireless technology, cellular and other radio frequency communication systems in some possible aspects.

In an exemplary aspect, a display device716, such as a monitor, liquid crystal display device, projector, or touch sensitive display device, is also connected to the system bus784via an interface, such as a video adapter718. In addition to the display device716, the computing device710can include various other peripheral devices (not shown), such as speakers or a printer.

When used in a local area networking environment or a wide area networking environment (such as the Internet), the computing device710is typically connected to a network such as network520shown inFIG.5and network615shown inFIG.6through a network interface720, such as an Ethernet interface. Other possible embodiments use other communication devices. For example, certain aspects of the computing device710may include a modem for communicating across a network.

The computing device710typically includes at least some form of computer readable media. Computer readable media includes any available media that can be accessed by the computing device710. By way of example, computer readable media include computer readable storage media and computer readable communication media.

Computer readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any device configured to store information such as computer readable instructions, data structures, program modules or other data. Computer readable storage media includes, but is not limited to, random access memory, read only memory, electrically erasable programmable read only memory, flash memory or other memory technology, compact disc read only memory, digital versatile disks or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device710. Computer readable storage media does not include computer readable communication media.

Computer readable communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, computer readable communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

The computing device illustrated inFIG.7is also an example of programmable electronics, which may include one or more such computing devices, and when multiple computing devices are included, such computing devices may be coupled together with a suitable data communication network so as to collectively perform the various functions, methods, or operations disclosed herein.

FIG.8is a block diagram illustrating additional physical components (e.g., hardware) of a computing device800with which certain aspects of the disclosure may be practiced. The computing device components described below may have computer executable instructions for receiving an input that modifies a step associated with an application action; receiving a request to execute a test action corresponding to the application action; identifying one or more test steps for executing the test action; executing each of the one or more test steps in a test framework utilizing test metadata; determining whether an unexpected result occurred during the execution of the one or more steps; and causing a graphical representation of the unexpected result to be displayed if an unexpected result is determined to have occurred.

Computing device800may perform the above-described functions alone or in combination with a distributed computing network such as those described with regard toFIGS.5and6which may be in operative contact with computing device618A,618B, and/or618C, wherein computing devices618A,618B, and/or618C may represent personal a computing device, a tablet computing device, and/or a mobile computing device which may communicate and process one or more of the program modules and/or program engines described inFIG.8including: auto-test creation engine816, test metadata analysis engine818, object effects engine820, value effects engine822, base metadata analysis engine824, tenant metadata analysis engine826, and/or lifecycle metadata analysis engine828. According to additional examples, computing device800may be in communicative contact via the distributed computing networks described inFIGS.5and6and computing device800may comprise and describe any of computing devices502A,502B,502C,502D,502E and502F.

In a basic configuration, the computing device800may include at least one processor and system memory. Depending on the configuration and type of computing device, the system memory810may comprise, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory810may include an operating system812and one or more program modules and engines814suitable for testing custom application software, such as one or more components in regards toFIG.8and, in particular, auto-test creation engine816, test metadata analysis engine818, object effects engine820, value effects engine822, base metadata analysis engine824, tenant metadata analysis engine826, and/or lifecycle metadata analysis engine828. The operating system812, for example, may be suitable for controlling the operation of the computing device800. Furthermore, aspects of the disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system.

The computing device800may have additional features or functionality. For example, the computing device800may also include additional data storage device (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG.8by storage804. It will be well understood by those of skill in the art that storage may also occur via the distributed computing networks described inFIG.5andFIG.6. For example, computing device800may communicate via network520inFIG.5and data may be stored within network servers506and transmitted back to computing device800via network520if it is determined that such stored data is necessary to execute one or more functions described herein. Additionally, computing device800may communicate via network615inFIG.6and data may be stored within server620and transmitted back to computing device800via network615if it is determined that such stored data is necessary to execute one or more functions described herein.

As stated above, a number of program modules and data files may be stored in the system memory810. While executing the processor802, the program modules and engines814(e.g., test metadata analysis engine818) may perform processes including, but not limited to, the aspects described herein. Other program modules and engines may be used in accordance with aspects of the present disclosure.

Aspects of the present disclosure, for example, are described above with reference to block diagrams and/or operational illustrations of methods, systems, and computer program products according to aspects of the disclosure. The functions/acts noted in the blocks may occur out of the order as shown in any flowchart. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure as claimed in any way. The aspects, examples, and details provided in this application are considered sufficient to convey possession and enable others to make and use the best mode of the claimed disclosure. The claimed disclosure should not be construed as being limited to any aspect, example, or detail provided in this application. Regardless of whether shown and described in combination or separately, the various features (both structural and methodological) are intended to be selectively included or omitted to produce an embodiment with a particular set of features. Having been provided with the description and illustration of the present disclosure, one skilled in the art may envision variations, modifications, and alternate aspects falling within the spirit of the broader aspects of the general inventive concept embodied in this application that do not depart from the broader scope of the claimed disclosure.

The various embodiments described above are provided by way of illustration only and should not be construed to limit the claims attached hereto. Those skilled in the art will readily recognize various modifications and changes that may be made without following the example embodiments and applications illustrated and described herein, and without departing form the true spirit and scope of the following claims.

Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

The phrases “in some embodiments,” “according to some embodiments,” “in the embodiments shown,” “in other embodiments,” and the like generally mean the particular feature, structure, or characteristic following the phrase is included in at least one implementation of the present technology, and may be included in more than one implementation. In addition, such phrases do not necessarily refer to the same embodiments or different embodiments.

To reduce the number of claims, certain aspects of the technology are presented below in certain claim forms, but the applicant contemplates the various aspects of the technology in any number of claim forms. For example, while only one aspect of the technology is recited as a computer-readable medium claim, other aspects may likewise be embodied as a computer-readable medium claim, or in other forms, such as being embodied in a means-plus-function claim. Any claims intended to be treated under 35 U.S.C. § 112(f) will begin with the words “means for,” but use of the term “for” in any other context is not intended to invoke treatment under 35 U.S.C. § 112(f). Accordingly, the applicant reserves the right to pursue additional claims after filing this application to pursue such additional claim forms, in either this application or in a continuing application.