Integrated testing system utilizing a test script and a test environment created based on the script

Systems, methods, apparatuses, and computer readable media for testing information technology systems and/or applications are provided. In some examples, data may be categorized as frequently used and stored at an information technology system testing system. One or more portions of the frequently used data may then be identified for use in testing an information technology system. The systems, methods, and the like may further include building a testing environment and receiving a test script. In some examples, one or more data types may be identified for use in testing the information technology system based on various project criteria, the received test script, and the like. In some examples, additional data types and data associated therewith may be identified as associated with the one or more identified data types based on a predefined relationship. This additional data may then be automatically included in the testing of the information technology system.

BACKGROUND

New information technology applications and systems are being created each day. However, in addition to developing the new applications and systems, the applications should be adequately tested prior to being implemented to ensure any defects are identified, remedied, and the like. Testing is often performed by the team developing the application and may also be performed independently of that team. However, conventional testing methods can be slow and inefficient, particularly when dealing with large amounts of data. Accordingly, an integrated, stream-lined approach to testing new information technology systems and applications would be advantageous.

SUMMARY

According to one or more aspects, systems, devices, computer readable media and methods for testing information technology systems and/or applications are provided. In some examples, data may be categorized as frequently used and stored at an information technology system testing system. One or more portions of the frequently used data may then be identified for use in testing an information technology system. The systems, methods, and the like may further include building a testing environment and receiving a test script. In some examples, one or more data types may be identified for use in testing the information technology system based on various project criteria, the received test script, and the like. In some examples, additional data types and data associated therewith may be identified as associated with the one or more identified data types based on a predefined relationship. This additional data may then be automatically included in the testing of the information technology system.

DETAILED DESCRIPTION

FIG. 1illustrates a block diagram of a generic computing device101(e.g., a computer server) in computing environment100that may be used according to an illustrative embodiment of the disclosure. The computing device101may have a processor103for controlling overall operation of the device and its associated components, including random access memory (RAM)105, read-only memory (ROM)107, input/output (I/O) module109, and memory115.

I/O109may include a microphone, mouse, keypad, touch screen, scanner, optical reader, and/or stylus (or other input device(s)) through which a user of server101may provide input, and may also include one or more of a speaker for providing audio output and a video display device for providing textual, audiovisual and/or graphical output. Software may be stored within memory115and/or other storage to provide instructions to processor103for enabling server101to perform various functions. For example, memory115may store software used by the server101, such as an operating system117, application programs119, and an associated database121. Alternatively, some or all of server101computer executable instructions may be embodied in hardware or firmware (not shown).

The computing device101may operate in a networked environment supporting connections to one or more remote computers, such as terminals141and151. The terminals141and151may be personal computers or servers that include many or all of the elements described above relative to the server101. The network connections depicted inFIG. 1include a local area network (LAN)125and a wide area network (WAN)129, but may also include other networks. When used in a LAN networking environment, the server101may be connected to the LAN125through a network interface or adapter123. When used in a WAN networking environment, the server101may include a modem127or other network interface for establishing communications over the WAN129, such as the Internet131. It will be appreciated that the network connections shown are illustrative and other means of establishing a communications link between the computers may be used. The existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP, HTTPS, and the like is presumed.

Computing device101and/or terminals141or151may also be mobile terminals (e.g., mobile phones, PDAs, notebooks, and the like) including various other components, such as a battery, speaker, and antennas (not shown).

The disclosure may be described in the general context of computer-executable instructions, such as program modules, being executed by one or more computers and/or one or more processors associated with the computers. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Aspects of the disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The above-described systems may be used in various businesses or corporate entities, such as financial institutions or other entities, to aid in testing one or more information technology systems. For example, the systems, methods, apparatuses, and computer-readable media described herein may include receiving test or project parameters and criteria and preparing to test the information technology system, building a test platform including receiving one or more test scripts and conducting the test. In some examples, the test(s) performed on the information technology system as described herein may be an additional test, performed independently of another test of the system. For instance, the creators of the information technology system may test the system. Once those tests are completed, the system may be tested again (by the same or a different entity), independently of the first test, using the systems, methods, apparatuses and computer-readable media described herein.

In some examples, the information technology testing system may provide efficiency improvements, such as identification of frequently used data, automatic identification of data that may be associated with the data being tested, and the like. These advantages will be discussed more fully below.

Further, although several examples used herein may include reference to a financial institution or projects associated with the financial institution, the systems, methods, and the like described herein may be used in a variety of industries (e.g., service and non-service industries), corporations, businesses, government agencies, universities, other types of organizations and the like. Nothing in the specification or figures should be viewed as limiting the invention to only use with banking or financial services related entities.

FIG. 2illustrates one information technology system testing system according to at least some examples described herein. The testing system200may be part of or associated with an entity202, such as the entity implementing the system. The entity202may be a business, corporation, university or other educational institution, government agency, and the like. In some examples, the entity may be a financial institution, such as a bank. For simplicity, the testing system200will be described in terms of a financial institution. However, nothing in the specification or figures should be viewed as limiting any of the features or aspects described herein to only banks or banking related issues. In some examples, the testing system200may be external to or separate from the entity202(e.g., provided by or associated with a 3rdparty or outside vendor).

In some examples, the testing system200may be accessed via a network, such as the Internet. Additionally or alternatively, the testing system200may be accessed by systems internal to the entity202, via a private network such as an intranet.

The testing system200may include a test preparation module204. The test preparation module204may receive information associated with the project or system being tested. For instance, the test preparation module204may receive project criteria such as requirements or expectations for performance of the system being tested, functional requirements of the system being tested, timing of the test, expected completion dates, and the like. Additional project criteria may be received without departing from this invention.

The testing system200may further include a frequently used data module206(e.g., a database). The frequently used data module206may store data that, based on historical records, is often used in testing. In some arrangements, the data may be stored in the frequently used data module206upon determining that the data has been previously used a predetermined number of times, or has been used a number of times greater than a pre-determined threshold. The data may then be stored in the frequently used data module206and may be available for use. For instance, this frequently used data may be stored at the testing system200for a current test being run as well as future tests. In some examples, this may permit storage of commonly used information (e.g., customer data associated with customers of a bank, deposit data, and the like), in a single location. Since this data and these data types are frequently used, this may reduce or eliminate the need to copy over or store this data for each newly created test. Rather, the data can be stored in one location and accessed as needed. This may conserve data storage capacity and reduce costs associated with the testing system. In some examples, the frequently used data module206may store between 2 and 4 terabytes of data and, in some arrangements, may be refreshed on a periodic basis (e.g., monthly, every six weeks, and the like) in order to update data.

In some examples, the test preparation module204may identify a portion of the frequently used data stored in the frequently used data module206to be accessed for testing a particular information technology system. Identifying the data to be accessed may be performed, in some examples, automatically based on the received project criteria, parameters, and the like. For instance, the project or system for testing may include one or more particular criteria. In some arrangements, those criteria may be associated with one or more types of data or data generally stored in the frequently used data module206. The data or data type may be identified based on this association (e.g., via a look-up table, and the like).

The testing system200may further include a test script module208. The test script module may receive one or more test scripts for use in testing the information technology system. In some examples, the test script may be built by a user accessing the system, such as via user computer216, or may be previously used script received by the test script module208.

Various components of the testing system200may be connected to one or more data stores205a,205b. The data store may be internal to or associated with the entity202(e.g., customer data of a financial institution stored within the financial institution) such as data store205a. In other examples, the data store may be external to or not associated with the entity202, such as data store205b. External data may, in some examples, include third party data, such as from a data warehouse, and the like. Although two data stores205a,205bare shown inFIG. 2, more or fewer data stores may be used without departing from the invention. For instance, in some examples, very large amounts of data may be used by the testing system200. Accordingly, multiple different data stores may be connected to the system in order to access all data desired for conducting testing on various information technology systems.

In some examples, one or more data types may be identified by the test script module208for use in testing an information technology system. The data types may be automatically identified based on project criteria received by the test preparation module204or based on one or more parameters of a test script received by the test script module208. Data corresponding to the identified data types may be retrieved from the data stores205a,205bfor use in testing the information technology system. In some examples, the data may be retrieved and forwarded to test module212.

The testing system200may also include a related data type module210. The related data type module210may identify one or more data types related to the data types identified by the test script module208that may be useful in testing the information technology system. For instance, the data types identified by the test script module208may be associated with (e.g., in a look-up table, and the like) with other data types. Identification of the data types by the test script module208may prompt the related data type module210to search for additional data types associated with the identified data types and automatically identify the additional data types. In some examples, the relationships between data types may be predefined and identification of additional data types may include automatically retrieving data of the additional data types and, in some examples, transmitting the data to the test module212.

Additionally or alternatively, the related data type module210may identify data types that may have a potential relationship with or association to the identified data types. In these examples, the data associated with the additional data types might not be automatically retrieved but rather a user may be prompted with a notification that additional data types have been identified and may be retrieved. A user may receive this notification and respond via user computer216. User computer216may include one or more computers (e.g., desktop computers, laptop computers, netbooks, computing terminals, and the like) such as computer100ofFIG. 1, cell phone, smart phone, and the like.

As mentioned above, the testing system200may include a test module212. The test module212may execute the one or more test scripts using the identified and retrieved data. The system200may further include a change tracking module214. The change tracking module may monitor the testing process and may record any changes made during the test (e.g., data used, test script modifications, and the like). The changes may be stored and may be reviewed or may be used in creating historical records of system testing.

FIG. 3illustrates one example method of testing an information technology system according to one or more aspects described herein. In step300, frequently used data is received (e.g., at testing system200inFIG. 2). The frequently used data may be received once at the system and remain stored within the system for later use as desired, in some arrangements. The frequently used data can be updated to include additional data meeting the criteria to be identified as frequently used, as discussed above. In step302, project criteria are received. As discussed above, the project criteria may include the information technology system being tested, test parameters, functionality of the information technology system being tested, type of system being tested. In step304, a portion of the received frequently used data may be identified for use in testing the information technology system based on the received project criteria. For instance, data may be identified based on, for example, the type of information technology system being tested, operating system of the information technology system, and the like. In some examples, identification of the portion of the frequently used data for use may include automatically retrieving the desired data.

In step306, a first test script may be received and the testing environment may be built or created. For instance, building or creating the test environment may include various test parameters may be established and/or implemented, instructions for testing the system may be implemented, tables may be developed, data protection parameters may be put in place, and the like. The first test script may include one or more test processes or instructions for testing the information technology system. Additionally or alternatively, the first test script may identify a first data type for use with testing the information technology system. The first data type may include a domain or type of data, such as customer data, deposit data, loan data, and the like. The first data type may, in some examples, include multiple data types that may be tested together. In step308a determination is made as to whether the first data type is associated with a predefined relationship. For instance, a determination is made as to whether a predefined relationship between the first data type and another data type exists. If the first data type is not associated with a predefined relationship, data of the first type will be received in step310and the first test script may be executed in step316.

If the first data type is associated with a predefined relationship in step308, a second data type may be identified in step312that is associated with the first data type. In step314, data of the first data type and the second data type is received and the first test script is run or executed with the data in step316.

FIG. 4illustrates another example method of testing an information technology system according to one or more aspects described herein. In step400, the first test script is received and the first data type is identified. As discussed above, a data type or domain may include various types of data and, in some examples, may include multiple different types of data that may be tested together. Step400may be similar to step308inFIG. 3and may be preceded by steps similar to steps300,302,306. In step402, a determination is made as to whether the first data type is associated with a potential predefined relationship. For instance, in some examples, a search of one or more look-up tables may be conducted to determine whether the first data type has any predefined relationships. That is, the first data type may not be explicitly identified in a predefined relationship but may have potential connection to or association with one or more other data types that may be useful in testing the information technology system. If the first data type is not associated with a potential predefined relationship, data of the first type is received in step404and is used in executing the first test script in step412.

If, in step402, the first data type is associated with a potential predefined relationship, additional data types may be identified based on the first data type and the predefined relationship in step406. In step408, a determination is made as to whether the identified additional data type will be included in the testing process (i.e., whether data of the additional data type will be received and used in testing). In some examples, this determination may be made by receiving user input. In other examples, the determination may be made automatically based on one or more project criteria or parameters If the additional identified data type is not included, data of the first type is received in step414and the test is executed in step412. If, in step408, the additional identified data type is included, data of the first type and the additional identified data type are received in step410and the first test script is executed in step412using the data of the first type and the additional identified data type.

In some examples, modifications or adjustments may be made to the system being tested upon completion of a testing method (e.g., upon completion of execution of the first test script). The testing method (e.g., first test script) may then be re-run after the modifications have been made to assess performance of the system being tested. These testing loops may continue until all testing is complete or until performance of the system being tested reaches a satisfactory result.

FIG. 5illustrates one example user interface including a notification that may be transmitted to a user to identify an additional data type identified based on a potential predefined relationship. The user interface500includes field502which provides an identification of the data type identified as potentially being associated with the first identified data type of the process. The user interface500may further include options for accepting or rejecting the identified additional data type. For instance, selection of “yes” option504may include the additional identified data type in running the test script and may cause the system to retrieve data of the additional identified type. Alternatively, selection of “no” option506may ignore the identified additional data type when executing the test script. User interface500may further include a “cancel” option508which may cancel any selections made or information identified.

FIG. 6illustrates one example method of identifying frequently used data. In step600, data of an identified type is received. In step602, a determination is made as to whether the data of the identified type has been received (such as by testing system200) a number of times greater than a predetermined threshold. If the identified data type has not been received a number of times greater than the threshold, the process may proceed to step606to determine whether additional data types are available for evaluation. If so, the process may return to step600to receive data of an identified type.

If the data has been received a number of times greater than the threshold, the data of the identified type may be added to the frequently used data in step604. In step606, a determination is made as to whether additional data types are available for evaluation. If so, the process may return to step600to receive data of an identified type. If not, the frequently used data may be stored and, in some examples, transmitted to a testing system (such as system200) for storage and/or use in testing.

The information technology system, method, apparatuses and computer-readable media described herein provide an integrated approach to testing information technology systems and may streamline testing information technology systems and may provide for more efficient testing methods. For instance, by providing access to frequently used data at the testing system, some data for testing may be automatically provided, without requiring copying of data, or storing the data in multiple locations. Further, automatically identifying data and data types that may be associated with other data types already included in the test plan may reduce the need for re-testing or reworking tests with the additional data. That is, data being used often is associated with or “touches” various other types of data. By automatically identifying the additional types of data that are associated, the proper data for testing is provided from the start, rather than later in the testing process. Because of the efficiency of the system and the integrated approach to testing described herein, the systems, methods, apparatuses, and computer-readable media described herein may be used to test systems using very large quantities of data. For instance, the testing systems, methods, and the like described herein may be used with various amounts of data. In some examples, the amount of data used during testing may be between 2 and 20 terabytes. Some arrangements may involve testing between 4 and 8 terabytes of data, while other arrangements (e.g., a major conversion) may involve 15 to 20 terabytes of data. The above-described ranges are merely some example amounts of data that may be used with the systems and methods described herein. Smaller or larger amounts of data may be tested without departing from the invention.

Further, the testing systems may include tracking capabilities that may identify changes to the test script, test parameters, project criteria, and the like and may store any changes made. The changes may be reviewed upon completion of the test or during the test to identify changes that may need to be made to the information technology system. This may provide the opportunity to identify problems, defects or issues with the information technology system while it is still in the development process, rather than in a post-production, in-use situation. In addition, the status of the testing being performed may also be tracked and, in some examples, the status information may be available for viewing, such as by various team members, the development team associated with the information technology system being tested, and the like.

The methods and features recited herein may further be implemented through any number of computer readable media that are able to store computer readable instructions. Examples of computer readable media that may be used include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, DVD, or other optical disc storage, magnetic cassettes, magnetic tape, magnetic storage and the like.

While illustrative systems and methods described herein embodying various aspects are shown, it will be understood by those skilled in the art that the invention is not limited to these embodiments. Modifications may be made by those skilled in the art, particularly in light of the foregoing teachings. For example, each of the elements of the aforementioned embodiments may be utilized alone or in combination or sub-combination with the elements in the other embodiments. It will also be appreciated and understood that modifications may be made without departing from the true spirit and scope of the present invention. The description is thus to be regarded as illustrative instead of restrictive on the present invention.