Patent ID: 12210443

It is to be appreciated that elements in the figures are illustrated for simplicity and clarity. Common but well-understood elements that may be useful or necessary in a commercially feasible embodiment may not be shown in order to facilitate a less hindered view of the illustrated embodiments.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Embodiments of the present invention are related to apparatus, systems, and methods for an automated testing platform. According to some embodiments, the automated testing platform captures data in a consistent manner over time (persistent) for quality assurance in performance testing of devices and/or services. For example, in a case where a vendor updates a device (or service), the automated testing platform can compare the performance of the device to vendor claims (about performance), compare the performance of the device to previous performance profiles of the device (i.e., before the update), compare the performance of the device to the performance of other devices known to the automated testing platform, and the like.

According to example embodiments, a framework automates a test processes for evaluating one or more technologies. Example embodiments include a framework that automates a test process configured to evaluate disparate components.

Consider the following example implementation where a developer (or other user) wants to evaluate a 3rd party application that claims to perform device fingerprinting. This application is configured to generate a connection alert to a homeowner's device whenever a device (e.g., a test client) connects or disconnects to a home's network. The application is further configured to generate a history alert about whether the test client has been connected to the network before on a device connection event. According to at least one example, the history alert is included in the connection alert so that a single alert includes the connection information and the history information. In the example, the user wants to conduct a test where a new device (a first test client), “Alice's phone,” is connected to the network and disconnected after one minute. After another minute passes, Alice's phone is to be reconnected to the network and disconnected after one additional minute. In this scenario, the user expects to receive four alerts (in the case of the combined connection/history alert) from the application, as follows:1. time=0 secondsa. Alice's phone has connected to the networkb. This device has never connected to the network before2. time=60 secondsa. Alice's phone has disconnected from the network3. time=120 secondsa. Alice's phone has reconnected to the networkb. This device has been on the network before4. time=180 secondsa. Alice's phone has disconnected from the network

In this example scenario, the user wants to correlate several data sources (e.g., multiple test clients) to understand the usefulness of the application. The user may want to know if the alerts are correct (i.e., in steps 1 and 3, does the application correctly recognize if the device has been on the network before?). Additionally, the user may want to know about how long it takes for the application to process the event (i.e., when Alice's phone connects to the network for the first time at time=0, is the homeowner alerted at time=1 or time=10?).

Because the first test client, Alice's phone, is periodically connected to the network, an automated testing platform according to an example embodiment is configured to listen for the alerts from the application on a second test client, “Bob's server.” The testing platform allows Alice's phone and Bob's server to execute this test as a team, with correlated timestamps between the two testing devices. This allows the data from both devices to be correlated after the test is completed. Additionally, because the test configuration is defined in code of the testing platform, the test can be executed multiple times, consecutively, to gain a richer data set; this functionality is built into the automated testing platform.

According to some embodiments, and as illustrated in the example block diagram100ofFIG.1, a testing cloud application101, one or more test clients102, a user103interfacing with the testing cloud application, e.g., through a server/computer, and a third party cloud104facilitate the synchronization of data and the ability for multiple test clients to be able to execute operations (e.g., testing the cloud104) simultaneously. It should be understood that while a third-party cloud is described as a device under test (DUT), the DUT can be any device or service, internal or external to a test system.

According to an example embodiment, the testing cloud application101manages tests, processes and assigns jobs to the test clients102, and gathers data from a device and/or service under test, such as the third party cloud104.

According to an example embodiment, the test clients102wait for jobs from the testing cloud application101and processes jobs received from the testing cloud application101. According to an example embodiment, the test clients102can perform various functions, including checking in with the third party cloud104, announcing capabilities, subscribing for jobs, receiving jobs, sending job data output, and updating job availability. According to some embodiments, the test clients102are components of the testing cloud application101.

According to an example embodiment, the third party cloud104can be a third party cloud. According to an example embodiment, the third party cloud104provides data and functionality, for example, to perform plugin executables/methods. According to an example embodiment, the third party cloud104can be the device under test.

According to an example embodiment, the user103designs and loads the tests, and interprets data returned by the other components (e.g., the testing cloud application101, the test clients102, and/or the third party cloud104). According to an example embodiment, the user103can create a job file comprising, for example, a test template (see, for example, test template definition800,FIG.8), a new test (see, for example, test definition900,FIG.9), an environment (see, for example, environment definition1000,FIG.10), and a test series (see, for example, series definition1100,FIG.11).

It should be understood that the examples shown inFIGS.8-11are examples, and are not limiting. These and other embodiments of the test template, new test, environment, and test series definitions are contemplated.

According to an example embodiment, and as illustrated in the example block diagram ofFIG.2, the automated testing platform200includes one or more test clients201, a transport orchestrator202, an Application Program Interface (API)203, a client manager204, and a data processor205. According to an example embodiment, the components of the automated testing platform200enable the synchronization of data, and the ability for several test clients201to be able to execute operations simultaneously. According to at least one embodiments, these operations are applied against a DUT (e.g., a third party cloud).

According to some embodiments, the test clients201are devices that execute test operations and report results. One example of a test client201includes a Wi-Fi connected device (such as a phone or laptop) that can browse the internet or hop between different network connections. Another exemplary test client201includes a cloud-to-cloud adapter, integrating 3rd party APIs into test data. These and other test clients are contemplated within embodiments of the present invention.

According to some embodiments, the transport orchestrator202is configured for establishing and maintaining connections with the test clients201. According to some embodiments, the transport orchestrator202is configured for routing messages between the DUT and a particular, uniquely addressable test client. According to some embodiments, the transport orchestrator202is configured to support multiple routing and transport protocols, allowing the testing cloud application to not be concerned with client specific transport protocols.

According to some embodiments, the API203is configured for storing test-related resources. This allows a user206to define the job file of test templates, tests, environment resources, and test series. According to some embodiments, the API203is configured to allow a user configuration to associate tags to test clients, which allows for test isolation and client specificity. According to at least one embodiments, the tags enable the user to claim a test client, direct the test client to perform an action (e.g., connect to WiFi network X), define an owner, etc. Test isolation refers to an ability to group test clients at a given location, such that, according to some embodiments, the number of jobs running at that location can be guaranteed to be one, e.g., the test job. According to some embodiments, tags with locations can be associated with the test clients and an isolation parameter for the given location. Client specificity refers to the addressability of test clients, e.g., through tagging, and enables the instruction (by the automated testing platform200) of identified test clients to run certain tests.

According to some embodiments, the API203is configured to allow the user206to execute a test series, and retrieve data as the test results are reported/uploaded. The API203exposes the functionality to create and review comprehensive reports of test series data after a series (e.g., test client operations against the DUT) has finished executing. The API203exposes endpoints so the user206can claim test client ownership (if permitted by the test client), subscribe to cloud events (e.g., a test series has finished running), etc.

According to some embodiments, the client manager204is configured to ensure data about new test clients is stored in database, e.g.,213, available to the API203, and that individual test client status updates are reflected in proper event streams (for example, “offline”, “ready”, “pending”, “busy”, etc.). According to some embodiments, the client manager204is configured to listen for heartbeats to establish if a client is currently offline or online, and to publish any change in status to an event bus207.

According to some embodiments, the data processor205is configured to consume data from individual test clients201and, at the end of a series, create generic data reports overviewing the emitted events by the test clients201. According to some embodiments, the data processor205is configured to read messages requesting more demanding or series-specific reports from the event bus207, then process the data and create those reports. According to some embodiments, after a report is created, it is put onto the event bus207where the API203can consume it and store it as a resource.

According to an example embodiment, the automated testing platform200further includes a user-facing graphic user interface (GUI)208enabling the user206to create test templates, new tests, test environments, etc. According to an example embodiment, the GUI208enables access to the API203. According to an example embodiment, the GUI208enables access to a client ownership module214, developer webhooks215, data analytic APIs216, etc.

According to an example embodiment, the automated testing platform200connects to a DUT, such as the third-party cloud ofFIG.1, through one or more test facilities, e.g.,212. According to some embodiments, the test facility212includes one or more of a test client device201and plugin executables209. According to some embodiments, the test client device201, for example a computing device, can run the plugin executables209.

According to some embodiments, the plugin executables209run on the data processor205of the automated testing platform200, running the testing cloud application101, and/or the test client(s)201. In this example, the test facility212is internal to the automated testing platform200. Accordingly, the testing cloud application101(seeFIG.1) can be configured within the automated testing platform200(e.g., configured to properly handle test client operations, challenges, requests, etc.).

According to some embodiments, the plugin executables209can have multiple jobs.

According to one or more embodiments, the third party clouds104, or more generally a DUT, interacts with either the testing cloud application101or the test client(s)102. These interactions can be implementation dependent.

Embodiments of the present invention can further make use of a certificate validation service module210and a transport server211. According to some embodiments, the transport server211established connections to the test facility212via a bi-directional transport network and appropriate communication protocols, e.g., encrypted security protocols and the like. According to at least one embodiment, a certificate (e.g., a device fingerprint) is provided by each test client to the automated testing platform200, where the certificate validation service module210validates and/or authenticates each test client.

Moreover, according to some embodiments, one or more of the transport orchestrator202, the API203, and the data processor205include, or have access to, a memory (e.g., a database (DB).

According to at least one embodiment, and referring toFIG.3, a method300of operating an automated testing platform is presented in a case where a user206creates and runs a test using a client201.

According to at least one embodiment, at step(s)301, the user206defines a test template via the API203. A test template defines the job types that will run as part of a test. A non-limiting example of a job type includes a test client configured to connect to, and 30 seconds later disconnect from, a DUT, such as an Access Point or other third party hardware under test. Another non-limiting example of a job type includes a test client configured to subscribe to a 3rd party cloud to receive alerts when a device connects or disconnects from an Access Point. Yet another non-limiting example of a job includes a test client configured to randomize its MAC address, connect to an Access Point, and disconnect from the Access Point 30 seconds later.

According to some embodiments, jobs, such as the non-limiting examples above, are defined in a Test Template. In one example, the three jobs are defined in the same test template in a case where they are part of the same test.

According to some embodiments, the test template can contain environmental variables that can be referenced in the job types (e.g., and defined in other documents). According to some embodiments, the test template defines the method in which test data will be sent to the DUT (e.g., cloud), which can be during the tests, after each test iteration, or after an entire test series.

According to at least one embodiment, at step(s)302, the user206creates a test definition via the API203. A test definition references a test template, and includes extra information that gives further context to the test template. For example, the test definition can define how many iterations of the test take place. Additionally, the test definition can specify how many test clients will be performing each job type (e.g., the user wants4clients to be connecting to the Access Point and disconnecting 30 seconds later).

According to at least one embodiment, at step(s)303, the user206creates an environment via the API203. According to some embodiments, the environment is created according to a defined context (or environmental variables) in which a test can take place. In one or more embodiments, the environment is created according to the environmental variables required by the test template, for example, an environment having one WiFi device on a Service Set Identifier (SSID) (e.g., “wifi_one_ssid”). According to some embodiments, the environmental variables can contain a list of “exclusive tags” used to ensure only one test is running on a test facility (e.g., a set of test clients); this is useful, for example, if a test consumes a large amount of bandwidth, and the user does not want other tests to be running on the network simultaneously. According to at least one embodiment, the user can configure a test client to perform jobs in parallel, and/or the conditions under which parallel jobs are permitted. According to some embodiments, the environmental variables can make references to job types, and use test client tags that ensure specific jobs run on specific test clients; this can be useful, for example, if the user wants the test client connecting to and from the DUT to be the test client with the tag: “room=bedroom”.

According to at least one embodiment, at step(s)304, the test clients201are powered on and running a client firmware or application that allows for communication with the testing cloud application (e.g., the event bus207and transport orchestrator202). According to at least one embodiment, upon connection, the test clients201send metadata about themselves, which includes a list of capabilities, or operations that can be executed. According to some embodiments, the client manager204ensures each test client201has a status of “online” in the testing cloud application.

According to at least one embodiment, at step(s)305, the user206creates a test series via API203. According to at least one embodiment, the test series encapsulates a plurality of elements of a test, and an order in which the elements are to be performed. For example, one element may instruct a test client to connect to a DUT, while another element may instruct the test client to perform another function.

According to some embodiments, by creating the test series, the user206is instructing the testing cloud application to start the test. For example, the user206can send a reference to the test definition and the environment, which must already be created for a series to successfully be created. When a user206creates a test series, the API203ensures test clients201with the necessary capabilities (and tags, if applicable) are available to run the jobs at step(s)306(e.g., using a publishing jobs event through the client manager204). The API203will then reserve the test clients201(if applicable) and create jobs for each test client, which defines the exact operations (with environment variables fully populated) the test client needs to perform. The API203associates all of these jobs with the series, and the newly created jobs are routed to the test client201where the job needs to run.

According to at least one embodiment, at step(s)307, a handshake is performed between test clients201and the testing cloud application. According to at least one embodiment, the test clients201acknowledge when they have received the job and wait for the signal from the testing cloud application to start executing job operations. The testing cloud application correlates all of the client acknowledgments and waits for all clients to check in and report that they are ready to start executing job operations.

According to at least one embodiment, at step(s)308, a job start handshake is initialized between the testing cloud application and the test clients201. The testing cloud application signals to every test client to begin processing job operations. According to at least one embodiment, from this point onwards, the test clients do not need to be connected to the testing cloud application and can still be processing a job at step(s)309.

According to at least one embodiment, at step(s)309, before job execution by the test clients201, each test client201executes the “before” section of its job file. This section will get the test client201to a state where it is ready to perform a test section of the job file (e.g., if a test client201must first connect to an Access Point (the DUT in this example) then disconnect 30 seconds later during a test, the test client201would disconnect from the Access Point during this step so it can connect during the test). According to at least one embodiment, there is a specified duration in the job file.

According to at least one embodiment, at step(s)309, test iterations are performed by the test clients201. According to some embodiments, each test client201executes the operations defined in the “beforeEach,” “Test,” and “afterEach” section of its job file the number of times specified by the “iterations” fields of the job file. According to some embodiments, each of these sections has a specified duration in the job file.

According to at least one embodiment, and again at step(s)309, the “afterEach” section is executed by the test clients201. According to some embodiments, each test client201executes the “afterEach” section of the job file. According to some embodiments, this section will get the test client201to a normal state (e.g., the test client201connected to the internet and the testing cloud application). According to at least one embodiment, the test clients201send data during this phase. According to at least one embodiment, the “afterEach” section has a specified duration in the job file.

FIG.4is a sequence diagram showing a method of initializing a test client connecting to the testing cloud application400in advance of running a job in accordance with an example embodiment. According to some embodiments, at bracket401the test client201provides a certificate to the certificate validation service module210, such that the test client201is recognized by the transport server211of the testing cloud application, and an encrypted tunnel between the testing cloud application and the test client can be established. According to at least one embodiment, at bracket402the test client may transmit an announcement of available functions, processes, abilities, etc., to the plugin executable, device type (of the test client), etc., indicating to the testing cloud application that the test client is ready for a job. According to some embodiments, at event403the API203ensures that a test client database is accessible, where the test client database is populated with, for example, tags associated with the test clients201, the functions of each test client exposed as capabilities, a status of each test client (e.g., connected, ready), etc. According to some embodiments, at event(s)404, a request to connect the test client to the testing cloud application is transmitted by the client manager204to the event bus207, and the test client's status (e.g., status: connected) is stored in the test client database. According to at least one embodiments, at bracket405the status of the test client is finalized in a client connection event and a connection with the test client is established via the transport orchestrator202and transport server211.

FIG.5is a sequence diagram showing a method500of a test client reporting data in accordance with an example embodiment. Referring toFIG.5, “job complete” status reports can be sent by test clients to the testing cloud application according to some embodiments of the present invention. According to at least one embodiment, the test clients send test results data to the testing cloud application and send a message to report that all operations for a job have been completed. Test clients may report this information during a job, e.g., after each iteration501and/or after a certain phase of the job (test)502. Test clients may report this information after completing a job503, for example, in a case where the test clients do not have a connection to the testing cloud application during the test. According to at least one embodiment, the test client201sends test data to the transport orchestrator202, which publishes the test data to the event bus207and data processor205to further processing. According to some embodiments, the testing cloud application marks related resources as appropriate (e.g., the testing cloud application may remove the job from a test client document, update test client status, etc.) at step(s)504.

According to at least one embodiment, data reports are automatically created by the data processor205upon the completion of a test or test series. According to at least one embodiment, the data processor205creates and publishes the report to the event bus207for interested parties, such as the user206. The API203, for example, can consume the report and make it available to the user206. According to at least one embodiment, data reports are created by the data processor in response to a request by the user to the API, for example, where the user requests a specific or detailed data report. This may be a request by the user to the API, where the request is published to the event bus, read by the data processor, and the data processor then publishes the report to the event bus.

According to at least one embodiment, test data is available to the user206. According to some embodiments, reports automatically created by the data processor205are also available. The test series can be marked as complete (e.g., in the test client database) by the testing cloud application. According to some embodiments, the user206can run a new test series with the same test definition and the environment (as desired); that is, tests are repeatable using prior test definitions. According to some embodiments, the user206can modify and run a new test series with the modified context. For example, the API203enables user access to a database213storing test definitions, enables the user to develop and/or store new test definitions, enables the user to analyze test results, e.g., using data analytic APIs216provided by the API203, etc. According to some embodiments, the automated testing platform200

Recapitulation

Given the discussion thus far, it will be appreciated that, in general terms, an exemplary method for operating an automated testing platform600(seeFIG.6), according to an aspect of the invention, includes the steps of providing a testing cloud application running on the automated testing platform601; providing a test facility602; receiving, by the testing cloud application, a job file of a test603; connecting the test facility to the testing cloud application604; publishing job data prescribed in the job file of the test to the test facility, wherein the job data configures the test facility to perform at least one test of a device or service under test and synchronizes a plurality of elements of the test605; receiving, by the testing cloud application, result data from the test facility, which has performed at least one of the plurality of elements of the test606; and generating a report of the result data607.

An automated testing platform200system includes a test facility212; an application program interface (API)203storing a job file; a transport orchestrator202configured provide the job file to the test facility, wherein the job file includes synchronized elements of a test to be performed by the test facility on a device under test; and a data processor205configured to receive test data from the test facility, create a report using the test data, and publish the report to the application program interface as a resource.

The invention can employ hardware aspects or a combination of hardware and software aspects. Software includes but is not limited to firmware, resident software, microcode, etc. One or more embodiments of the invention or elements thereof can be implemented in the form of an article of manufacture including a machine-readable medium that contains one or more programs which when executed implement one or more method step(s) as disclosed herein; that is to say, a computer program product including a tangible computer readable recordable storage medium (or multiple such media) with computer usable program code configured to implement the method steps indicated, when run on one or more processors. Furthermore, one or more embodiments of the invention or elements thereof can be implemented in the form of an apparatus including a memory and at least one processor that is coupled to the memory and operative to perform, or facilitate performance of, exemplary method steps.

Yet further, in another aspect, one or more embodiments of the invention or elements thereof can be implemented in the form of means for carrying out one or more of the method steps described herein; the means can include (i) specialized hardware module(s), (ii) software module(s) executing on one or more general purpose or specialized hardware processors, or (iii) a combination of (i) and (ii); any of (i)-(iii) implement the specific techniques set forth herein, and the software modules are stored in a tangible computer-readable recordable storage medium (or multiple such media). Appropriate interconnections via bus, network, and the like can also be included.

As is known in the art, part or all of one or more aspects of the methods and apparatus discussed herein may be distributed as an article of manufacture that itself includes a tangible computer readable recordable storage medium having computer readable code means embodied thereon. The computer readable program code means is operable, in conjunction with a computer system, to carry out all or some of the steps to perform the methods or create the apparatuses discussed herein. A computer readable medium may, in general, be a recordable medium (e.g., floppy disks, hard drives, compact disks, EEPROMs, or memory cards) or may be a transmission medium (e.g., a network including fiber-optics, the world-wide web, cables, or a wireless channel using time-division multiple access, code-division multiple access, or other radio-frequency channel). Any medium known or developed that can store information suitable for use with a computer system may be used. The computer-readable code means is any mechanism for allowing a computer to read instructions and data, such as magnetic variations on a magnetic media or height variations on the surface of a compact disk. The medium can be distributed on multiple physical devices (or over multiple networks). As used herein, a tangible computer-readable recordable storage medium is defined to encompass a recordable medium, examples of which are set forth above, but is defined not to encompass transmission media per se or disembodied signals per se. Appropriate interconnections via bus, network, and the like can also be included.

FIG.7is a block diagram of at least a portion of an exemplary system700that can be configured to implement at least some aspects of the invention, and is representative, for example, of one or more of the client(s), server(s), apparatus or modules shown in the figures. As shown inFIG.7, memory730configures the processor720to implement one or more methods, steps, and functions (collectively, shown as process780inFIG.7). The memory730could be distributed or local and the processor720could be distributed or singular. Different steps could be carried out by different processors, either concurrently (i.e., in parallel) or sequentially (i.e., in series).

The memory730could be implemented as an electrical, magnetic or optical memory, or any combination of these or other types of storage devices. It should be noted that if distributed processors are employed, each distributed processor that makes up processor720generally contains its own addressable memory space. It should also be noted that some or all of computer system700can be incorporated into an application-specific or general-use integrated circuit. For example, one or more method steps could be implemented in hardware in an ASIC rather than using firmware. Display740is representative of a variety of possible input/output devices (e.g., keyboards, mice, and the like). Every processor may not have a display, keyboard, mouse or the like associated with it.

The computer systems and servers and other pertinent elements described herein each typically contain a memory that will configure associated processors to implement the methods, steps, and functions disclosed herein. The memories could be distributed or local and the processors could be distributed or singular. The memories could be implemented as an electrical, magnetic or optical memory, or any combination of these or other types of storage devices. Moreover, the term “memory” should be construed broadly enough to encompass any information able to be read from or written to an address in the addressable space accessed by an associated processor. With this definition, information on a network is still within a memory because the associated processor can retrieve the information from the network.

Accordingly, it will be appreciated that one or more embodiments of the present invention can include a computer program comprising computer program code means adapted to perform one or all of the steps of any methods or claims set forth herein when such program is run, and that such program may be embodied on a tangible computer readable recordable storage medium. As used herein, including the claims, unless it is unambiguously apparent from the context that only server software is being referred to, a “server” includes a physical data processing system running a server program. It will be understood that such a physical server may or may not include a display, keyboard, or other input/output components. Furthermore, as used herein, including the claims, a “router” includes a networking device with both software and hardware tailored to the tasks of routing and forwarding information. Note that servers and routers can be virtualized instead of being physical devices (although there is still underlying hardware in the case of virtualization).

Furthermore, it should be noted that any of the methods described herein can include an additional step of providing a system comprising distinct software modules or components embodied on one or more tangible computer readable storage media. All the modules (or any subset thereof) can be on the same medium, or each can be on a different medium, for example. The modules can include any or all of the components shown in the figures. The method steps can then be carried out using the distinct software modules of the system, as described above, executing on one or more hardware processors. Further, a computer program product can include a tangible computer-readable recordable storage medium with code adapted to be executed to carry out one or more method steps described herein, including the provision of the system with the distinct software modules.

Accordingly, it will be appreciated that one or more embodiments of the invention can include a computer program including computer program code means adapted to perform one or all of the steps of any methods or claims set forth herein when such program is implemented on a processor, and that such program may be embodied on a tangible computer readable recordable storage medium. Further, one or more embodiments of the present invention can include a processor including code adapted to cause the processor to carry out one or more steps of methods or claims set forth herein, together with one or more apparatus elements or features as depicted and described herein.

Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be made by one skilled in the art without departing from the scope or spirit of the invention.