Implementing device-specific libraries for validation testing

Techniques are described herein for implementing device-specific libraries to perform validation testing for XR applications executing on various XR capable devices. The techniques include receiving an XR application executing on an XR capable device. The XR application utilizes one or more software modules that are integrated into the XR application. One or more validation tests may be performed for the XR application in response to a request. The request can at least specify usage of the one or more software modules corresponding to the XR capable device and testing specifications corresponding to the XR capable device. The XR application is mapped to the one or more software modules and the one or more validation tests are executed for the XR application according to the testing specifications. Test results from the one or more validation tests are reported to a testing log via the one or more software modules.

BACKGROUND

Software validation is determining if a system complies with the requirements and performs functions for which it is intended and meets an organization's goals and user needs. Validation is typically completed at the end of a software development process and is performed after a software product is produced against established criteria to ensure that the product integrates correctly into the environment. However, validation may also be employed during testing, provisioning, updating, patching, and so on.

In many cases, validation process and testing may be performed for these various scenarios on a number of user devices operating under different network conditions as organizations may wish to perform tests to validate certain configurations, modifications, patches, or bug fixes to properly supply the functionality desired by organizations and users alike. Additionally, the user devices may be produced by various Original Equipment Manufacturers (OEMs) that maintain different device-specific testing specifications. As a result, supporting the validation process and testing can require specialized equipment and a wide range of resources.

DETAILED DESCRIPTION

Techniques are disclosed for onboarding XR capable devices for performing validation testing. In one example, the validation process and testing may be applied to an extended reality (XR) application executed on one or more user devices or XR capable devices. As used herein, “XR” refers to all real and virtual environments generated by computer graphics and wearables. The “X” in XR is simply a variable that can stand for any letter. XR is the umbrella category that covers all the various forms of computer-altered reality, including Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR).

In one aspect, a testing device may be configured to perform one or more validation tests for an XR application being executed on a plurality of XR capable devices. The testing device may also include a testing robot or a simulator that is configured to mimic or replicate usage of the XR application on the XR capable device and/or execute a predetermined set of operations based at least on one or more of the validation tests being performed within a testing environment. The testing device can comprise an edge device that is operatively connected to a wireless communication network.

The testing device may be operated by a testing entity. In one aspect, the testing device receives, from the testing entity, an XR application executing on an XR capable device. The testing entity may be a telecommunications service provider or other such third parties. The XR application may utilize one or more software modules that are integrated into the XR application. The one or more software modules can comprise a Software Development Kit (SDK) that are provided by an Original Equipment Manufacturer (OEM) of an XR capable device. The SDK may include libraries, documentation (e.g., device specification), code samples, processes, guides, and/or other components of software tools and programs such as an Application Programming Interface (API).

The testing device may receive a request, from the testing entity, to perform one or more validation tests for the XR application executing on an XR capable device. The request may at least specify usage of the one or more software modules and include testing specifications corresponding to an XR capable device, wherein the testing specifications may specify which validation tests to apply. In some examples, the validation tests may include functional tests, performance tests, security tests, usability tests, compatibility tests, recoverability tests, regression tests, and/or so forth. The testing specifications may also include information relating to the operating system, file system, hardware, and/or other software systems of an XR capable device that includes the XR application.

In some aspect, the testing device may perform one or more validation tests for the XR application executing on multiple XR capable devices. The individual XR capable devices executing the XR application being validated may include different software features and hardware components. Accordingly, the individual XR capable devices may correspond to a set of testing configurations based at least on respective testing specifications. The one or more software modules and testing specifications may be locally stored at the testing device or at a database that is managed by the testing entity.

Before performing the one or more validation tests, the testing device may first map the XR application executing on the XR capable device to the one or more software modules such as the API of the SDK that is associated with the XR capable device. Upon performing the one or more validation tests for the XR application according to the testing specifications, the testing device may report or upload, via the API, test results, metrics, event data, and/or other such data to external systems such as a testing log that is managed by the testing entity and/or to an OEM of an XR capable device. The individual test results, metrics, event data, and/or other such data may be associated with a test case ID corresponding to a validation test.

In some aspects, at a testing device communicatively coupled to an XR capable device, the testing device may receive a request, from a testing entity, to perform one or more validation tests for an XR application executing on the XR capable device. The request may include a device identifier corresponding to the XR capable device. Based at least on the device identifier, the testing device may identify a device-specific library for the XR capable device and query the device-specific library to obtain testing specifications and one or more software modules that are available from the device-specific library. The device-specific library may be managed by an OEM of the XR capable device.

The testing device may perform multiple validation tests for multiple XR capable devices concurrently. Accordingly, the testing device may query multiple device-specific libraries in parallel, the individual device-specific libraries being managed by a unique OEM. In response to receiving testing specification and one or more software modules from the respective device-specific libraries, the testing device may map the individual testing specifications and one or more software modules to the XR application. The testing device executes one or more validation tests according to the testing specifications. Upon performing the one or more validation tests for the XR application, the testing device may report, via the one or more software modules, test results from the one or more validation tests to a testing log.

The testing log may be communicatively coupled to the testing and evaluation process monitoring server that may be operated by a testing entity. In some aspects, the testing and evaluation process monitoring server may store product requirements (e.g., implementation requirements, performance targets, service requirements, functions and features, quality requirements, customer experience/requirements, operations requirements, other technical requirements) and validation criteria (e.g., acceptance level, threshold for number of defects identified, expected result). Test data and test parameters for the validation tests may also be stored at a testing and evaluation process monitoring server. The testing and evaluation process monitoring server may receive the validation report from the testing log and validate the XR application based at least on the results in view of the product requirements and the validation criteria. In some embodiments, the testing device may also validate the XR application based at least on one or more of the results in the validation report during the validation process as one or more tests are being applied.

The techniques described herein may streamline the validation testing process for XR applications operating on multiple XR capable devices with, wherein the individual XR capable devices are associated with unique testing specifications and software modules for integrating with various XR applications. The techniques described herein may be implemented in a number of ways. Example implementations are provided below with reference to the following figures.

Example Network Architecture

FIG. 1illustrates an example architecture100for onboarding various XR capable devices to execute testing of an XR application. The architecture100may include one or more XR capable devices102(1) and102(2). The individual XR capable devices102(1) and102(2) may be produced or provided by an OEM106(1) and106(2), respectively. InFIG. 1, the first XR capable device104(1) may be a head-mounted device (HMD) and the second XR capable device104(2) may be a smartphone. In various embodiments, the XR capable devices102(1) and102(2) may be other mobile devices, personal digital assistants (PDAs), or other electronic devices having a wireless communication function that is capable of receiving input, processing the input, and generating output data. The XR capable devices102(1) and102(2) are connected to a telecommunication network112utilizing one or more wireless base stations or any other common wireless or wireline network access technologies.

The network112can be a cellular network that implements 2G, 3G, 4G, 5G, and long-term evolution (LTE), LTE advanced, high-speed data packet access (HSDPA), evolved high-speed packet access (HSPA+), universal mobile telecommunication system (UMTS), code-division multiple access (CDMA), global system for mobile communications (GSM), a local area network (LAN), a wide area network (WAN), and/or a collection of networks (e.g., the Internet).

InFIG. 1, each of the XR capable devices102(1) and102(2) includes an XR application104. Examples of XR applications include architecture, visual art, commerce, education, emergency services, video games, medical, military, navigation, and workplace applications. In any of these applications, a user may wear the first XR capable device102(1) such as an HMD to interact with one or more digitally-generated entities displayed by the first XR capable device102(1). Additionally, a user may utilize the second XR capable device102(2) such as a smartphone in order to interact with one or more digitally-generated entities displayed by the second XR capable device102(2).

FIG. 1also includes a testing and evaluation process monitoring server118that may store test data and test parameters for the validation tests. In one example, validation tests may include functional tests, performance tests, security tests, compatibility tests, recoverability tests, and/or so forth. Regression testing may also be performed after the desired changes or modifications are made to the existing code after completing one or more validation tests.

The functional tests generally evaluate whether the XR application104operates as specified in product requirements. In this regard, the functional tests may evaluate the interactions between individual software modules of the XR application104. The functional tests may also evaluate the interactions between the software modules of the XR application104and the operating system, file system, hardware (e.g., input devices such as a microphone, a keyboard/keypad, mouse devices, a touch-sensitive display that accepts gestures, voice or speech recognition devices, hand gesture recognition devices, etc.), and/or other software systems of the XR capable devices102(1) and102(2) on which the XR application104resides. The functional tests may also validate whether the XR application104performs according to the requirement of all versions of mobile communication protocols (e.g., 3G, 4G, LTE, 5G, etc.). Further, the functional tests may validate whether the XR application104enables the XR capable devices102(1) and102(2) to display appropriate notifications and error messages.

The performance tests determine whether the XR application104performs acceptably under certain performance requirements (e.g., access by a large number of users) or different load conditions. For example, the performance tests may determine whether a network coverage can support the XR application104at a peak user level, an average user level, and a minimum user level. The performance tests may also determine whether the XR application104performs acceptably under certain performance requirements when a network is changed to Wi-Fi from a cellular network or vice versa while moving around with the XR capable device, or when only intermittent phases of connectivity are available.

The performance tests may further determine whether the existing client-server configuration setup provides the required optimum performance level (e.g., as specified under product requirements and/or evaluation criteria), and help identify various application and infrastructure bottlenecks which prevent the XR application104from performing at the required acceptability levels. Additionally, the performance tests may help validate whether the response time of the XR application104is as per the requirements. In some aspects, the performance tests may also evaluate whether the energy consumption (e.g., battery lifespan) of the XR capable devices102(1) and102(2) can support the XR application104to perform under projected load volumes.

The security tests may generally analyze the data storage and data validation requirements, enable the session management for preventing unauthorized users to access unsolicited information, and determine whether the business logic implementation is secured and not vulnerable to any attack from external systems. The compatibility tests generally determine whether the XR application104is compatible with the XR capable devices102(1) and102(2). The XR capable devices102(1) and102(2) may have different operating systems, resolution, screen, version, and hardware. For instance, the compatibility tests may validate that the user interface of the XR application104fits the screen size of the XR capable devices102(1) and102(2), such that no text or content on the user interface is only partially visible or inaccessible on the XR capable devices102(1) and102(2). The recoverability tests generally evaluate crash recovery and transaction interruptions. For example, the recoverability tests may determine how the XR application104handles recovery after unexpected interruption/crash scenarios. In another example, the recoverability tests validate how the XR application104handles transactions during a power failure or when a connection to a network is suspended.

The validation tests may be executed via a testing device114. The testing device114can comprise a robotic device that is configured to mimic or replicate usage of the XR application104on the XR capable devices102(1) and102(2) and/or execute a predetermined set of operations based at least on one or more of the validation tests being performed within a testing environment. The testing device114may also include general-purpose computers, such as desktop computers, tablet computers, laptop computers, servers (e.g., on-premise servers, edge servers), or other electronic devices that are capable of receiving input, processing the input, and generating output data. The testing device114may store data in a distributed storage system, in which data may be stored for long periods and replicated to guarantee reliability.

The testing device114may be equipped with testing hardware122that may be configured to operate the XR capable devices102(1) and102(2). For instance, the testing hardware122may include an image capturing device or an image sensor, whereby the testing device114may be configured to support or wear the first XR capable device102(1) (i.e., an HMD) such that the image capturing device or the image sensor may be aligned to view the display screen of the first XR capable device102(1). In another example, the testing hardware122may include a robotic hand and/or digits that may be configured to secure a grasp around the second XR capable device102(2) (i.e., a smartphone) and interact with the touchscreen of the second XR capable device102(2).

Accordingly, the testing hardware122may include mechanical components for providing movement that is equivalent to human gesture (e.g., head movement, neck movement, arm movement, hand movement, finger movement, etc.) that may be communicatively coupled to a navigational or directional component in response to movement commands by a test administrator. The testing device114may also include additional user interface, data communication, or data storage hardware. For example, the user interfaces may include a data output device (e.g., visual display, audio speakers), and one or more data input devices (e.g., sensors) for receiving user input.

The testing device114comprises a testing and evaluation module124for performing one or more validation tests. Before conducting validation tests, however, the testing device114may initially onboard the XR capable devices102(1) and102(2). In one example, the testing device114may register the XR capable devices102(1) and102(2) with a server (e.g., testing and evaluation process monitoring server118) of the testing entity by providing device identifiers (e.g., serial numbers), device type (e.g., HMD, smartphone, gaming apparatus, etc.), device model, manufacturer information, and/or so forth. Additionally, the testing device114may obtain testing specifications and one or more software modules from the XR capable devices'102(1) and102(2) OEMs106(1) and106(2).

In one example, to onboard the first XR capable device102(1), the testing device114may query a first device-specific library108(1) requesting testing specifications110. The first device-specific library108(1) may be maintained and operated by the first OEM106(1) that produces the first XR capable device102(1). Similarly, the second device-specific library108(2) may be maintained and operated by the second OEM106(2) that produces the second XR capable device102(2). The device-specific libraries108(1) and108(2) inFIG. 1may include logical organizational structures that store data, such as relational databases, object databases, object-relational databases, and/or key-value databases. Accordingly, the device-specific libraries108(1) and108(2) may store multiple records such as testing specifications110and one or more software modules.

In response to the query from the testing device114, the device-specific library108(1) may return available testing specifications110corresponding to an XR capable device102(1). The testing specifications110may include information relating to the operating system, file system, hardware, and/or other software systems of the first XR capable device102(1). In some aspects, the testing device114may provide the testing specifications110to the testing and evaluation process monitoring server118. The testing specifications110may be stored locally at the testing device114and/or cached at the testing and evaluation process monitoring server118. The testing and evaluation process monitoring server118is configured to provide testing configurations132to the testing device114based at least on the testing specifications110. The testing configurations132may be based at least on product requirements and the evaluation criteria and may also instruct the testing device114which validation tests to apply and the pre-requisite for each test.

The testing device114may also query the first device-specific library108(1) requesting one or more software modules associated with the first XR capable device102(1). In the illustrated embodiment, the one or more software modules can comprise a Software Development Kit (SDK)116. The SDK116may include libraries, documentation (e.g., device specification corresponding to the first XR capable device102(1)), code samples, processes, guides, and/or other components of software tools and programs such as an Application Programming Interface (API)120. In one example, the API120may be configured to facilitate communications of test events and other data related to validation testing between the first XR capable device102(1) and/or the OEM106(1) and one or more computing nodes or servers (e.g., testing device114, testing and evaluation process monitoring server118, testing log134, etc.) of the testing entity. In response to the query, the device-specific library108(1) may return SDK116corresponding to the first XR capable device102(1).

The testing device114may include a device-specific library management module126. The device-specific library management module126may include an interface component for communicating with the device-specific libraries108(1) and108(2) and other components of the testing device114such as the testing and evaluation module124and the validation module128. In some aspects, the device-specific library management module126may use a header based authentication model to a request from the device-specific library108(1) testing specifications110and/or one or more software modules (e.g., SDK116). Authentication credentials may be associated with the testing device114and the testing entity. In some aspects, the authentication credentials may be associated with an account identifier for a test account, a developer account, and/or a beta account of the testing entity. As the testing device114performs multiple validation tests for multiple XR capable devices, the device-specific library management module126is configured to manage a data structure (e.g., a mapping table) stored in a local memory of the testing device114or a memory maintained at a testing entity. In some examples, the device-specific library management module126may implement a management logic for testing specifications and SDKs from multiple device-specific libraries108(1) and108(2) operated by one or more OEMs106(1) and106(2).

The testing device114comprises a validation module128that interfaces with the testing and evaluation module124. Upon completing the validation testing, the validation module128may generate a validation report130to be transmitted, for example, via the API120, to a testing log134. The validation report130may include test results from the individual validation tests. Additionally, the validation report130may include other test events from the individual validation tests. The testing log134may interface with the testing and evaluation process monitoring server118. In some aspects, the testing log134may pass the validation report130and/or event-related data to the testing and evaluation process monitoring server118. The testing and evaluation process monitoring server118may also interface with OEMs106(1) and106(2) and a network provider to receive product requirements and evaluation criteria. In response to receiving the validation report130, the testing and evaluation process monitoring server118may analyze the test results and perform validation of the XR application104based at least on the product requirements and the evaluation criteria.

If multiple validation tests are applied, the validation module128may analyze individual test results as the validation tests are completed in a sequence. If the validation module128determines that a test result from a first validation test indicates that the XR application is not validated at least in part, then the validation module128may instruct the testing and evaluation module124to terminate or suspend the validation testing process before completing additional validation tests and transmit the validation report130via the API120to the testing log134. Alternatively, all of the validation tests may be completed before the validation module128validates the XR application104.

Example Computing Device Components

FIG. 2is a block diagram showing various components of illustrative testing devices200that performs validation testing. It is noted that the testing devices200as described herein can operate with more or fewer of the components shown herein. For example, the testing devices200may include specific hardware or mechanical components for operating various types of XR capable devices such as HMD, smartphones, wearable devices, gaming apparatus, and/or so forth. Additionally, the testing devices200as shown herein or portions thereof can serve as a representation of one or more of the computing devices of the present system.

The testing devices200may include a communication interface202, one or more processors204, hardware206, and memory208. The communication interface202may include wireless and/or wired communication components that enable the testing devices200to transmit data to and receive data from other networked devices. In at least one example, the one or more processor(s)204may be a central processing unit(s) (CPU), graphics processing unit(s) (GPU), both a CPU and GPU, or any other sort of processing unit(s). Each of the one or more processor(s)204may have numerous arithmetic logic units (ALUs) that perform arithmetic and logical operations as well as one or more control units (CUs) that extract instructions and stored content from processor cache memory, and then execute these instructions by calling on the ALUs, as necessary during program execution.

The one or more processor(s)204may also be responsible for executing all computer applications stored in the memory, which can be associated with common types of volatile (RAM) and/or nonvolatile (ROM) memory. The hardware206may include additional user interface, data communication, or data storage hardware. For example, the user interfaces may include a data output device (e.g., visual display, audio speakers), and one or more data input devices. The data input devices may include but are not limited to, combinations of one or more of keypads, keyboards, mouse devices, touch screens that accept gestures, microphones, voice or speech recognition devices, and any other suitable devices.

The processors204and the memory208of the testing devices200may implement an operating system210, a device-specific library management module212, a testing and evaluation module216, and a validation module218. The operating system210may include components that enable the testing devices200to receive and transmit data via various interfaces (e.g., user controls, communication interface, and/or memory input/output devices), as well as process data using the processors204to generate output. The operating system210may include a presentation component that presents the output (e.g., display the data on an electronic display, store the data in memory, transmit the data to another electronic device, etc.). Additionally, the operating system210may include other components that perform various additional functions generally associated with an operating system.

The device-specific library management module212may correspond to the device-specific library management module126ofFIG. 1. The device-specific library management module212is configured to interface with one or more device-specific libraries and/or other external systems operated or controlled by OEMs to onboard XR capable devices for validation testing. In some aspects, the device-specific library management module212may send a request for testing specifications and/or one or more software modules (e.g., SDK) and manage the received testing specifications and SDKs from multiple device-specific libraries operated by one or more OEMs. For example, the device-specific manager212may implement management logic to identify the XR capable device executing an XR application being validated and identify requisite and available testing specification and/or one or more software modules from a device-specific library associated with the XR capable device.

The device-specific library management module212includes a mapping module214. The mapping module214is configured to process or analyze the request for testing specifications and/or one or more software modules to identify a target device-specific library associated with the request or characterize the request as associated with a specific OEM and/or a specific XR capable device. In some aspects, the request may include a device identifier that is associated with an XR capable device and an OEM. Accordingly, the mapping module214may implement OEM identifier detector logic that may cause the requests including device identifiers to be mapped to individual device-specific libraries that are associated with target OEM identifiers. The mapping module214may populate a mapping table with information to indicate how requests from the device-specific library management module212are mapped or routed to specific device-specific libraries.

Similarly, the mapping module214may populate a mapping table with information to indicate how information from the device-specific libraries is mapped to an XR application executing on an XR capable device. In some aspects, the mapping module214may be configured to map the one or more software modules from a device-specific library to the XR application executing on the XR capable device associated with the OEM that manages the device-specific library. For instance, the mapping modules214may map one or more APIs of the SDK to the XR application executing on the XR capable device. The API may be an Onboard Device API to enable the testing devices200to add or register the XR capable device to a test account during an onboard process. The API may also be a Delete Device API to enable the testing devices200to delete or remove the XR capable device from a test account. Additionally, the API may be a Get Events API to download validation test events and test results from the testing devices200to the testing entity such as the testing and evaluation process monitoring server and/or the testing log.

The testing and evaluation module216may correspond to the testing and evaluation module124ofFIG. 1. The testing and evaluation module216is configured to perform one or more validation tests based at least on test configurations. The validation tests may include functional tests, performance tests, security tests, usability tests, compatibility tests, recoverability tests, regression tests, and/or so forth. In some aspects, the testing and evaluation module216may prioritize one or more validation tests according to the test configurations. Additionally, the testing and evaluation module216may assign individual validation tests to one or more computing instances of the testing devices200.

The validation module218may correspond to the validation module128ofFIG. 1. The validation module218may receive the test results from the testing and evaluation module216to generate a validation report and transmit the validation report to a testing log. In some aspects, the validation module218may analyze test results generated by one or more computing instances that execute one or more validation tests to determine whether the XR application is validated. The validation module218may analyze test results as individual validation tests are completed (e.g., in a sequence). Accordingly, the validation module218may instruct the testing and evaluation module216to terminate testing if one or more test results indicate that the XR application is not validated. Conversely, the validation module218may analyze test results when all validation tests are completed. In some aspects, the validation module218may analyze the test results in accordance with product requirements and evaluation criteria. Accordingly, the validation module218may indicate in the validation report whether the XR application is validated.

FIG. 3is a block diagram showing various components of illustrative computing devices300that manages validation testing and testing configurations. It is noted that the computing devices300as described herein can operate with more or fewer of the components shown herein. Additionally, the computing devices300as shown herein or portions thereof can serve as a representation of one or more of the computing devices of the present system.

The computing devices300may include a communication interface302, one or more processors304, hardware306, and memory308. The communication interface302may include wireless and/or wired communication components that enable the computing devices300to transmit data to and receive data from other networked devices. In at least one example, the one or more processor(s)304may be a central processing unit(s) (CPU), graphics processing unit(s) (GPU), both a CPU and GPU, or any other sort of processing unit(s). Each of the one or more processor(s)304may have numerous arithmetic logic units (ALUs) that perform arithmetic and logical operations as well as one or more control units (CUs) that extract instructions and stored content from processor cache memory, and then execute these instructions by calling on the ALUs, as necessary during program execution.

The one or more processor(s)304may also be responsible for executing all computer applications stored in the memory, which can be associated with common types of volatile (RAM) and/or nonvolatile (ROM) memory. The hardware306may include additional user interface, data communication, or data storage hardware. For example, the user interfaces may include a data output device (e.g., visual display, audio speakers), and one or more data input devices. The data input devices may include but are not limited to, combinations of one or more of keypads, keyboards, mouse devices, touch screens that accept gestures, microphones, voice or speech recognition devices, and any other suitable devices.

The processors304and the memory308of the computing devices300may implement an operating system310, a test management module312, and a load management module318. The operating system310may include components that enable the computing devices300to receive and transmit data via various interfaces (e.g., user controls, communication interface, and/or memory input/output devices), as well as process data using the processors304to generate output. The operating system310may include a presentation component that presents the output (e.g., display the data on an electronic display, store the data in memory, transmit the data to another electronic device, etc.). Additionally, the operating system310may include other components that perform various additional functions generally associated with an operating system.

The test management module312may manage test data314and test parameters316for one or more validation tests. The test data314may be created manually, by using data generation tools, or retrieved from an existing production environment. Similarly, the test parameters316may be created manually, by using parameter generation tools or retrieved from an existing production environment. If created manually, the test parameters316may replace hardcoded values. In some aspects, parameter values for the test parameters316specified may be derived from another test or test script. For example, one or more of the parameter values may be derived from a previously performed validation test.

Additionally, the test management module312may generate testing configurations based at least on testing specifications from a device-specific library. In one example, the testing configurations may specify which validation tests to apply. The testing configurations may also identify which computing instance to instantiate in the testing and evaluation pipeline, and further assign individual computing instances to execute one or more validation tests.

During testing, the load management module318may be configured to act as a load balancer for one or more computing instances in a testing and evaluation pipeline. In some aspects, the load management module318may assign validation tests to the individual computing instances upon determining that an output queue for a computing instance is full. Upon completion of the one or more validation tests, the test management module312may receive a validation report from the testing and evaluation pipeline via a testing log. In turn, the test management module312may analyze the test results in the validation report to perform validation of the XR application based at least on product requirements320and evaluation criteria322.

Example Processes

FIGS. 4 and 5present illustrative processes400-500for performing validation tests from the perspective of one or more testing devices. The processes400-500are illustrated as a collection of blocks in a logical flow chart, which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the process. For discussion purposes, the processes400-500are described with reference toFIGS. 1-3.

FIG. 4is a flow diagram of an example process400for onboarding XR capable devices for performing validation testing from a perspective of a testing device. At block402, a testing device, via a testing and evaluation module, receives an XR application executing on an XR capable device. The XR capable device may be registered with a testing and evaluation process monitoring server of a testing entity. The XR capable device may comprise various types of devices associated with a unique OEM. The XR application utilizes one or more software modules such as an SDK that is configured to be integrated into the XR application.

At block404, the testing and evaluation module may receive a request to perform one or more validation tests for the XR application. The request may be transmitted from a testing entity and can specify the usage of the one or more software modules and testing specifications corresponding to the XR capable device. In some aspects, the testing and evaluation module may interface with a device-specific library management module of the testing device to receive testing specifications from one or more device-specific libraries maintained by OEMs. The testing specifications may include information relating to the operating system, file system, hardware, and/or other software systems of the XR capable device that includes the XR application.

At block406, a mapping module of the device-specific library management module maps the XR application executing on the XR capable device to the one or more software modules. In one example, the mapping module may map the XR application to an API of the SDK. The API may enable one or more computing nodes of the testing entity to download validation test events from the testing device. At block408, the testing and evaluation module executes the one or more validation tests for the XR application according to the testing specifications. The validation tests may include functional tests, performance tests, security tests, usability tests, compatibility tests, recoverability tests, regression tests, and/or so forth. One or more of these tests may be applied in a predefined sequence.

At block410, a validation module of the testing device may report, via the one or more software modules, test results from the one or more validation tests to a testing log. In some aspects, the validation module may validate the XR application executing on the corresponding XR device based at least on test results and one or more evaluation criteria. For example, the validation may include comparing each set of the test results to corresponding evaluation criteria. Accordingly, the test result generated by each computing instance may be determined to be validated when a set of test results meet corresponding evaluation criteria.

FIG. 5is a flow diagram of an example process for identifying device-specific libraries based at least on XR capable devices performing validation testing, from the perspective of a testing device. At block502, a testing and evaluation module of the testing device receives a request to perform one or more validation tests for an XR application executing on an XR capable device. The request may be transmitted from a testing entity and can include a device identifier corresponding to the XR capable device. At block504, a device-specific library management module of the testing device identifies a device-specific library for the XR capable device based at least on the device identifier (e.g., device serial number). In one example, the device-specific library management module includes a mapping module that may implement OEM identifier detector logic that may cause the requests including device identifiers to be mapped to individual device-specific libraries that are associated with target OEM identifiers. The mapping module may populate a mapping table with information to indicate how requests from the device-specific library management module are mapped or routed to specific device-specific libraries.

At block506, the device-specific library management module obtains testing specifications and one or more software modules such as an SDK that are available from the device-specific library. In one example, the device-specific library management module may identify a device-specific library for the XR capable device and query the device-specific library to obtain testing specifications and one or more software modules that are available from the device-specific library. At block508, the mapping module of the device-specific library management module maps the XR application to the one or more software modules. In some aspects, the mapping module may populate an additional mapping table with information to indicate how information from the device-specific libraries is mapped to an XR application. At block510, the testing and evaluation module executes the one or more validation tests for the XR application according to the testing specifications. The one or more validation tests may be performed concurrently for an XR application executing on one XR capable device. Additionally, or alternatively, the one or more validation tests may be performed in a sequence for individual XR capable devices. At block512, a validation module of the testing device reports via the one or more software modules, test results from the one or more validation tests to a testing log. The test results may also be downloaded from the testing device to a computing node at the testing entity via an API of the SDK.

CONCLUSION