Automated configurable portable test systems and methods

In one embodiment, a test system comprises a first interface for communicating with remote devices; a second interface for communicating with local devices; a memory for storing information, including information received from the first interface and second interface; a processor for automatically configuring test system components in accordance with the information stored in the memory. The test system components comprise a network access point simulation component and a local control component. The network access point simulation component is configured to simulate communication network access point operations comprising test interactions with user equipment. The number of devices under test included in the user equipment and distinct network access points that are coincidentally simulated can be variable. The local control component is configured to direct the network access point simulation component and to control the test interactions with the user equipment.

RELATED APPLICATIONS

This application is related to the following U.S. patent applications, all of which are incorporated herein by reference in their entirety:

FIELD OF THE INVENTION

Embodiments of the present invention relate to the field of device testing.

BACKGROUND

Numerous electronic technologies such as digital computers, video equipment, and telephone systems have facilitated increased productivity and reduced costs in processing information in most areas of business, science, and entertainment. More and more the components used in these activities interact with a network (e.g., the internet, the cloud, etc.). The number of electronic devices used in these activities is growing rapidly, with new versions and new types of devices with diverse capabilities being continuously and rapidly introduced. Thorough testing of the devices under many different scenarios is important to make sure the devices will function correctly. Providing proper testing environments is often critical to achieving accurate test results. However, when the devices are interacting with very large networks beyond the control of the tester it is difficult to ensure accurate test results.

Traditional attempts at testing devices that communicate with large networks often involve trying to simulate the large communication network. This typically involves significant resources. The traditional approaches are typically implemented in a large stationary facility or room with lots of costly equipment attempting to simulate the large communication network. In addition, providing radio frequency interference mitigation for the large facilities is also typically very expensive and involving numerous individual different test devices in a large shielded room (e.g., oscilloscopes, voltmeters, etc.). These large facilities often require significant manual interaction and supervision to accurately test a device. Each different type of device under test often involves a complete reset and reconfiguration of the large facility. It is also usually inconvenient and disruptive for ongoing field operations to ship products to a single facility for testing. Traditional attempts to automate some aspects of the testing are typically limited. Conventional approaches typically require significant manual support for various activities such as configuring the test environment, equipment maintenance, test case delivery, device profile delivery, test data collection, data analytics and reporting, and consulting, for example. These factors contribute significantly to the cost of traditional device testing.

SUMMARY

In one embodiment, a test system comprises a first interface that communicates test management information; a second interface that communicates with user equipment; a memory that stores information, including information received from the first interface and second interface; a processor for that automatically configures test system components in accordance with the information stored in the memory. The test system components comprise a network access point simulation component and a local control component. The network access point simulation component is configured to simulate communication network access point operations comprising test interactions with user equipment. The number of devices under test included in the user equipment and distinct network access points that are coincidentally simulated can be variable. The local control component is configured to direct the network access point simulation component and to control the test interactions with the user equipment. The local control component comprises a test executive operable to direct simulation of communication network operations and the test interactions in accordance with information received from the remote control components. The network access point simulation component and local control component can be portable. The local test control component can be implemented by the processor.

The test configuration information can include user equipment control configuration information. The types of test network components that are configured can vary. The type of the test network components that are configured can include: a small cell, an evolved packet core (EPC) component, evolved node B (eNodeB) component, Internet Protocol Multimedia System (IMS) component and application servers. Additional test network simulation components are configured and the number can vary. The automatically configuring of test network simulation components operable to simulate test network components can be performed locally. The number of devices under test and the type of devices under test in the user equipment can vary. Automatically configuring user equipment test control components operable to control communications with user equipment can be performed locally. In one embodiment, a test box is communicatively coupled to the network access point simulation component. The test box can include material operable to shield contents of the test box from electromagnetic radiation interference, wherein contents of the test box includes at least one of the devices under test.

In one embodiment a test method comprises: receiving test configuration information; automatically configuring a test network simulation component operable to simulate test network components comprising test network communication components based on the test network configuration information; and automatically configuring a user equipment test control component operable to control communications with user equipment in accordance with the under test control configuration information. The type of the test network component that is configured can vary. In one exemplary implementation the type of the test network components that are configured is selected from the group comprising: a small cell, an evolved packet core (EPC) component, evolved node B (eNodeB) component, Internet Protocol Multimedia System (IMS) component and application servers. The number and type of devices under test in the user equipment can vary. The number and type of test network simulation components that are configured can vary. Automatically configuring a test network simulation component operable to simulate test network components can be performed locally.

DETAILED DESCRIPTION

Efficient and effective flexible test systems and methods are presented. In one embodiment, a test system is readily adaptable to a variety of configurations. The configurations can be automatically implemented locally and can be based on a large reservoir or database of test information stored and managed remotely. The test systems can be automatically configured to simulate network communication interactions that correspond to various different implementations (e.g., small cell operations, EnodeB operations, evolved packet core (EPC) operations etc,). The test systems can be configured to operate in a variety of implementations (e.g., various different types of devices under test, a single device under test, a plurality of devices under test, a single network access point is simulated, a plurality of network access points are simulated, etc.). The test systems are portable and can be conveniently deployed in local environments.

The local test systems and methods facilitate easily implemented convenient local testing of various user equipment. The local test systems and methods can be portable and easy to use, unlike traditional test systems. Unlike conventional test approaches that typically have a number of limitations, traditional testing approaches usually have very cumbersome and complicated test equipment and configuration procedures that consume significant resources to implement and maintain. Even though traditional approaches consume significant resources, the testing capabilities of the traditional testing approaches are also usually limited. For example, the configuration of UE to eNodeB and EPCs (e.g., one to one, one to multiple, and multiple to multiple, etc.) are typically limited or not possible in traditional approaches. A number of traditional test systems and method are also typically directed to limited types of devices that are tested. Local test systems and methods are easily adaptable to and configurable for different UE devices under test. Traditional approaches do not even typically attempt this flexibility and scalability due to the cost of the traditional resources and daunting traditional configuration issues. In a local test system and method, the local test system components have reasonably costs to implement and the automated configuration can be substantially effortless from a user's perspective.

FIG. 1is a block diagram of an exemplary test system100in accordance with one embodiment. Test system100includes a remote management environment101and a local test environment102. The local test environment101includes local control component110and local test user equipment interface component120. In one embodiment, the test system is configured to test user equipment130. Local control component110is configured to direct the local test user equipment interface component120and to control test interactions with the user equipment130. The local test user equipment interface component120is operable to communicate with the user equipment130during test operations. In one exemplary implementation, the local test user interface component120and local control component110are portable.

In one embodiment, the local test user equipment interface120is configured to simulate various communication characteristics and features (e.g., communications in accordance with a communication infrastructure component, protocol, network, architecture, etc.). The local test user equipment interface120can include communications mechanisms compatible with various different types of communication links for communicating with the user equipment130. The communication links can include wireless communication links (e.g., cellular, WiFi, small cell, etc.), wired communication links (e.g., coaxial radio frequency (RF) link, Ethernet, universal serial bus (USB), etc.), or combinations of different types of communication links.

It is also appreciated that user equipment (UE) can include a variety of different devices under test. The devices under test may provide end users with many different capabilities (e.g., cell phones, computers, tablets, laptops, devices in the Internet-of-things (IoT), etc.). The user equipment can include the capability to collect and exchange data among themselves and with other devices over a network. The user equipment can communicate over networks through a wired or wireless medium or communication link using different types of network protocols, such as but not limited to the 3rd Generation Partnership Project (3GPP) Long-Term Evolution (LTE) standard.

Test system100is compatible with simulating various communication network environments or architectures for communicating with the user equipment130. In one exemplary implementation, the local test user interface component120is a network access point simulation component configured to simulate network access point operations. The local control component110can simulate a network communication core. The components of the test system can be automatically configured. The configuration of the test environment topology is also flexible. The user equipment can comprise a single device under test or a plurality of devices under test. The number of devices under test and distinct interfaces or network access points that are coincidentally simulated is variable. A single network access point can be simulated or a plurality of network access points can be simulated.

It is appreciated that a plurality of simulated network components and network interfaces or access points can be implemented coincidentally. In one embodiment, the plurality of simulated network components and network interfaces or access points can be implemented substantially simultaneously or in parallel. It is also appreciated that a plurality of UE devices under test can be can be implemented coincidentally. In one embodiment, the plurality of simulated UE devices under test can be can be implemented substantially simultaneously or in parallel.

In one embodiment, local control component110includes an interface component190configured to communicate management information. Interface component190can be configured to communicate management information from a variety of sources and received via a variety of mechanisms. The mechanism for delivery of management information can be an external network communication connection to a component external to the local test environment, an internal communication connection to another internal test environment management component, and so on. InFIG. 1the source of the management information is a remote management environment (e.g., test management services received from the cloud, a remote site, etc.). In another embodiment, the source of the management information is a from local test management related resource. In one exemplary implementation, the management information communicated via a network to the local test system environment. In one exemplary, implementation the management information is loaded onto a portable storage device (e.g., a jump drive, other flash storage mechanism, DVD, etc.) at a remote location and the storage device is transported to a local test environment where the management information is downloaded from the portable storage device to a component in the local test environment (e.g., local control component110, local test user equipment interface component120, etc.).

In one embodiment, local control component110can communicate with a remote component via a real network in the remote environment101. In one embodiment, the real network (e.g., the Web, the internet, the Cloud, etc.) is a “real” network for communicating information in a normal mode as opposed to being part of a “simulated” network used for testing. The remote component can be a server that provides various testing related information (e.g., device profile test information, test management information, etc.)

While the above management information communications and remote management environment interactions are described as flowing to the local test environment, it is appreciated the local test environment can forward information in the opposite direction. In one embodiment, the local test environment can forward information externally through similar communication mechanisms (e.g., via a communication network, via physical transportation of portable storage devices, etc.). The information can be communicated to a remote management environment.

In one embodiment, configuration of the local control component110and local test user equipment interface120is automated. The configuration can be based upon information received from the remote management environment101. The automated configuration can include automated configuration of various aspects of a local test system (e.g., software, firmware, hardware, etc.).

The automated configuration can include automated configuration of local test systems described in Ser. No. 15/236,326 entitled “Local Portable Test Systems and Methods” by Dinesh Doshi et al., which is incorporated herein by reference. In one exemplary implementation, the configuration of local control component110and local test user equipment interface120is performed with little or no local manual interaction. The remote management environments101can be similar to remote management environments described in Ser. No. 15/236,292 entitled “Cloud-Based Services For Management Of Cell-Based Test Systems” by Dinesh Doshi et al., which is incorporated herein by reference. In one exemplary implementation, the configuration of local control component110and local test user equipment interface120is performed with little or no local manual interaction.

FIG. 2is a block diagram of an exemplary automated test configuration method200in accordance with one embodiment. In one embodiment, the automated test configuration method200can be performed with little or no local manual interaction. In one exemplary implementation, a user can participate at least in part in the configuration operations. The automated test configuration method can be initiated in response to updates or newer versions of UE. The newer version of the UE may have different capabilities and communication characteristics. Thus, the local test system is automatically configured or reconfigured to handle testing of the different capabilities and communication characteristics when running tests on the UE.

In block210, test configuration information is received. The test configuration information can include user equipment control configuration information. The test configuration information can be received from a remote management environment. The test configuration information can include device profile information. The device profile has the information or intelligence to control a given type of user equipment being tested. A device profile may be a standard application program interface (API). The API can be modified or adapted by an original equipment manufacturer (OEM) for a particular piece of UE. In one embodiment, a device profile is or includes a test script that adapts generalized software (test code) associated with a component and executed by the computer system to the specific type, make, model, and/or features of the user equipment being tested. In one exemplary implementation, the generalized software is associated with an EPC or IMF at a generic level (e.g., characteristics, features, etc. common to multiple components of the same type, etc.) and the test script adapts the generalized EPC or IMF software to an EPC config file indicating how a specific vendor or communication service provider wants an EPC or IMF configured.

In block220, a test network simulation component operable to simulate test network components is configured, including test network communication components based on the test network configuration information. The configuration of test network simulation components can be performed locally.

The types of test network components that are configured can vary. The type of test network components that are configured can include: a small cell, an evolved packet core (EPC) component, evolved node B (eNodeB) component, Internet Protocol Multimedia System (IMS) component and application servers. The number of test network simulation components that are configured can vary. In one exemplary implementation, the configuration includes configuring a local control component. The local control component can be similar to local control component110.

In block230, a user equipment test control component operable to control communications with user equipment is configured in accordance with the test configuration information. The number of devices under test and the type of devices under test in the user equipment can vary. The configuring of the user equipment test control components can be performed locally. In one exemplary implementation, the configuration includes configuring a local test user equipment interface component. The test user equipment interface component can be similar to test user equipment interface component120. In one embodiment, test user equipment interface component can at least in part be implemented on a local control component.

In one embodiment, a local control component is implemented on a computer system. The computer system can be portable. The local control component can include various test associated modules implemented on the computer system.FIG. 3is block diagram of exemplary testing system modules in accordance with one embodiment. Local control component310includes User Equipment Control Module323, Network Interface Control Module322, Core Network Control Module321, and Services Module327. User Equipment Control Module323controls test interactions with the user equipment. Network Interface Control Module322controls configuration of a local test interface component. In one embodiment, the local test interface component is a network access simulation component. The local test interface component can be a network access point component. Core Network Control Module321controls configuration and operations of a simulated communications network core. Services Module327directs simulation of various services operations. Local control component310communicates with user equipment350via network interface simulator340. Local control component310also communicates with remote management services301.

It is appreciated the test system is compatible with simulating various network environments for communicating with user equipment. The simulated network access point can simulate evolved node B (eNodeB) component operations, small cell operations, evolved packet core (EPC) operations, and so on. The configuration of the test network topology is also flexible. The user equipment can comprise a single device under test or a plurality of devices under test. A single network access point can be simulated or a plurality of network access point topologies described in Ser. No. 15/236,326 entitled “Local Portable Test Systems and Methods” by Dinesh Doshi et al., which is incorporated herein by reference.

In one embodiment, a small cell is a network access node that utilizes relatively low power radio communications with a limited range. In one exemplary implementation, the range is between approximately 10 meters and up to approximately 2 kilometers. A small cell can be a femtocell, a picocell, a microcell and so on. The small cell can include a wide range of interfaces (e.g., GSM, LTE interfaces including eNodeB, other 3GPP interfaces, CDMA 2000, WCDMA, LTE, Wi-Fi, TD-SCDMA, etc.).

FIG. 4is a block diagram of an exemplary local test environment400in accordance with one embodiment. Local test environment400includes portable test equipment409and user equipment430. Portable test equipment409includes local control component410and network interface simulation component417. Local control component includes test executive415. Local test environment400can communicate with remote test environment401.

In one embodiment, local control component410directs network interface simulation component417and controls communication with user equipment430via network interface simulator417. In one exemplary implementation, local control component410directs configuration of network interface simulation component417. The local control component410can also direct configuration of itself. The configurations can be based on information received from a remote management environment. The configuration of both local control component410and network interface simulation component417can be substantially or completely automated.

In one embodiment, the local control component410and network interface simulation component417are configured to simulate a communication network420. The simulated communication network420can include network core component430functions and simulated network interfaces441and442. The network interface simulation component417can simulate communications for interacting with the user equipment430in accordance with characteristics and features of a simulated network interface441and442. The simulated network core component functions can correspond to a public data network (PDN) component functions. The local control component411can also simulate interactions with various services472and network administrative operations471. Network interfaces441and442can include a variety of different types of interfaces with the user equipment450. In one embodiment, network interface441is a cellular wireless interface and network interface442is a land line interface.

It is appreciated that the local flexible test system and method approach can enable testing configuration installation and maintenance that facilitates convenient and effective testing of user equipment. In one exemplary implementation, local flexible test systems and methods can handle configurations and re-configurations associated with ongoing evolutions and revisions to communication network technology.

In one embodiment, automated configuration is utilized to handle evolution of a Universal Mobile Telecommunications System (UMTS) network communication to a Long Term Evolution (LTE) network communication architecture. It is appreciated that this is not a trivial task. It is complicated and complex to deal with and implement changes from a Node B network access point interface in a Universal Terrestrial Radio Access (UTRA) of an UMTS architecture to a Evolved UMTS Terrestrial Radio Access (E-UTRA) Node B or eNode B in the UTRA of an LTE architecture. The reconfiguration can include changing the UTRA protocols of Wideband Code Division Multiple Access (WCDMA) or Time Division-Synchronous Code Multiple Access (TD-SCMA) on E-UTRA protocol Uu interfaces to Orthogonal Frequency Division Multiple Access (OFDMA) for downlinks and Single Carrier-Frequency Domain Multiple Access (SC-FDMA) for uplinks on the LTE-Uu interfaces. Even understanding what the vast complex terminology means at high levels of description, let alone the intricate and sophisticated details of how the details of such network communication systems are implemented is very difficult to say the least. Traditional systems typically required highly trained and specialized users to configure the testing. Automated configuration in the local flexible test system and method approach enables efficient and convenient testing proliferation to a vast number of users.

FIG. 5is a block diagram on an exemplary local test environment evolved packet core (EPC) simulation in accordance with one embodiment. architecture. In one embodiment, the evolved packet core nodes and network access points are compatible with a system architecture evolution (SAE) architecture and core. Local test environment500includes portable test equipment520and user equipment530. Portable test equipment520includes local control component510and network access point simulation component517. Local control component includes test executive515. Local test environment500can communicate with remote test environment501. In one embodiment, the local control component510and network access point simulator517are configure to simulate a communication network520. The simulated communication network520can include network core component530functions and simulated network access points541and542.

The network access point simulation component541simulates an eNobeB access point. The simulated eNodeB access point can be compatible with 3rd Generation Partnership Project (3GPP) accesses (e.g., GPRS, UMTS, EDGE, HSPA, LTE, LTE advanced, etc.). The eNodeB network access point can be a wireless communication network access point (e.g., similar to a cellular communication system base station, a small cell, etc.). The access point542simulates a non-eNodeB access point (e.g., non-3GPP access technology, WiFi, etc).

The local control component510simulates LTE network core operations. The simulated LTE network core operations can include simulation of operations typically associated with a Mobile Management Entity (MME)532, a Serving Gateway (S-GW)532, a Home Subscriber Server (HSS)533, a PDN Gateway (PGW)534and535, an AAA component537and an Evolved Packet Data Gateway (EPDG)538. The simulated network core can include simulated interactions is various services571on PDN(s) (e.g., IMS, Internet, etc.) and administrative services572including Access Network Directory Selection Function (ANDSF).

FIG. 6is a block diagram of exemplary modules for implementing a communication network in accordance with one embodiment. Test executive610is implemented on a local control component in a local test environment and cloud services690and internet services698are implemented in a remote environment. Test executive610is similar to and LTE compatible implementation of test executive515inFIG. 5. Test Executive610includes UE control module620, eNB control module650, EPC control module640. In one embodiment, EPC control module640can direct implementation of a user interface (UI) that enables a user to interact with the test system.

The UE control module620directs simulation of various UE functions including Hayes command set Attention (AT) modem control function621, Operating System Android Debug Bridge (OS ADB) and US control function622, physical robotic control function623and Universal Integrate Circuit Card (UICC) function624. The eNode control function includes LTE PHY function631, layer 2 Media Access Control (MAC)/Radio Link Control (RLC)/Packet Data Convergence Protocol (PDCP) function632, Radio Resources Control (RRC) function633, Non Access Stratum (NAS) EMME/ESM/USI function624, Internet Protocol (IP) function635, User Datagram Protocol (UDP)/Real-time Transport Protocol (RTP)637, and Transmission Control Protocol (TCP)638. The eNode B control module also simulates various other communication functions (e.g., WiFi, L2, L3, L4/5/6, etc.).

EPC control module640directs simulation of various EPC functions including HSS function642, SGW function643, PGW function644and PCRF function645and ePDG function671. The EPC control module640can also direct simulation of various server functions650including Evolved Multimedia Broadcast Services (eMBMS), BMSC, content functions551, Over The Air Device Management (OTA-DM) functions652, IP Fader functions653. IP Multimedia Subsystem (IMS) functions654, Domain Name Service (DNS) functions655, File Transfer Protocol (FTP)/Hyper Text Transfer Protocol (HTTP) functions656, Streaming functions657and Subscriber Identification Module Over The Air management (SIM OTA) functions658. The remote cloud services690can include test case library691, UE library672, Test User Manager694, Reporting and Data Analytics695and SME Engineering697. Working together the EPC control module640and eNodeB control module630can direct simulation of UICC function673.

FIG. 7is a block diagram of an exemplary local test environment general network switching architecture and core simulation in accordance with one embodiment. Local test environment700includes portable test equipment720and user equipment730. Portable test equipment720includes local control component710and network access point simulation component717. Local control component includes test executive715. Local test environment700can communicate with remote test environment701. In one embodiment, the local control component710and network access point simulation component717are configured to simulate a communication network720. The simulated communication network720can include network core component functions and simulated network access points741and742. In one embodiment, the network access point741is wireless and network access point742is wired. The simulated network core component functions can correspond to communication switches731,832,734and735. The simulated network core can include simulated interactions with various services772and administrative functions771.

It is appreciated the same physical local controller and network access simulator can be automatically reconfigured to simulate a different network architecture.FIG. 8is a block diagram of local test environment700being reconfigured to simulate a Global System for Mobile Communications or Group Special Mobile (GSM) network. Simulated communications network820includes base station network access points841,842,843, and844, Base Station Controllers (BSCs)831,832, and833, Mobile Switching Center (MSC)823, General Mobile Switching Center (GMSC)827, Service Control Point (SCP)822and Service Data Point (SDP)821. The switching centers can include a Visitor Location Registry (VLR). The core components can also communicate with a Short Message Service (SMS) center.

FIG. 9is a block diagram of an exemplary test system in accordance with one embodiment. Local test environment900includes portable test equipment920and user equipment930. Portable test equipment920includes local control component910and network access point simulation component917. Local control component includes test executive915. Local test environment900can communicate with remote management environment901. In one embodiment, the local control component910and network access point simulation component917are configured to simulate a communication network920in accordance with information received from the remote management environment901. Remote management environment901real network access points903and904, physical network core infrastructure905and remote test management components909. The components of remote management environment901are real network components that participate in normal network communication operations as opposed to simulated network components in the local test environment that participated in testing operations. In one embodiment, the communications in the local test environment are protected from communication interference by components in the remote management environment.

In one embodiment, operations of a local test system are validated. The validation can include checking interactions between the local test components and a trusted reference component that simulates user equipment. The reference component can also be used for calibrating the local test components. In one exemplary implementation, the configuration of the local test system is validated.

FIG. 10Ais a block diagram of an exemplary test system in accordance with one embodiment. Local test environment1000includes portable test equipment1020and user equipment1030. Portable test equipment1020includes local control component1010and network access point simulation component1017. In one exemplary implementation, local control component includes test executive1015. Local test environment1000can communicate with remote management environment1001. In one embodiment, the local control component1010and network access point simulation component1017are configured to simulate a communication network1020in accordance with information received from the remote management environment1001.

In one embodiment, the network access point simulation component1017is communicatively coupled to reference component1050. Reference component1050is operable to validate results of the local control component automatic configuration of the test system components. In one exemplary implementation, the reference component1050validates the simulated network communications are operating correctly. The network access point simulation component, local control component and reference component are portable. The reference component simulates user equipment communications. The validation includes calibration of the test system components.

FIG. 10Bis a block diagram of an exemplary reference component1070in accordance with one embodiment. Reference component1070includes verification and calibration control component1071, wired connection interface1087and wireless connection interface1080. The verification and calibration control component1071directs the verification and calibration. In one embodiment, verification and calibration control component1071performs gathers and analyzes information from interactions with external components (e.g., a local test system, a remote management system, etc.).

In one exemplary implementation, verification and calibration control component1071includes processing component1072, memory1071, and reference signal generator1075. Processing component1072generates the processing component1072, directions and can optionally perform analysis of the verification and calibration results. Memory1071stores instructions and information for processing component1072. Reference signal generator1075generates reference signals with particular characteristics (e.g., particular frequency, voltage, etc.). Wireless connection interface1080communicates with external components wirelessly. In one exemplary implementation, Wireless connection interface1080includes antenna1085, transceiver1081, and signal processing component1082.

In one embodiment, the reference component can be an integral part of a local test system component. In one exemplary implementation, the reference component in integrated in the local test control component. The reference component can be integrated in a network access interface simulation component.

It is appreciated that a reference component can be configured to verify and calibrate various different characteristics and features of a local test system. The reference component can verify and calibrate characteristics of a physical layer, protocol layer, and data layer. The validation of the physical layer can include checking communication signal characteristics (e.g., signal strength, frequency, waveform shape, in proper RF bandwidth, channel, MIMO correlations, differences in uplink/downlink, etc.). The reference component can verify and calibrate protocol layer activities including simulated communication network component operations (e.g., EPC operations, server operations, sequencing and scheduling of component attachment, security conformity, etc.). Verification and calibration can include data layer operations (e.g., IP, RMF, data throughput, etc.). In one embodiment, the reference component can be used to verify the integrity of the testing control in a local environment (e.g., check if a test box is shielding from environment electrical interference on RF signals, humidity of test environment, etc.).

It is appreciated that the verification and calibration can be iterative and progressive. In one embodiment, a particular verification and calibration process is performed iteratively. In one exemplary implementation, signal strength is checked and if it is week a calibration change is made to increase the strength and then the signal is checked again, and so on until the signal strength is verified or validated as correct. In one embodiment, a verification and calibration is performed progressively. In one exemplary implementation, the signal characteristics are verified and calibrated and if resolved satisfactorily, then simulated component verification and calibration are performed. If there is still an issued verification of the integrity of the testing condition control can be performed.

In one embodiment, interactions between a remote management system and a reference component include updating the reference component information and configuration information. The interactions can be communicated with or without the local test system in the communication path (e.g., the reference component can communicatively couple directly to a remote network or can go through the local test system components). In one exemplary implementation, if there is a change to UE (e.g., new or updated version of the EU is introduced by a original equipment manufacturer (OEM), correction of a bug in the UE, etc.), and a reference component is meant simulate that UE, then the remote management component can initiate a corresponding appropriate change to the reference component.

FIG. 11Ais a block diagram of a test method1100in accordance with one embodiment.

In block1110, test configuration information is received. The test configuration information can include and user equipment control configuration information.

In block1120, a test network simulation component operable to simulate test network components including test network communication components based on the test network configuration information is automatically configured. The type of the test network component that is configured varies. The type of the test network components that are configured is selected from the group comprising: a small cell, an evolved packet core (EPC) component, evolved node B (eNodeB) component, Internet Protocol Multimedia System (IMS) component and application servers. The number and type of devices under test in the user equipment varies.

In block1130, a user equipment test control component operable to control communications with user equipment in accordance with the under test control configuration information is automatically configured.

In block1140, configuration and operations of the test network simulation component and the user equipment test control component are verified. The verification can include calibrating the test network simulation component and the user equipment test control component. The number and type of the test network simulation components that are configured varies. The verification is performed locally. In one embodiment, a verification process is performed.

FIG. 11Bis a flow chart of a verification process1150in accordance with one embodiment. The automated verification process1150can be triggered by a variety of events or conditions that can be used to trigger the automated verification. The verification can be triggered based on time periodically (e.g., every day, week, year, etc.). The verification can be based on usage, such as after a certain number of UE devices under test have been tested, before certain particular types of tests are performed (an AT command test, application retry test etc.), and so on. The verification can be based on analytics of testing results. In one embodiment, if a local test system begins to indicate issues with a threshold or number of UE devices under test, a verification process can be triggered to check to make sure the local test system is working correctly and not accidentally finding issues with correctly operating UE devices under test.

The validation process can be triggered by a remote management system. In one embodiment, the remote management system performs a variety of different analytics that may indicate a validation process is appropriate. In one exemplary implementation, information from local test system interactions with a reference device is reported back to the remote management system. The information from UE testing can also be reported back to the remote management system.

Based on analytics performed by the remote management system new or additional validation operations can be triggered. In one exemplary implementation, the remote management system collects information from various different sources regarding UE testing (e.g., from OEMs, from other local test systems, etc.) and if a particular local test system UE test results are outside a norm or threshold based on the remote management system information and analytics, the remote management system can trigger a validation process for the particular local test system.

In block1151, verification interactions between a reference device and a local test system are performed. The verification interactions are directed to verifying operations of the local test system. The verification interactions can be performed in response to an automatic triggering event. The verification interactions can include verifying communication signal characteristics, simulated component operations (e.g., simulated communication network components, other simulated components, etc.) and so on.

In block1152, results of the verification interactions are reported. The results of the verification interactions can include indications of acceptability and problems with the operations being verified. The results can be reported to the local test system. In one exemplary implementation, the results are reported to a remote management system.

In block1153, the local test system is calibrated. The calibration can be based upon the verification results. In one embodiment, the calibration is directed to correcting (e.g., adjusting signal power, frequency, signal shape, etc.) issues in the verification results. In one exemplary implementation, the calibration is performed in accordance with information received from a remote management system.

In one embodiment, the automated local test system configuration ensures that the user is able to easily setup and operate the local test system. The configuration automatically installs pre-requisite software, a test executive, test cases, UE library and reporting components. Various communication network simulated components are automatically installed and configured (e.g., eNodeB, EP, IMS server and other application servers for a given configuration, etc.). Software component can be automatically initialized. Validation and calibration can be performed, including using test case routines, to verify successful operation of the local test system. Again, it is appreciated that automated configuration of local test systems is user friendly and convenient, unlike traditional configuration approaches that typically involve significant manual interaction.

In one embodiment, a local test system and method automated configuration call flow includes an initialization process, retrieving information from a remote management system, performing a verification/calibration process, and so on. In one embodiment, an initialization process includes activating the local control component, receiving information from a remote management system, installing and launching local control component modules based on the received information. Retrieving information from a remote management system can include: signing on/registering account with the remote management system, downloading pre-requisites and configuration wizard information, installing virtual box applications (e.g., eNodeB, EPC, and other pre-requisites, etc.), automatically configuring the username/password, set host and confirm host status IP address. Installing and launching includes the QuiNS, Test Executive modules and simulated network components (e.g., EPC, network core components, etc.), An IMS and other application servers can be launched and verified. The eNodeB is powered up and QuiNS connects to eNodeB and verifies successful connectivity.

In one embodiment, a local test system and method automated configuration includes a validation and self calibration process. The validation and self calibration process can include communicative coupling of a reference component to a local test system environment. In one embodiment, the reference component is similar to reference component1050. The validation and calibration process can be initiated or started automatically or manually. The local test system initializes itself, including initialization of servers (e.g., IMS, FTP, etc.). The local test system can include a local test control component and a local test user equipment interface component). In one exemplary implementation, the local test system retrieves device profile information for a reference component or reference UE. The local test system reboots the reference component and monitors network to verify successful reference component or device attachment. The IMS is monitored for successful registration. The local test system verifies several operations or activities. The operations and activities can include verification of successful mobile originated and mobile terminated SMS operations, successful data throughput operations, and various RF signal strengths. The verification can be repeated at multiple signal strength levels. Verification results can be recorded locally for each level. In one exemplary implementation, RF signal strength loses can be reported for each RF strength level. The results can also be reported externally or remotely. In one embodiment, the results can be reported to remote management environment via a variety of mechanisms. Calibration can be performed based on the verification results. The calibration can include application of proper compensations or offsets to correct issues identified during verification.

While the local testing system and method operations are automated, it is appreciated local testing systems and methods can be readily adapted to or implement varying degrees of manual interaction. In one embodiment, a UE test or local test system validation can be triggered or initiated by manual inputs and the remaining associated test or validation operations performed automatically. In one exemplary implementation, a local test system includes a local user interface. The local user interface can include a presentation of testing information (e.g., local test system configuration information, UE device under test information, testing result information, result information associated with verification/calibration of the local test system, etc.). A reference component can include a user interface for conveying verification/calibration related information and receiving user input.

A remote management system can include a remote user interface. The remote user interface can include a presentation of testing information (e.g., local test system configuration information, UE device under test information, testing result information, result information associated with verification/calibration of the local test system, etc.). In one embodiment, automated aspects of the remote management system, the remote user interface, or combinations of both can remotely monitor or take remote control of local test system and method operations.

FIG. 12is a diagram illustrating an example of certification testing of SMS over IMS capabilities using a local test system. In this example, the UE is a smartphone reference component and the testing is used as part of a validation process. It is appreciated that similar steps can be use to test regular user equipment. The local control component sends a “turn airplane mode on” command to the UE and then the local control component sends a “turn airplane mode off” command to the UE. The UE accomplishes LTE detachment followed by a LTE attach. The local control system verifies the LTE attachment. The UE registers with an IMS server. In one embodiment, the IMS server is simulated on a local computer system. The local control component instructs the base station to verify successful IMS registration. The simulated base station sends Mo 3GPP2 SMS to the reference UE. The base station “sends” an indication to the IMS server that verifies if the SMS was sent successfully. The local control component sends a MT 3GPP2 SMS to the UE and then verifies if UE successfully receives MT SMS.

FIG. 13is a diagram illustrating an example of certification testing of AT command capabilities using a local test system. In this example, the UE is a smartphone. The local control component sends a “turn airplane mode on” command to the UE and then the local control component sends a “turn airplane mode off” command to the UE. The UE accomplishes LTE detachment followed by LTE attach. At a forth operation, the local control component instructs the simulated base station to verify successful LTE attachment. The UE registers with an IMS server. In one embodiment, the IMS server is simulated on a local computer system. The local control component instructs the base station to verify successful IMS registration. In one operation, the local control component sends an AT command (plus COPN) to the UE to get operator names. In another operation, the local control component sends an AT command (plus CGPIAF) to the UE to print the IP address format. The local control component can also send an AT command (plus CGPIAF) to the UE to modify the IP address format.

FIG. 14is a diagram illustrating an example of RCS testing using a local test system. The local control component instructs the UE devices A, B, C, and D to turn Airplane mode. Then for each UE device the local control component forwards instructions including: instructing the respective UE to turn the Airplane mode off, instructing the simulated base station to verify successful LTE Attach for the respective UE device, and verify IMS Registration with the simulated reference client/IMS server for each respective UE. Then UE device A calls UE device B, UE device C calls UE device A, and UE device D calls UE device B. The local control component verifies SIP Invite from A to B, SIP Invite from C to A, and continues to monitor SIP events for an Appropriate Sequence. Device B swaps to A, device C swaps to A, and device-D swaps to B. Then the calls on devices A, B, C, and D are ended.

FIG. 15is a diagram illustrating an example of a test for different scenarios using a local test system. In this example, the UE is an IoT device. The local control component instructs the simulated base station to configure an LTE band and to n to configure an Internet path. The UE is powered on by the local control component in a third operation. If applicable, the UE accomplishes LTE attach and registration with an IMS server (not shown) in a forth operation. A path to the IoT server502(which may or may not be one of the servers102a-102nofFIG. 1) is established by the UE. The local control component can instruct the simulated base station to monitor IP connectivity and pattern. The UE auto-registers with the IoT server and the local control component instructs the simulated base station to monitor registration activity and pattern.

FIG. 16is a diagram illustrating an example of a test for another application retry using a local test system. In a first test scenario, the local control component directs the simulated base station to vary the RF condition. The UE device reacts to the RF variance and sends an indication to the local control component. The local control component monitors the UE device behavior.

In a second test scenario, the local control component directs the simulated base station to simulate an IP connection loss. The UE device reacts to the IP connection loss and sends an indication to the local control component. The local control component monitors the UE device behavior.

In a third test scenario, the local control component directs the simulated base station to simulate a SMS wakeup. The UE device reacts to the SMS wakeup and connects to the IoT server. The local control component monitors the UE device behavior.

While users may have varying degrees of understanding of normal communication networks, the automated configuration capabilities and features of the local test systems and methods enables the test systems and methods configurations to be performed by users with little or no manual interfacing or understanding of how the test systems and methods themselves work. Unlike traditional systems that typically require very sophisticated users that have a thorough understanding of intricate internal workings of the numerous components in a traditional complex test system itself (in addition to the vast different types of complicated network communication architectures and protocols involved in the testing), local test systems and methods facilitate easy configuration from a user standpoint.

Many of the described examples and embodiments of the local flexible test systems and methods are described in terms of single complex communication architectures in order not to obfuscate the invention. It is appreciated that some embodiments of the local flexible test systems and methods can be readily expanded to handle much more complicated and complex testing communication architectures and environments. In one embodiment, multiple communication core architectures can be simulated. In one exemplary implementation, configuration of the local flexible test systems can be expanded to test network communications as user equipment travels from an EPC core network to a GSM network.

Automated testing is flexibly scalable to large numbers of different devices and can be accomplished quicker, more systematically, and at less expense than, for example, manual testing. This is turn can increase test coverage, scalability and reliability while reducing time-to-market, and the cost to both manufacturers and consumers.

Some portions of the detailed descriptions are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means generally used by those skilled in data processing arts to effectively convey the substance of their work to others skilled in the art. A procedure, logic block, process, etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps include physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, optical, or quantum signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present application, discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “displaying” or the like, refer to the action and processes of a computer system, or similar processing device (e.g., an electrical, optical or quantum computing device) that manipulates and transforms data represented as physical (e.g., electronic) quantities. The terms refer to actions and processes of the processing devices that manipulate or transform physical quantities within a computer system's component (e.g., registers, memories, other such information storage, transmission or display devices, etc.) into other data similarly represented as physical quantities within other components.