Patent Publication Number: US-2019182134-A1

Title: Cable modems/emtas under test

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application entitled: CABLE MODEMS/EMTAS UNDER TEST, Ser. No. 15/624,950, filed Jun. 16, 2017 and published on Oct. 5, 2017 as U.S. Patent Publication No. 2017/0288993, which is a continuation of U.S. patent application entitled: CABLE MODEMS/EMTAS UNDER TEST, Ser. No. 14/948,143, filed Nov. 20, 2015, now U.S. Pat. No. 9,992,084, which is hereby incorporated by reference in its entirety. 
     This application is related to U.S. patent application entitled CORE TESTING MACHINE, Ser. No. 14/866,720, filed Sep. 25, 2015; now U.S. Pat. No. 9,810,735, and to U.S. patent application entitled UNIVERSAL DEVICE TESTING INTERFACE, Ser. No. 14/866,752, filed Sep. 25, 2015 and published on Mar. 30, 2017 as U.S. Patent Publication No. 2017/0093683; and to U.S. patent application entitled UNIVERSAL DEVICE TESTING SYSTEM, Ser. No. 14/866,630, filed Sep. 25, 2015 and published on Mar. 30, 2017 as U.S. Patent Publication No. 2017/0093682; and to U.S. patent application entitled SET TOP BOXES UNDER TEST, Ser. No. 14/866,780, filed Sep. 25, 2015, now U.S. Pat. No. 9,491,454; and to U.S. patent application entitled HARDWARE ARCHITECTURE FOR UNIVERSAL TESTING SYSTEM: CABLE MODEM TEST, Ser. No. 14/929,180, filed Oct. 30, 2015 and published on May 4, 2017 as U.S. Patent Publication No. 2017/0126536; and to U.S. patent application entitled: HARDWARE ARCHITECTURE FOR UNIVERSAL TESTING SYSTEM: WIRELESS ROUTER TEST, Ser. No. 14/929,220, filed Oct. 30, 2015 and published on May 4, 2017 as U.S. Patent Publication No. 2017/0126537, each of which is hereby incorporated by reference in its entirety. This application is also related to U.S. patent application entitled WIRELESS ROUTERS UNDER TEST, Ser. No. 14/948,925, filed Nov. 23, 2015, now U.S. Pat. No. 9,838,295, and to U.S. patent application entitled TEST SEQUENCES USING UNIVERSAL TESTING SYSTEM, Ser. No. 14/987,538, filed Jan. 4, 2016 and published on Jul. 6, 2017 as U.S. Patent Publication No. 2017/0195071. 
    
    
     TECHNICAL FIELD 
     The present invention is directed to a system for testing devices. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       For a better understanding of the aforementioned aspects of the invention as well as additional aspects and embodiments thereof, reference should be made to the Description of Embodiments below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures. 
         FIG. 1  illustrates a high-level system architecture for testing devices, according to certain embodiments. 
         FIG. 2  illustrates some of the testing components and the interaction between the testing components, according to certain embodiments. 
         FIG. 3  illustrates a sample architecture that includes the testing components, according to certain embodiments. 
         FIG. 4  illustrates a cable modem/eMTA device under test, according to certain embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     Methods, systems, user interfaces, and other aspects of the invention are described. Reference will be made to certain embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with the embodiments, it will be understood that it is not intended to limit the invention to these particular embodiments alone. On the contrary, the invention is intended to cover alternatives, modifications and equivalents that are within the spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. 
     Moreover, in the following description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these particular details. In other instances, methods, procedures, components, and networks that are well known to those of ordinary skill in the art are not described in detail to avoid obscuring aspects of the present invention. 
     According to certain embodiments, an innovative system can test a set of devices simultaneously. Further, such a testing system is capable of testing disparate devices simultaneously. 
     According to certain embodiments, such a testing system provides a separate set of interfaces for each device that is under testing of the set of devices. Further, such a system is designed to be adaptive by being extendable for testing new devices with corresponding new testing interfaces without fundamentally changing the core architecture of the testing system. As a non-limiting example, the testing system includes a core testing subsystem with a user interface and asynchronous communication among the system components such that new types of devices and new tests can be added and executed in a seamless fashion. 
     According to certain embodiments, the user interface can communicate through web sockets with the universal tester. Such communication is in real-time, bi-directional and asynchronous so that the user can control and monitor the testing of multiple devices simultaneously and independently of each other using the same universal tester and its associated test bench. 
     According to certain embodiments, the testing system is capable of testing a set of similar types of devices or a set of disparate devices. 
     According to certain embodiments, a testing solution system can be a three layer implementation. The number of layers may vary from implementation to implementation.  FIG. 1  illustrates a high-level system architecture for testing devices, according to certain embodiments.  FIG. 1  shows a test bench browser interface  102  that is in communication with a web-socket  104 , that is, in turn, in communication with a core testing processor  106 . According to certain embodiments, the communication between the test bench browser  102 , web-socket  104  and core testing processor  106  can be a TCP/IP communication. As a non-limiting example, the web browser is used as a user interface that communicates through web-sockets with the core testing processor. As a non-limiting example, communication may be in the form of JSON messages using TCP/IP protocol, according to certain embodiments. JSON is Java script object notation for transmitting data between the server and web applications. 
       FIG. 2  illustrates some of the testing components and the interaction between the testing components, according to certain embodiments.  FIG. 2  shows a user interface  202 , web-sockets  204 , a core testing processor  206 , database  208 , test configuration modules  210 , testing environment modules  212 , a plurality of probes ( 214 ,  216 ,  218 ) to connect the devices under test (DUT) to the core testing processor  206 , and a speed test module  220 , according to certain embodiments. Speed testing is used for evaluating the performance of the Wifi and other media network connection and accessibility of the device under test.  FIG. 2  shows as non-limiting examples, a Wifi probe  214 , an Ethernet local area network (LAN) probe  216  and a MOCA probe  218 . In other words, according to certain embodiments, various probes can be included such as a wireless local area network (WLAN) probe, an Ethernet wide area network (WAN) probe, a multimedia over coax alliance (MoCA) WAN probe, a MoCA LAN probe and a wireless probe via antenna. According to certain embodiments, servers and other components in the testing system may be distributed over a plurality of computers. 
     According to certain embodiments, core testing processor  206  loads and reads files from test configuration modules  210  and test environment modules  212  to initialize various components of the testing system. When the system is ready to begin testing after the initialization process, the system notifies a user that is using the testing system to test one or more devices (DUTs) of the readiness of the testing system. The user installs each device or DUT (of the set of DUTs that are to be tested) in a separate Faraday cage (slot) in the test bench and the serial number of each DUT is scanned. According to certain embodiments, there are several Faraday cages (slots) in a given test bench so that a plurality of DUTs can be tested simultaneously using the same test bench and same universal tester. The core testing processor  206  receives the serial number information of each DUT and using the serial number, retrieves further information associated with each DUT based on the serial number from database  208 , according to certain embodiments. The core testing processor  206  dynamically loads test configuration information  210  and test environment information  212  based on device information such as make, model etc of a given DUT. After the test configuration and test environment information are loaded, the core testing processor  206  begins executing the various tests corresponding to each DUT so that the set of DUTs can be tested simultaneously. Each test may correspond to underlying testing modules associated with Wifi, LAN, WAN or MoCA etc, interfaces of the DUT and such modules can be executed locally, remotely or at the device. 
     According to certain embodiments, the test configuration information identifies the test modules and corresponding testing scripts that are to be executed by the core testing processor  206  at run time. The core testing processor  206  also provides the test results and other feedback information to the user via the browser user interface  202  and web sockets  204 . Further, the user can send user input and requests to the system through the browser user interface  202  and web sockets  204 . 
     According to certain embodiments, core testing processor  206  determines the success or failure of a given test based on the test configuration parameters and output results of the testing. Further, upon failure of a given test, core testing processor  206  may continue further testing or halt test execution based on test configuration parameters, according to certain embodiments. 
     Upon completion of the relevant tests, a success message can be sent to the user via the browser user interface  202  and web sockets  204 . Even though the DUTs in the set of DUTs are tested simultaneously, the user does not have to wait until all the testing of the DUTs in the set have been completed to begin installing other devices that need testing. Further, the testing of the devices need not be started at the same time. Soon after the testing is completed for a given DUT, the tested DUT may be uninstalled from its slot (Faraday cage) in the test bench and a new DUT can be installed in its slot so that testing can begin for the newly installed device. 
     According to certain embodiments, the test results can be stored locally and/or pushed to the cloud so that the results can be viewed remotely from any location. Further, the test results can be aggregated. According to certain embodiments, aggregated data includes data combined from several data measurements. Summary reports can be generated from such aggregated data. Non-limiting examples of summary reports include charts and graphs that display information on all the DUTs or at least a subset of the DUTs. Thus, the summary reports generated from the aggregated data can provide an overview of the testing information and characteristics of the DUTs. The aggregated data can reveal trends and other related information associated with the DUTs. Further, the aggregated data can include user-level data, access account activity, etc. According to certain embodiments, the testing system includes a billing system to charge for the testing services for each device. 
       FIG. 3  illustrates a sample architecture that includes the testing components of a universal tester, according to certain embodiments.  FIG. 3  shows a browser user interface or operator dashboard  302 , a test controller  304 , a universal tester  306  and a device under test (DUT)  308 . There may be multiple devices under testing simultaneously but only one device under test is shown for convenience in  FIG. 3 . 
     According to certain embodiments, browser user interface or operator dashboard  302  may include information  310  associated with each device under test. The information  310  can include DUT serial number  311 , and testing progress information  312 . Browser user interface or operator dashboard  302  may also include user command function buttons  314  and drop down menus (not shown in  FIG. 3 ). According to certain embodiments, the user can configure slot details (e.g., port numbers, IP address for the slot, etc), configure testing preferences such as push to cloud, export to billing, etc. 
     According to certain embodiments, test controller  304  may include a universal tester webserver  316  that is in communication (e.g., TCP/IP) with a universal tester database  318 . A billing process within the controller (not shown in  FIG. 3 ) may be in communication with a billing service or application (not shown in  FIG. 3 ). As a non-limiting example, database  318  can be a SQL database. Database  318  can store information associated with each slot in the test bench. As non-limiting examples, database  318  can store for each slot, test details, test history, test logs, DUT information (e.g., DUT serial number, model name, etc), testing preferences/configuration, user interface details/preferences/configuration, billing information, cloud push information, MSO/customer information (media subscriber organization), OEM (original equipment manufacturer) information, slot information, user information, and any persistent data needed by the universal device testing system for running tests. 
     According to certain embodiments, universal tester  306  may include web sockets  320  that are in communication (e.g., TCP/IP) with browser user interface or operator dashboard  302  and core testing processor  324 . According to certain embodiments, core testing processor  324  is in communication with test controller  304  (e.g., TCP/IP) and in communication (e.g., Telnet/SSH secure shell) with probes/containers ( 328 ,  330 , . . . ,  332 ,  334 ). Core testing processor  324  is also in communication with configuration modules  322  (e.g., testing and environment configuration). Non-limiting examples of probes include Wifi probe  328 , LAN probe  330 , MoCA probe  332  and WAN probe  334 . There may be other types of probes including MoCA WAN probe, MoCA LAN probe and other types of wireless probes besides Wifi probes depending on the characteristics of the device being tested. 
     According to certain embodiments, Wifi probe  328 , LAN probe  330 , MoCA probe  332  and WAN probe  334  communicate with the respective device under test through the relevant ports on the device such as Wifi port  336 , LAN port  338 , MoCA port  340  and WAN port  342 . Core testing processor  324  executes the relevant configured tests for the respective DUT. Status and test results can be sent to the user&#39;s dashboard (using JSON format messages as a non-limiting example) via the web-sockets. 
     Non-limiting examples of devices under test (DUTs) include set top boxes, cable modems, embedded multimedia terminal adapters, and wireless routers including broadband wireless routers for the home or for commercial networks. 
       FIG. 4  illustrates a testing architecture  400  for a cable modem/eMTA under test, according to certain embodiments. As previously explained, multiple similar or disparate devices can be tested simultaneously and independently of each other using the same universal tester. Thus, multiple cable modems/eMTAs can be tested simultaneously and independently of each other using the same universal tester, along with other types of devices using the same universal tester. For purposes of simplicity only one cable modem/eMTA is shown in  FIG. 4 .  FIG. 4  shows a universal tester  404  and cable modem/eMTA (embedded multimedia terminal adapter)  402 , which is the device under test for this specific case. Universal tester  404  includes a plurality of virtualization containers (probes) for communicating with corresponding interfaces of cable modem/eMTA  402 . For example, the core testing processor of the universal tester (as described herein) uses the LAN probes/containers  406   b,    408   b,    410   b,    412   b  to test corresponding LAN interfaces  406   a,    408   a,    410   a,    412   a  of cable modem/eMTA  402 . Similarly, FXS (foreign exchange station) probe/container  414   b  can be used to test the FXS interface  414   a  of cable modem/eMTA  402 . WLAN (wireless LAN) probe/container  416   b  can be used to test WLAN interface  416   a  of cable modem/eMTA  402 . MoCA LAN probe/container  418   b  can be used to test MOCA LAN interface of cable modem/eMTA  402  via MoCA LAN bridge  418   a  and splitter  420 . NCS (network based call signaling protocol specification)/IMS (IP multimedia subsystem) probe/container  424  can be used to test DOCSIS WAN/MOCA LAN interface  430  of cable modem/eMTA  402  via CMTS (cable modem termination system)  422  and splitter  420 . Provision probe/container  426  can be used to test DOCSIS WAN/MOCA LAN interface  430  of cable modem/eMTA  402  via CMTS  422  and splitter  420 . WAN probe/container  428  can be used to test DOCSIS WAN/MOCA LAN interface  430  of cable modem/eMTA  402  via CMTS  422  and splitter  420 . The associated core testing processor executes the relevant configured tests for the cable modem/eMTA  402 . Status and test results can be sent to the user&#39;s dashboard (using JSON format messages as a non-limiting example) via the web-sockets. 
     According to certain embodiments, when executing a specific test for a given DUT, the core testing processor loads and reads test configuration information (for example from an XML structure) and identifies the relevant test script that needs to be executed. Inputs that are needed for executing the relevant test script are retrieved and supplied as inputs to the relevant test script. The following is a non-limiting sample test procedure.
     Create DUT object &amp; Environment Object   Verify Serial Number   Verify Warranty   Check Report Server   Check DUT Staging   

     Checks for DUT Serial number in Database or Webservice
     Get DUT Readiness Information   

     Checks Web-service for test readiness status of DUT in the test process
     Configure Container Environment   Clear Environment Temp Files   Confirm Factory Reset   

     Waits for operator to confirm that DUT was factory reset and booted up properly
     Check Ethernet LAN connections to DUT   

     Ping connections: Eth LAN 1, 2, 3, 4 
     Fails if any ping to these connections fail
     Detect DUT   

     Checks connection to DUT through socket connection
     Reset Password   

     Operator scans password which is stored temporarily for use in the remainder of test until finished
     Login to GUI   

     Done through web-scraping
     Get DUT Information and compare values   

     Information retrieved through web-scraping
     Confirm Power   Confirm all LAN Ethernet LEDs   Confirm WiFi LED   Configure Wireless Network   

     Through telnet commands 
     Sets N Mode 
     Enables Privacy 
     Sets WPA (Wi-Fi Protected Access) 
     Removes WEP (Wired Equivalent Privacy) 
     Assigns WiFi Channel to DUT (channel different by slot) 
     [Channel 1: slots 1, 4, 7, 10, 13, 16] 
     [Channel 6: slots 2, 5, 8, 11, 14] 
     [Channel 11: slots 3, 6, 9, 12, 15] 
     Verifies changes through GUI 
     Disables WiFi once done through telnet
     Check Firmware Version and Upgrade Firmware (if needed)   

     Firmware version varies by model
     Cage Closed Confirmation Check   

     Asks Operator to Close Door on Cage
     Connect Wireless Card   

     Waits on shared Resource Server (located on TC) for Resource L2 (Layer 2 ) Lock
         Lock waiting timeout: 600 sec   All L2 Locks are able to run in parallel but not when any L3 (Layer 3) Lock is running       

     Obtains Lock 
     Enables WiFi through telnet 
     Set WiFi Card
         Total Retries allowed: 6 (2 sets of 3 retries)       

     Ping WiFi from DUT 
     L2 ARP Test on WiFi: must receive 10/10 ARP packets
         Total Retries allowed: 6 (2 sets of 3 retries)       

     If either Set WiFi Card or L2 ARP Test Fail after its 3 retries, Ask Operator to Check Antennas 
     Performs one more retry in full (set of 3 retries each for Set WiFi Card and L2 ARP Wifi Test) after Check Antennas 
     Disables WiFi through telnet 
     Releases Lock
     Wireless to LAN Ethernet Speed Test   

     Waits on shared Resource Server (located on TC) for Resource L3 Lock
         Lock waiting timeout: 1800 sec   L3 Locks must be run one at a time and when no L2 Lock is running       

     Obtains Lock 
     Enables WiFi through telnet 
     Connects WiFi Card 
     Iperf3 Speed Test, 5 seconds for UDP Speed Test, 7 seconds for TCP Speed Test, Sending 200 Mbps Bandwidth 
     Bandwidth must be greater than 60 Mbps on TCP (Reverse) or 70 Mbps on UDP (Forward)
         If Fail after 2 retries, ask operator to Check Antennas   Retries up to 2 times more if still Fail   Therefore, Total Retries allowed: 4 (2 sets of 2 retries)       

     Runs sudo iwlist wlan0 scan and returns all Wireless Signals seen
         Results parsed to print all visible SSIDs and its matching Signal level       

     Disables WiFi through telnet 
     Releases Lock
     Confirm WPS LED   Confirm LAN Coax LED   Confirm USB 1+2 LEDs   Confirm US/DS LEDs (upstream/downstream LEDs)   Confirm Online LED   Confirm Telephone LEDs   L2 Test on LAN Ethernet   

     Arp Test from Eth LAN 1 to Eth LAN 2, 3, 4 
     Must receive 10/10 on all LAN connections
     LAN Ethernet to LAN Ethernet Speed Test   

     From Eth LAN 1 to Eth LAN 2, 3, 4 
     Iperf3 Speed Test, 5 seconds Reverse and Forward, (e.g., Sending 1200 Mbps Bandwidth) 
     Bandwidth must be greater than threshold (e.g., 700 Mbps) Threshold may vary depending on the model 
     Total Retries allowed: 2
     LAN MoCA to LAN Ethernet FTP Speed Test   

     From Eth LAN 1 to LAN MoCA 
     Iperf3 Speed Test, 5 seconds Reverse and Forward,(e.g., Sending 240 Mbps Bandwidth) 
     Bandwidth must be greater than threshold (e.g., 60 Mbps) Threshold values may vary depending on the model 
     Total Retries allowed: 2
     LAN Ethernet to WAN DOCSIS FTP Speed Test   

     From Eth LAN 1 to DOCSIS WAN 
     Iperf3 Speed Test, 5 seconds Reverse and Forward, (e.g., Sending 1200 Mbps Bandwidth) 
     Bandwidth must be greater than threshold (e.g., 700 Mbps) Threshold values may vary depending on the model 
     Total Retries allowed: 2
     Voice Testing   

     Test capability to support incoming and outgoing calls 
     Test capability to support 2-port, 4-port, and 8-port 
     Test capability to support NCS and DIP protocols
     Clear Persistent Logs   Final Factory Restore   

     According to certain embodiments, the core testing processor uses a reflection and command design pattern to invoke the relevant configured script(s) corresponding to each DUT being tested. For example, in the command design pattern one or more of the following are encapsulated in an object: an object, method name, arguments. According to certain embodiments, the core testing processor uses the Python “reflection” capability to execute the relevant test scripts for a given DUT. The core testing processor is agnostic of the inner workings of the relevant test scripts for a given DUT. 
     According to certain embodiments, lightweight software containers (virtualization containers) are used to abstract the connection of probes to the different DUT interfaces in order to avoid conflicts. Non-limiting examples of virtualization containers are Linux containers. As a non-limiting example, Linux container is an operating-system-level virtualization environment for running multiple isolated Linux systems (virtualization containers) on a single Linux control host. In other word, lightweight virtualization containers are used to ensure isolation across servers. By using virtualization containers, resources can be isolated, services restricted, and processes provisioned to have an almost completely private view of the operating system with their own process ID space, file system structure, and network interfaces. Multiple virtualization containers share the same kernel, but each virtualization container can be constrained to only use a defined amount of resources such as CPU, memory and I/O. The relevant test script connects to the DUT interfaces through the virtualization containers to execute the tests. The core testing processor receives the test results from miming the relevant test scripts. The core testing processor can further process and interpret such results and can also send the results to the user&#39;s browser via web sockets. According to certain embodiments, the respective core testing processors are in communication (e.g., Telnet/SSH secure shell) with the virtualization containers (there may be multiple virtualization containers). The virtualization containers (probes) are in communication with corresponding DUT interfaces using Telnet/SSH/TCP/UDP/HTTP/HTTPS etc, as non-limiting examples. 
     According to certain embodiments, a system for testing a plurality of devices comprises: a universal tester; at least one test controller; a plurality of sets of testing probes; and a plurality of web sockets; wherein: 
     the plurality of devices includes a plurality of cable modems/eMTA devices; 
     the universal tester is enabled for communication with a platform independent user interface through the plurality of web sockets; 
     the plurality of sets of testing probes comprising: 
     a plurality of LAN probes for testing corresponding LAN interfaces of a cable modem/eMTA device of the plurality of cable modem/eMTA devices; 
     at least one FXS probe for testing a corresponding FXS interface of the cable modem/eMTA device of the plurality of cable modem/eMTA devices; 
     at least one WLAN probe for testing a corresponding WLAN interface of the cable modem/eMTA device of the plurality of cable modem/eMTA devices; 
     at least one MoCA LAN probe for testing a corresponding MoCA LAN interface of the cable modem/eMTA device of the plurality of cable modem/eMTA devices; 
     at least one NCS/IMS probe for testing a corresponding DOCSIS WAN interface of the cable modem/eMTA device of the plurality of cable modem/eMTA devices; 
     at least one provision probe for testing the corresponding DOCSIS WAN interface of the cable modem/eMTA device of the plurality of cable modem/eMTA devices; 
     at least one WAN probe for testing the corresponding DOCSIS WAN interface of the cable modem/eMTA device of the plurality of cable modem/eMTA devices; and 
     the plurality of web sockets enable real-time bi-directional and asynchronous communication between the platform independent user interface and the universal tester for simultaneously testing the plurality of devices under test by the universal tester. 
     According to certain embodiments, the system for testing a plurality of devices further comprises a MoCA LAN bridge. 
     According to certain embodiments, the system for testing a plurality of devices further comprises a cable modem terminal system (CMTS). 
     According to certain embodiments, the system for testing a plurality of devices further comprises a splitter. 
     According to certain embodiments, the real-time bi-directional and asynchronous communication of the plurality of web sockets enable a user to control the testing of the plurality of devices simultaneously but asynchronously and independently of each other using the universal tester. 
     According to certain embodiments, the plurality of devices installed in the universal tester for purposes of simultaneous testing comprise a set of disparate devices. 
     According to certain embodiments, the plurality of devices installed in the universal tester for purposes of simultaneous testing comprise a set of similar devices. 
     According to certain embodiments, the testing system is adaptable to augmenting the test controller, the plurality of web sockets, the user interface and the plurality of sets of testing probes to accommodate testing of new types of devices. 
     In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.