Patent ID: 12254665

DETAILED DESCRIPTION

The present teachings are described more fully hereinafter with reference to the accompanying drawings, in which the present embodiments are shown. The following description is presented for illustrative purposes only and the present teachings should not be limited to these embodiments. Any system configuration, device configuration, or processor configuration satisfying the requirements described herein may be suitable for implementing the system and method to validate task completion of the present embodiments.

For purposes of explanation and not limitation, specific details are set forth such as particular structures, architectures, interfaces, techniques, etc. in order to provide a thorough understanding. In other instances, detailed descriptions of well-known devices and methods are omitted so as not to obscure the description with unnecessary detail.

Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to a/an/the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated. The use of “first”, “second,” etc. for different features/components of the present disclosure are only intended to distinguish the features/components from other similar features/components and not to impart any order or hierarchy to the features/components.

Referring to the figures in detail and first toFIG.1, a system and method are provided to detect LEDs and their states in an augmented reality context. The system includes a computer1, a device4, and a database5.

The computer1may include a processor, computer, remote computer, computer server, network, or any other computing resource. The computer1may include memory, input/output devices, storage, and communication interfaces—all connected by a communication bus. The storage may store application programs and data for use by the computer1. Typical storage devices include hard-disk drives, flash memory devices, optical media, network and virtual storage devices, and the like. The communication interfaces may connect the system to any kind of data communications network, including either wired networks, wireless networks, or a combination thereof. The memory may be random access memory sufficiently large to hold necessary programming and data structures of the disclosed subject matter. The memory may constitute a single entity or comprise a plurality of modules. The input/output devices may be used to update and view information stored in a database5, as described later in more detail

The computer1may be in communication with the mobile device2. The mobile device2may be a computer, desktop computer, laptop, smartphone, tablet, or other electronic device capable of transmitting data to or receiving data from the computer1. The mobile device2may include a processor, computer, remote computer, computer server, network, or any other computing resource.

The system may be configured to communicate through a network with the mobile device2that is associated with a user who is using the system. The network may comprise a local-area network (LAN), a wide-area network (WAN), metropolitan-area network (MAN), and/or the Internet, and further may be characterized as being a private or public network. The mobile device2may be a mobile phone, smart glasses, AR/VR glasses, personal digital assistant, tablet, laptop, or the like. However, in other embodiments, the mobile device2may be a non-mobile device, for example a desktop computer. The mobile device2contains an application(s) which, when executed by a processor of the mobile device2, delivers information from the system to the user and provides real-time guidance. The application(s) of the mobile device2generates graphical user interfaces for presenting information regarding the system, as discussed in more detail below, and facilitates user interaction with the graphical user interfaces(s) as described herein.

The mobile device2may have a camera3. The camera may be capable of taking still photographs or video. For example, the camera3may include a CCD sensor, a CMOS sensor, or a combination of both. The camera3is adapted to capture at least one static image, or a video (a plurality of moving images), or a combination of both of the monitored device4. The mobile device2may transmit the images31from the camera to the computer1. The transmission may be real-time or may be delayed. Alternatively, the mobile device2may be a camera3, and the images31may be stored in memory before being accessed for use as part of the system.

The camera3is used to take images31of a device4. Device4may be any type of device with light indicators41,42,43, etc. Light indicators41,42,43, etc., are preferably LEDs, given their incorporation into many devices today. For example, device4may be a modem, router, printer, laptop, refrigerator, cable (such as CAT5) or any other device that uses LEDs to communicate status. The number of LED indicators on the device may vary.

Each LED indicator may correspond to a state of the device. For example, an LED may show if a device is on or off. Some LEDs may blink to communicate a state. Some LEDs may change colors to communicate a state. Combinations of LED states may indicate a condition of the device4. For example, where the device is a router, an LED may be used to show an ON/OFF state, whether the router is transmitting/receiving, whether the router is connected to the internet, or whether the router has devices connected.

The computer1may also be in communication with a database5. The database5may store information regarding the system. The database5may be a storage drive or array accessible to computer1, or cloud storage. The database5may be integrated into the computer1, or the mobile device2. The database5may store reference images with reference feature points and reference zones51, condition tables52, and troubleshooting steps53.

The computer1receives an image31taken by camera3. This may be a still photograph or a frame of a video. The image31may be received over a network, such as the Internet, a WAN, a LAN, Bluetooth, or any other known network. The image31may be received via a memory card or USB memory device. Any receiver may forward the image31to the computer1.

The computer1may use deep learning to recognize objects in the image31. The deep learning may comprise a machine learning processor, a deep learning accelerator, an AI accelerator, and/or neural processor. The deep learning may be built on machine learning and/or artificial intelligence and may be based on artificial neural networks with representation and/or reinforcement learning. In some embodiments, the deep learning is a separate deep learning machine that uses neural networks. In some embodiments, the deep learning module may comprise computer instructions executable on a processor for determining an object. The deep learning module may include computer vision (with pattern recognition capability, for example) and may be trained to visually detect objects/components of the monitored device and their respective states and conditions from the image(s)31. In addition, or alternatively, the deep learning may be trained to visually detect the monitored device as a whole and determine its state or condition from the image(s)31.

The deep learning is configured to analyze the image31, specifically identifying an object of the monitored device4within the image(s)31and detecting a state of the object by comparing the image(s)31to at least reference data51. For example, in systems that use homography-based image alignment and registration, the database5may store reference images with reference feature points and reference zones. The reference data51may include predetermined labels and bounding boxes for an object of a monitored device4. To generate reference images with reference feature points and reference zones51, a picture is taken from a detected object bounding box to configure the detection parameters. Parameters may be generated to indicate where objects are in the sub-image defined by the bounding box. Feature points51may be extracted and saved from the sample image using a known algorithm, for example SIFT or SuperPoint.

In such a system, the computer1may retrieve reference images with reference feature points and reference zones51from the database5. A sample reference image is shown inFIG.4. The reference images with reference feature points and reference zones51may include predetermined labels and bounding boxes for an LED panel of a device4. To generate reference images with reference feature points and reference zones51, a picture is taken from a detected ‘LED panel’ bounding box to configure the LED detection parameters. Deep learning may be used to identify the LED panel and specific LEDs. Parameters may be generated to indicate reference zones where LEDs are in the sub-image defined by the bounding box, and the colors of the LEDs. Feature points51may be extracted and saved from the sample image using a known algorithm, for example SIFT or SuperPoint.

The following text shows an example of an LED configuration:

{“Colors”:[[254,79,6],[248,72,3]],“Locations”:[[[95,117],[112,145]], [[132,117],[149,146]]]“Names”:[« LED1 », « LED2 »]}

Colors indicate the range of RBG values for LED ON colors to detect. Locations indicate the relative locations of LEDs in reference config image. Names indicate the LED name and can be related to semantics of an LED, e.g., a power LED or a switch activity LED, or some other status.

The computer1determines feature points11of the image31. This may be performed by using a deep learning model for object detection that has been trained to detect the objects and states of a particular product. The deep learning model for object detection may determine a bounding box whose label is associated with an ‘LED panel’ in an image. The computer1may then extract a sub-image and determine feature points using a known algorithm (e.g., SIFT or SuperPoint). Notably, if an image31is framed to only capture the LED panel, no sub-image is necessary, and image31is used.

A sample image31of a device4is shown inFIG.2. A bounding box is shown around the LED panel. Sample video frames are shown inFIG.3A-3C, which show LED5blinking.

Image31is likely different from the reference image51in terms of the frame of reference. Therefore, there is a need to match image31to reference image51. Image alignment and registration is known and can be performed using homography. Homography works by capturing feature points and descriptions in a reference image, then matching them in a scanned sample, then creating a transform that warps the sample image to the reference image (or vice versa). Homography may be used to determine a homography matrix to transform one image into the same perspective as another. As shown inFIG.5, a known standard image may be provided on the left. An image, scan, or printout of that image may be provided in the middle. A corrected and remapped version of the image after homography is performed is shown on the right.

The computer1may match the determined feature points11with the reference image feature points51. Upon doing so, the computer1may compute a homography matrix of image31. The computer1may then use the reference zones51to determine where LEDs exist within the image31. The computer1may isolate the LED zones for processing, using a sub-image of image31.

The computer1may determine LED states13. LED states21may include ON/OFF, color, and blinking. To do so, the computer1may warp reference LED zones from reference image51to locations on an extracted sub-image of image31using the computed homography matrix. This allows the computer to detect LED locations, or bounding boxes therefor, in a sub-image of image31.

The computer may determine the ON/OFF state for each of the LEDs41,42,43in the image31. Computer vision may be used to determine the ON/OFF state. For example, the value of certain properties of the image31the brightness, contrast, color, etc. in the region of the LED41,42,43may be compared to reference datum, such as reference values, and used to determine whether the LED is ON or OFF.

To determine color, the computer1may create a combined color mask from configured colors. The OpenCV in Range function, for example, may be used. The computer1may use connected component analysis on the color mask to detect LED color presence.

To determine whether an LED is blinking, the computer1may run a time series analysis on successive frames of video. For example, a rolling window of the last 10 LED states may be kept in memory by the computer1. If the LED state is off for past 10 frames, the status of the LED is OFF. If the LED state is on for past 10 frames, then the status is ON. The status may be blinking if LED status is both on and off in last 10 frames.

The computer1may provide the determined state21of each or a subset of the LEDs to the mobile device2. The mobile device2may display the state21of the LEDs to the user.

The computer1may combine LED states21into conditions14to provide a higher level meaning of the set of LED states. The computer may retrieve condition tables52from the database5, a sample of which is shown inFIG.6. The computer1may then determine the device condition14. The computer pay provide the determine condition22to the mobile device2. The mobile device2may display the condition22of the device4to the user.

As shown inFIG.6, an example of four LEDs may be provided on a panel, which have the following meanings: Power, Up Stream/Down Stream, Online, and 2.4 GHz wireless. By combining the states of these LEDs, higher-level states (“conditions”) can be determined, including: Power Off, Powering On, Booting, 2.4 GHz Ready, Establishing Link, and READY.

Optionally, the computer1may determine troubleshooting steps15based on the condition22of the device4. Alternatively, the computer may determine troubleshooting steps15based on the state21of the LEDs, or any subset thereof. The computer may retrieve troubleshooting steps53from the database5. Upon determining troubleshooting steps23, the computer may provide them to the mobile device2. The mobile device2may display the troubleshooting steps23to the user.

In compliance with the statute, the present teachings have been described in language more or less specific as to structural and methodical features. It is to be understood, however, that the present teachings are not limited to the specific features shown and described, since the systems and methods herein disclosed comprise preferred forms of putting the present teachings into effect. The present disclosure is to be considered as an example of the disclosure and is not intended to limit the disclosure to a specific embodiment illustrated by the figures above or description below.

For purposes of explanation and not limitation, specific details are set forth such as particular architectures, interfaces, techniques, etc. to provide a thorough understanding. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description with unnecessary detail.

Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to a/an/the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated. The use of “first,” “second,” etc. for different features/components of the present disclosure are only intended to distinguish the features/components from other similar features/components and not to impart any order or hierarchy to the features/components. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. As used herein, the term “application” is intended to be interchangeable with the term “invention,” unless context clearly indicates otherwise.

To aid the Patent Office and any readers of any patent issued on this application in interpreting the claims appended hereto, Applicant that it does not intend any of the claims or claim elements to invoke 35 U.S.C. 112(f) unless the words “means for” or “step for” are explicitly used in the particular claim.

While the present teachings have been described above in terms of specific embodiments, it is to be understood that they are not limited to these disclosed embodiments. Many modifications and other embodiments will come to mind to those skilled in the art to which this pertains, and which are intended to be and are covered by both this disclosure and the appended claims. It is intended that the scope of the present teachings should be determined by proper interpretation and construction of the appended claims and their legal equivalents, as understood by those of skill in the art relying upon the disclosure in this specification and the attached drawings. In describing the disclosure, it will be understood that a number of techniques and steps are disclosed. Each of these has individual benefits and each can also be used in conjunction with one, more, or all of the other disclosed techniques. Accordingly, for the sake of clarity, this description refrains from repeating every possible combination of the individual steps in an unnecessary fashion. Nevertheless, the specification and claims should be read with the understanding that such combinations are entirely within the scope of the claims.