Display screen anomaly detection system

Techniques and systems for enhanced detection and correlation of transient anomalies in frames or graphics displayed on a display screen, such as flicker events. In one example, a method includes obtaining luminosity measurements of a display acquired over a series of frames depicted on the display, processing the luminosity measurements to form a luminosity signature for the series of frames, and detecting transient anomalies for the display based at least on the luminosity signature.

BACKGROUND

Display screens are employed in many computing systems for providing visual user interfaces to operators as well as for display of various media or content. The display screens contain various hardware elements, such as display panels, backlights, panel driver circuitry, and display communication links which couple to a computing system having software elements providing graphics that are displayed. However, various updates or changes to software or hardware elements of a computing system having a display can produce unwanted behavior in the display of graphics due to bugs, interoperability issues, unforeseen impacts, and other problems.

Thus, fault detection testing or other verification/validation testing is often performed for various components of a computing system. However, issues exhibited by display of graphics or media content over time can be difficult or cumbersome to detect. This can be especially difficult during playback monitoring of large media files comprising very dense data sets as displays comprise ever-increasing screen resolutions and pixel densities. Transient events can include flicker events, which are visual anomalies that produce rapid unwanted visual changes to a display, such as screen blanking events or screen color saturation events.

SUMMARY

Presented herein are techniques and systems for enhanced detection and correlation of transient anomalies in frames of graphics displayed on a display screen, such as flicker events. These transient anomalies can be detected in computing systems having graphical displays which depict user interface features to users, as well as provide various media/content playback. While the examples herein discuss visual anomaly detection, such as flicker event detection, similar techniques can be applied to other sensory anomaly detection, such as audio, when corresponding hardware interfaces and monitoring elements are employed.

In one example implementation, a method includes obtaining luminosity measurements of a display acquired over a series of frames depicted on the display, processing the luminosity measurements to form a luminosity signature for the series of frames, and detecting transient anomalies for the display based at least on the luminosity signature.

Another example implementation includes a processing system operatively coupled with one or more computer readable storage media, and program instructions stored on the one or more computer readable storage media. Based at least on being read and executed by the processing system, the program instructions direct the processing system to at least obtain luminosity measurements of a display acquired over a series of frames depicted on the display, process the luminosity measurements to form a luminosity signature for the series of frames, and detect transient anomalies for the display based at least on the luminosity signature.

In yet another example implementation, a system comprises a luminosity assembly including luminosity sensors configured to acquire luminosity measurements of a display over a series of frames depicted on the display. The system also comprises a luminosity processor configured to receive the luminosity measurements and process the luminosity measurements to form a luminosity signature for the series of frames, and correlate in time the luminosity signature to a reference signature to determine differences between the luminosity signature and the reference signature. The luminosity processor also can process the luminosity signature in a frequency domain at timeframes corresponding to the differences to determine frequency components above a threshold frequency selected to signify screen flicker anomalies.

DETAILED DESCRIPTION

Fault or anomaly detection during testing of computing systems can be employed to verify operation of display screens and the various elements forming a computing device coupled to the display screen, which include software elements that generate the graphics, user interface elements, or other visual data ultimately displayed on the display screen. One of the challenges in developing and testing software for computing devices is to detect and report any visual or audible problems introduced from software (or changes thereto) which may affect the customer experience. Moreover, individual devices might exhibit failures from unwanted transient behavior exhibited by a display screen during fault detection, manufacturing testing, or other processes.

However, issues exhibited by display of graphics or media content over long periods of time can be difficult or cumbersome to detect, often requiring large sets of test data that are post-processed to determine faults. This can be especially pronounced when performing testing related to software updates which might apply over many systems or devices. Software updates can also affect many components of a computing system which may be difficult to isolate from among operating systems, graphics drivers, graphics rendering engines, applications, and other various software. The examples herein can automate testing, deployment, and fault detection of computing systems, which can accelerate the development cycle for such computing systems and improve customer satisfaction by delivering higher quality software or hardware.

The example implementations discussed herein provide techniques, software, testing apparatuses, and systems for enhanced detection and correlation of transient anomalies in frames of graphics displayed on a display screen, such as flicker events. As discussed herein, transient events or transient anomalies can include flicker events on a display, which are visual anomalies that produce rapid unwanted visual changes to a display, such as screen blanking events or screen color saturation events. These transient anomalies can be detected in computing systems having graphical displays which depict user interface features to users, as well as provide various media/content playback. Humans can perceive flicker that is greater than approximately 33 milliseconds (ms) in duration. However, as display/graphics technology has increased to have refresh rates exceeding 120 Hz (i.e., 120 frames per second), some single-frame flicker events occur too quickly for the human eye to detect. For instance, one frame of flicker in 120 Hz video is approximately 8.3 ms in duration. Advantageously, the examples herein discuss various technical benefits and technical effects for automated anomaly detection in computing systems, such as flicker event detection. Similar techniques can be applied to other sensory anomaly detection, such as audio, when corresponding hardware interfaces and monitoring elements are employed.

Turning now to a first example implementation,FIG.1is presented.FIG.1includes testing environment100which includes various elements employed for computing systems and testing for anomalies during operation. Environment100includes test device110, test fixture120, and anomaly detection device140. Test fixture120is mounted to display screen111of test device110, and provides various measurement data to anomaly detection device140over link143.

Text fixture120can be applied to display screen111during a testing operation to perform measurements of display screen111. Measurements made by text fixture120can be provided over link143to anomaly detection device140. These measurements can include luminosity measurements made of display screen111, as well as audio measurements made for test device110. Test fixture120comprises a plurality of luminosity sensors which measure luminosity of display screen111at various locations. Test fixture120can optionally accumulate luminosity measurements over various sensors and over time, as well as audio measurements, and provide data comprising accumulated measurements over audio link144, and provide over link143.

During testing operations, a series of frames are depicted on display screen111, such as shown for frames150for test graphics130. Test graphics130can comprise playback of media content, sequences of user interface graphics, or other sequences of changing displayed graphics. While a static image might be displayed on display screen111, multi-point luminosity measurements can be obtained by test fixture120typically are made over time using changing graphics or images, such as encountered during movie playback, on-screen animations, user interface interactions, computer startup/boot processes, automated scripting of dynamic user interface activities, and the like. Display of graphics or other images on display screen111has a granularity of individual frames which are refreshed periodically according to a display refresh rate, which can be separate from any media frame rate. The measurements of multiple points of luminosity as well as multiple frames of images or graphics provides several technical benefits and technical effects. These technical benefits and technical effects include the ability to establish luminosity signatures or fingerprints which use less storage space and processing power in determination of transient anomalies than storing screen-captured recordings of a display screen during testing or other techniques. Moreover, technical benefits and technical effects provide for near-real time determination of anomalies, such as screen flicker, which do not employ extensive video post-processing operations.

A display refresh rate is often listed in terms of frames-per-second (FPS) or cycles-per-second (Hertz, or Hz), and is the rate at which display screen111updates corresponding graphics which are provided over a display link after being rendered by a graphics unit of test device110. A sample rate (shown for luminosity samples151) of the luminosity measurements made by test fixture120can comply with a Nyquist rate for the frame/refresh rate, such as two or more measurements/samples per frame. The sample rate might include 320 samples/second in some examples. Thus, the quantity of luminosity samples151can correspond to a sample rate multiplied by the quantity of sensors employed, such as eight (8) in this example. Other test setup parameters can include target brightness levels, such as over 75% of a maximum brightness, and when audio is employed, a target audio volume (e.g., over 75% of a maximum volume). Other test setup parameters can be employed.

Anomaly detection device140can comprise control interface141which can receive or otherwise obtain the luminosity measurements, such as samples151for each luminosity sensor, over link143. Additionally, audio data, which may comprise recorded audio, filtered audio, sampled audio, or other audio representations, can accompany the luminosity measurements over link143, or may be provided separately (e.g., over link144). Control interface141can provide control instructions or commands to test fixture120over link143, or other links, which can include indications on when to commence luminosity or audio measurements, boot up/start up instructions for processor elements of test fixture120. Operating power might be provided to test fixture120over link143from device140, such as when link143comprises a universal serial bus (USB) type-C compliant link, among others. Operating power might instead be provided to test fixture120from other sources, such as from test device110or from independent power sources.

Anomaly detection device140can include luminosity processor142, which in some examples, obtains luminosity measurements or audio measurements as samples made over time. In some examples, luminosity processor142can process the luminosity measurements to form a luminosity signature for the series of frames that correspond to the luminosity measurements, and then detect transient anomalies shown on display screen111based at least on the luminosity signature. The luminosity measurements comprise samples over the series of frames of intensities, magnitudes, or amplitudes corresponding to luminosity measured at multiple selected locations on display screen111. The luminosity signature can comprise an accumulated set of luminosity samples for the multiple luminosity sensors over a timeframe employed for the testing operations, and these luminosity samples can be correlated in time to each other such that each measurement location can have a corresponding timestamp or frame identifier among the series of frames. For example, if a testing operation proceeds for 10 minutes, then luminosity measurements for each luminosity sensor at the sample rate are combined into a composite set of luminosity samples correlated in time or to particular frames.

Luminosity processor142can process the luminosity signature in a time domain or a frequency domain to determine transient (visual or audio) anomalies. In one example, luminosity processor142processes the luminosity signature to identify a reference luminosity signature that corresponds to the series of frames depicted on the display. A reference luminosity signature can be formed as a “known-good” signature for a similar or overlapping set of frames comprising a same or similar series of graphical content, such as the same media content or same set of user interface operations, which produces a similar luminosity signature for the frames. Typically, the reference luminosity signature can be determined prior to a present testing process that measures a present luminosity signature, and then stored for later use by luminosity processor142. The reference luminosity signature might be determined using similar techniques as discussed herein for the present testing process. In other examples, a reference luminosity signature might not be employed, and instead luminosity processor142can employ a frequency domain processing can be employed to identify transient anomalies.

For time domain processing, the present luminosity signature might be shifted in time as compared to the reference luminosity processor. A time shift process can be performed to align the present luminosity signature to the reference luminosity signature, which might be keyed off of various trigger events, such as a start time of sequence of frames or a comparison over a segment of the frames to align various events among the luminosity signatures. Luminosity processor142can process the present luminosity signature against the reference luminosity signature to align the luminosity signature with respect to the reference luminosity signature in time according to luminosity events correlated among the luminosity signature and the reference luminosity signature. Alternatively, a pattern matching process can be performed across several of the samples in the present luminosity signature to find the reference luminosity signature from among several reference luminosity signatures formed for differing graphical content. Once the reference luminosity signature has been identified, then a sample-by-sample comparison might be made in the time domain to identify differences between the present luminosity signature and the reference luminosity signature. Luminosity processor142can compare at least a portion of the present luminosity signature against the reference luminosity signature to determine differences between the reference luminosity signature and the luminosity signature indicating transient anomalies. The use of reference luminosity signatures provide several technical benefits and technical effects. For example, the alignment of a present luminosity signature to a reference luminosity signature can provide for more effective searching and finding of a reference luminosity signature from among many reference luminosity signatures, as well as for a more efficient comparison among features within the present luminosity signature to features within the luminosity signature to identify anomalies.

In other examples, frequency domain processing can be employed to determine transient anomalies. Luminosity processor142can process frequency components of the present luminosity signature to determine frequency components above a threshold frequency selected to signify transient anomalies, such as screen flicker anomalies. Luminosity processor142can filter the present luminosity signature according to a bandpass filter having a frequency range that includes a frequency selected to indicate the transient anomalies. Then luminosity processor142can process the filtered (present) luminosity signature to determine instances of the filtered luminosity signature exceeding a threshold amplitude, magnitude, or intensity. The frequency range for filtering and selected frequency threshold that indicates transient anomalies can depend on various factors, including a frame rate, sample rate, and other properties of the luminosity signatures. In one example, for a graphics or display frame rate of 120 Hz and a luminosity measurement/sample rate of 240 Hz or greater, then a low frequency cut off frequency can be approximately 1 Hz, and a high frequency cut off frequency can be approximately 4 Hz. Then, signals which have amplitudes above a threshold amplitude after application of the filter can indicate transient anomalies, such as screen flicker. Frequency domain processing provides several technical effects and technical benefits, among which include increased processing efficiency and/or accuracy of identifying relevant flickering or other anomalies within luminosity signatures.

Regardless of if a time domain or frequency domain was employed to determine transient anomalies, indications of the transient anomalies can be provided to operators, users, log files, over a network or data interface, or other to destinations and links. These indications can include timestamps or sample identifiers among the luminosity signature, as well as what reference luminosity signature or graphical content is associated with the anomalies. Moreover, a correlation to system log events might be determined for test device110. Typically, test device110will be running an operating system or other software/firmware for which system-level logging or event tracking is employed. Luminosity processor142can receive log files or event logs for test device110through control interface141. Luminosity processor142can then correlate any discovered transient anomalies to operating system log events for test device110generating the series of frames and comprising the measured display screen111. Luminosity processor142can correlate or align in time the transient anomalies to the operating system log events based at least on a trigger event in the operating system log events indicating a graphics rendering start time for the series of frames. This graphics rendering start time might correspond to a graphics video decode engine ramping activity up from an idle or steady-state level of activity to a higher level of activity, such as a quick ramp or step-change in activity levels. The activity levels might be measured by processor utilization, memory usage levels, rendering software initiation events, or other events which indicate that the graphical content is being rendered for display on displace screen111by test device110. Technical benefits and technical effects for correlating anomalies or flicker events to log events or system events can include identifying performance aspects of an operating system (or other test device elements) during occurrence of anomalies or flicker events to more efficiently identify or localize faults or errors in devices under test.

In addition to the video or graphical data determined above, such as the luminosity measurements, audio data or audio measurements can also be employed to further localize transient events in graphical content or to identify audio-based transient events separate from graphical content. In such examples, luminosity processor142can receive audio measurements or audio data for test device110through control interface141which communicates over link143. These audio measurements can correspond to the series of frames used for luminosity measurements. Luminosity processor142can then correlate the audio measurements to portions of the luminosity signature. Luminosity processor142can detect anomalies in the audio measurements which correspond to visual transient anomalies for the display, and provide indications of these audio anomalies along with indications of visual/graphical transient anomalies. Processing of audio measurements along with luminosity measurements provides several technical benefits and technical effects, such as identification of systemic faults which affect both audio and video, using audio content to determine flicker in video, such as when audio glitches or spikes/dips occur, and correlating audio events to video events for more efficient identification of flicker or other anomalies.

Returning to a discussion on the elements ofFIG.1, test device110comprises processing circuitry, communication interface circuitry, and user interface elements including display screen111. Test device110can also include memory devices, storage devices, software, graphics rendering circuitry, or other computing components. Test device110can be a computer, laptop computing device, tablet computing device, smartphone, gaming console, customer equipment, access terminal, mobile Internet appliance, media player, or some other user computing apparatus, including combinations thereof. Test device110executes software, such as operating systems, display drivers, and applications to display graphical content on display screen111. This graphical content can comprise any suitable dynamic or changing content, with a frame-level granularity, such as media content, movies, animations, screen savers, operating system activities in a graphical user interface, terminal displays, test sequences, scripted graphical frames, image sequences, device boot or startup graphical sequences, or other content.

Display screen111comprises any suitable display, such as a liquid crystal display (LCD), light-emitting diode (LED) display, organic LED display, active-matrix organic LED display, micro-LED display, plasma display, electroluminescent display, quantum dot display, projector display, cathode ray tube display, or other electronic visual display comprising a screen onto which visual elements are presented. Display screen111may or may not include a backlight component, depending on the display technology. Display screen111typically includes a display interface or link which couples to other elements in test device110which render graphics and integrate the rendered graphics into a display protocol or display link format.

Anomaly detection device140is further exemplified inFIG.8, but can comprise various computing components with data processing elements suitable for at least the operations described above. Anomaly detection device140includes, but is not limited to, desktop and laptop computers, tablet computers, gaming computers, gaming consoles, media playback devices, mobile computers, audio devices, server computers, or other computing devices. Examples of anomaly detection device140may also include web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, container, and any variation or combination thereof. As mentioned above, anomaly detection device140includes control interface141and luminosity processor142.

FIG.2is included to further illustrate example implementations of test fixture120, although variations are possible. Specifically,FIG.2includes views200-201which illustrate different examples of test fixture220. View200illustrates test fixture220applied to test device210, which comprises display screen211. View201illustrates the opposite side of test fixture220from that seen in view200, which corresponds to a sensing face or side applied to test device210.

Test fixture210comprises chassis221which forms a frame or mounting plate to which various other components are coupled. Chassis221provides for mounting points for luminosity sensors224such that the sensors are arrayed over a selected portion of display screen211. Chassis221can comprise various materials sufficient to maintain a rigidity over the span used to measure display screen211, which can vary based on screen size, among other factors. Example materials include metallic materials, such as aluminum, titanium, magnesium, steel, and alloys thereof. Other example materials include various polymers, such as acrylic, ABS, PVC, wood, laminate/composite material, carbon fiber reinforced materials, printed circuit board material, or fiberglass, among others. Chassis221can be formed by various manufacturing operations, including milling, machining, etching, additive manufacturing (e.g.,3D printing), casting, and other techniques. Chassis221has various apertures and mounting points which can house or hold luminosity sensors224, processor assembly222, and screen mounting elements225.

Test fixture210can be mounted to provide luminosity sensors224with a view of display screen211. Mounting elements225can be included to hold test fixture210onto display screen211, and can comprise suction cups, adhesive portions, magnetic elements, clamps, or other features. Typically, mounting elements225will non-destructively and removably mount test fixture210to display screen211.

Luminosity sensors224comprise photodetectors which convert incident light into electrical signaling indicating an intensity of photo-illumination. Luminosity sensors224might comprise photoresistors, photodiodes, phototransistors, image sensors, or other types of photodetectors capable of microsecond-scale measurements. Luminosity sensors224can couple over links to processor assembly222having processor223. These links can include analog or digital links, along with any corresponding filtering, noise rejection elements, shielding, or other link elements.

Processor assembly222includes a printed circuit board that carries signaling among various processing elements, such as processor223. In some examples, processor223(or separate analog-to-digital circuitry) can be configured to obtain luminosity measurements in an analog format from luminosity sensors224and convert these measurements to a digital format, or obtain periodic digital samples made of analog signals for luminosity sensors224. Processor223can assemble, encapsulate, or package the measurements into luminosity signatures by time-correlating each of the luminosity measurements or assigning timestamps to the samples of each luminosity measurements. Processor223then transfers the luminosity signatures over a data link to an endpoint, such as an anomaly detection device or other computing device. Processor223can include audio measurements or audio recordings along with the luminosity measurements. The link type can be any discussed herein, such as USB, USB-C, Ethernet, wireless networking (e.g., Wi-Fi), discrete signaling, proprietary signaling, or any digital, analog, or packetized link. Examples of processor223include general purpose central processing units, graphical processing units, digital signal processors, application specific processors, and logic devices, Arduino® processing devices, as well as any other type of processing device, combinations, or variations thereof. Processor assembly222can also include various communication interfaces, power supply components, data storage elements, audio measurement or recording elements (microphone or interface to receive output audio from a test device).

FIG.3illustrates example operations for a testing environment in an implementation. The operations ofFIG.3are discussed in the context ofFIG.1, although other elements presented herein can instead be employed.

In operation310, anomaly detection device140obtains luminosity measurements of a display acquired over a series of frames depicted on the display. Anomaly detection device140obtains luminosity measurements received by control interface141and provided by test fixture120over link143. WhileFIG.1shows an eight-point measurement scheme, other quantities of multi-point measurements can be employed. A luminosity signature is formed from a collection of luminosity measurements for a series of frames over a selected timeframe for display screen111obtained by luminosity sensors of test fixture120. Luminosity processor142can process the luminosity measurements (operation311) to form the luminosity signature for the series of frames, or alternatively, luminosity processor142might obtain the luminosity signature as determined by test fixture120. The luminosity signature comprises a time-synchronized arrangement of the luminosity measurements, which might be accompanied by various metadata, system logs/events for test device110, operational telemetry data for test device110, audio data or measurements, identities of test device110and component specifications for internal components, and other information.

Based at least on the luminosity signatures, luminosity processor142can detect (operation312) transient anomalies for display screen111based at least on the luminosity signature. The transient anomalies might comprise visual flicker events or instances of screen flicker spanning at least one frame or one screen refresh cycle. Flicker events can include visual anomalies that produce rapid unwanted visual changes to display screen111, such as screen blanking events or screen color saturation events.

Luminosity processor142can employ different techniques to detect or identify transient anomalies. These include processing within a time domain (operation314) or frequency domain (operation313). A time domain includes samples of luminosity measurements in the luminosity signature, and a frequency domain includes performing a mathematical transform to convert the time domain representation into a frequency domain representation. This transform can include various type of Fourier transforms, including the Fast Fourier Transform (FFT), among others. Both the time domain and frequency domain might be employed to detect or identify transient events, including cross-correlation among events found in both the time domain and frequency domain.

For time domain processing, luminosity processor142can process a present luminosity signature against a set or collection of reference luminosity signatures to determine which reference luminosity signature corresponds to the same or similar graphical output as the present luminosity signature. For example, a set of reference luminosity signatures can be maintained as known-good or flicker-free luminosity signatures in a database or other data structure. Various dynamic graphical elements can be presented on a display screen under test, such as media content, movies, animations, operation system operations, and the like. Testing can be performed on test devices to obtain new luminosity signatures, and these can be compared against a corresponding reference luminosity signature to identify differences which might correspond to transient anomalies. Also, a timewise alignment can be established among the present luminosity signature and a reference luminosity signature to more effectively identify any differences. The alignment might be triggered or gauged to various events within a set of frames, such as a beginning of the series of frames or known-luminosity events within graphical content comprising the series of frames (i.e., scene changes, lighting changes, intermissions, animation/OS transitions, etc.).

For frequency domain processing, luminosity processor142can either transform the present luminosity signature into a frequency domain or process the time domain representation with one or more filters, among other frequency-based processing. In one example, a bandpass filter is applied to the present luminosity signature to identify luminosity events which fall within the filter passband, and events with an amplitude or intensity above a threshold level can be considered as transient anomalies or transient anomaly candidates. In another example, a frequency domain representation, such as an FFT representation, can be analyzed for events corresponding to a certain frequency range, such as that noted above for the bandpass filter. A bandpass filter might be applied in the frequency domain after the FFT transfer. Events with an amplitude or intensity above a threshold level for a range of frequencies in the FFT representation can be considered as transient anomalies or transient anomaly candidates. The set of candidates might be further processed to determine correlation among all luminosity channels for a given point in time, frame or set of frames, or refresh cycles to confirm if the candidates correspond to screen flicker, color saturation events, or other events of note.

Once one or more transient anomalies are identified, properties of the transient anomalies can be employed to correlate in time the transient anomalies to other activities of test device110. This can include correlating (315) the transient anomalies to system log events of test device110.

FIG.4shows example400as one example of correlating system log events to transient anomalies. For an identified transient anomaly410corresponding to flicker event153ofFIG.1, a timestamp can be identified from test device110. Transient anomaly410might also have a corresponding luminosity signature411which provides context for timeframes before and after transient anomaly410, as well as an audio sample412for timeframes before and after transient anomaly410. A system log of test device110, corresponding to operating system events, among other activities, can be obtained. This system log might be correlated in time to the timestamp of transient anomaly410, and a portion of the system log can be provided as a test context of transient anomaly410. As seen for system log portion420, a series of system events is included which precede transient anomaly410.

System log portion420can be a portion of a larger system event log which can be generated by monitoring services executed by a test device which provide a record or log of events associated with usage of associated applications or operating system elements. Any log or events associated with the system-under-test can be employed, such as any log determined by a test device or telemetry snooping on that test device. In some examples, log events or telemetry snooping can indicate when a graphics processing unit (GPU) decode engine ramps up to trigger anomaly detection operations. System log portion420can include logged events from hardware monitoring elements which provide sensor data, environmental data, user interface event data, or other information related to usage of hardware elements. These hardware elements can include computing systems, such as personal computers, server equipment, distributed computing systems, or can include discrete sensing systems, industrial or commercial equipment monitoring systems, medical sensing equipment, or other hardware elements. In further examples, system log portion420can include monitored elements of a virtualized computing environment, which can include hypervisor elements, operating system elements, virtualized hardware elements, software defined network elements, among other virtualized elements.

Operation316includes luminosity processor142providing indications of the transient anomalies, such as flicker events, along with various metadata, timestamps, system events correlated in operation315, and other accompanying information. From here, further determinations can be made as to root causes of transient anomaly410for debug, repair, software update regression testing failure identification and resolution, and other activities.

Turning now to example measurements of luminosity and processing of luminosity signatures,FIGS.5-7are presented.

FIG.5includes graph500indicating an eight (8) channel luminosity measurement of a display screen. A vertical axis of graph500corresponds to luminosity intensity or magnitude (in lumens or other suitable metric), and a horizontal axis of graph500corresponds to samples over time. Legend501indicates eight sensor plots, each corresponding to a different location on a display screen, and each synchronized in time in graph500. As can be seen, at time T1, a start of a test sequence is initiated, and at time T2, an end of a test sequence occurs. During the intervening time (T1-T2), various luminosity measurements are taken by associated luminosity sensors. These sensors are applied to corresponding locations on a display screen, and the on-screen images can fluctuate according to the various frames provided to the display screen. Transient events might occur during this timeframe, and further processing is provided to identify these events.

FIG.6includes graphs600-601indicating luminosity measurements of a display screen. Graph600corresponds to a present luminosity signature for a single luminosity sensor channel. Further channels can be acquired during measurement, but for clarity only a single channel is featured inFIG.6. Graph601corresponds to a reference luminosity signature for a single luminosity sensor channel. As can be seen inFIG.6, plot610in graph600is shifted in time (Δt) with respect to plot611of graph601. This time shift can be identified by one or more events correlated among plot610and611, such as the drop in luminosity shown indicating metric Δt. This time shifting can assist in processing the present luminosity signature against the reference luminosity signature, as well as for identifying a reference luminosity signature that corresponds to the content or graphics used to form the present luminosity signature.

FIG.7includes graphs700-703indicating various data and transforms of luminance measurements of a display screen. Graph700illustrates a time domain plot710for a present luminosity signature for a single luminosity sensor channel. A vertical axis of graph700corresponds to a luminosity magnitude, and a horizontal axis corresponds to samples over time. Graph701illustrates a frequency domain plot711the present luminosity signature for a single luminosity sensor channel. A vertical axis of graph701corresponds to a frequency component magnitude, and a horizontal axis corresponds to frequency (Hz). Further channels can be acquired during measurement, but for clarity only a single channel is featured inFIG.7. Frequency domain plot711can be determined from time domain plot710, such as by processing time domain plot710with an FFT style of transform. As can be seen for time domain plot710, various luminosity measurements are made, with some dramatic dips in luminosity shown, which might correspond to transient anomalies. However, further processing is provided to determine if these are actually transient anomalies.

From time domain plot710, a bandpass filtered version of time domain plot710is shown for graph702, which has a bandpass range of 1-4 Hz in this example. As with graph700, graph702has a vertical axis corresponding to a luminosity magnitude, and a horizontal axis corresponding to samples over time. However, graph702, due to the bandpass filter, has had any DC components filtered out. This results in plot712in graph702centered on a zero (0) luminosity, with positive and negative excursions from the zero point. Negative values correspond to falling edges of plot710, and positive values correspond to rising edges of plot701. The FFT in combination with a bandpass filter can produce a filtered signal which highlights or emphasizes sudden changes to a display, which can appear as higher frequency components of a luminosity signature.

Next, graph703illustrates plot713having flicker events, such as flicker event714. The excursions from the zero point of plot712can have an amplitude or magnitude threshold applied thereto, which forms a binary representation of excursions which exceed a threshold amplitude or magnitude in plot713. Plot713shows detected flicker events corresponding to samples where the luminance drops and increases occur withing a frequency window and exceeding a target magnitude. These flicker events can correspond to the transient anomalies discussed herein.

FIG.8illustrates computing device801that is representative of any system or collection of systems in which the various processes, programs, services, and scenarios disclosed herein may be implemented. Examples of computing device801include, but are not limited to, desktop and laptop computers, tablet computers, gaming computers, gaming consoles, media playback devices, mobile computers, audio devices, and wearable devices. Examples may also include server computers, web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, container, and any variation or combination thereof. Computing device801can be one example implementation of anomaly detection device140ofFIG.1or processor223ofFIG.2, although variations are possible.

Computing device801may be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing device801includes, but is not limited to, processing system802, storage system803, software805, communication interface system807, and user interface system809. Processing system802is operatively coupled with storage system803, communication interface system807, and user interface system809.

Processing system802loads and executes software805from storage system803. Software805includes and implements anomaly detection process806, which is representative of control interface141, luminosity processor142, or operations300, among other luminosity processing components, transient detection elements, operations, and processes. When executed by processing system802, software805directs processing system802to operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing device801may optionally include additional devices, features, or functionality not discussed for purposes of brevity. Processing system802may comprise a microprocessor and other circuitry that retrieves and executes software805from storage system803. Processing system802may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing system802include general purpose central processing units, graphical processing units, digital signal processors, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

Storage system803may comprise any computer readable storage media readable by processing system802and capable of storing software805. Storage system803may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic storage media, optical storage media, flash memory, virtual memory and non-virtual memory, or any other suitable storage media. In no case is the computer readable storage media a propagated signal. In addition to computer readable storage media, in some implementations storage system803may also include computer readable communication media over which at least some of software805may be communicated internally or externally. Storage system803may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage system803may comprise additional elements, such as a controller, capable of communicating with processing system802or possibly other systems.

Software805(including anomaly detection process806) may be implemented in program instructions and among other functions may, when executed by processing system802, direct processing system802to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, software805may include program instructions for implementing the luminance measurement and anomaly detection processes described herein. In particular, the program instructions may include various components or modules that cooperate or otherwise interact to carry out the various processes and operational scenarios described herein. The various components or modules may be embodied in compiled or interpreted instructions, or in some other variation or combination of instructions. The various components or modules may be executed in a synchronous or asynchronous manner, serially or in parallel, in a single threaded environment or multi-threaded, or in accordance with any other suitable execution paradigm, variation, or combination thereof. Software805may include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Software805may also comprise firmware or some other form of machine-readable processing instructions executable by processing system802.

In general, software805may, when loaded into processing system802and executed, transform a suitable apparatus, system, or device (of which computing device801is representative) overall from a general-purpose computing system into a special-purpose computing system customized to perform the luminance measurement and anomaly detection processes in an optimized manner. Indeed, encoding software805on storage system803may transform the physical structure of storage system803. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the storage media of storage system803and whether the computer-storage media are characterized as primary or secondary storage, as well as other factors. For example, if the computer readable storage media are implemented as semiconductor-based memory, software805may transform the physical state of the semiconductor memory when the program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. A similar transformation may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate the present discussion.

Communication interface system807may include communication connections and devices that allow for communication with other computing systems (not shown) over communication networks (not shown). Examples of connections and devices that together allow for inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over communication media to exchange communications with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication media. The aforementioned media, connections, and devices are well known and need not be discussed at length here. Communication between computing device801and other computing systems (not shown), may occur over a communication network or networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software defined networks, data center buses and backplanes, or any other type of network, combination of network, or variation thereof. Example communication links and interfaces further include Ethernet interfaces, Wi-Fi (802.11xx) interfaces, Peripheral Component Interconnect Express (PCIe) interfaces, Compute Express Link (CXL), serial interfaces, serial peripheral interface (SPI) links, inter-integrated circuit (I2C) interfaces, universal serial bus (USB) interfaces, and UART interfaces, among others.

Advantageously, the examples presented herein can provide transient anomaly detection for various visual flickers or other unwanted behavior for display screens. The use of a test device along with an externally-applied test fixture can test in situ an entire stack of graphics generation, processing, rendering, transfer, and display. The test fixtures discussed herein can readily be attached to any device with a display screen, and can automate the activity of investigating flicker or other visual artifacts and anomalies. Moreover, these techniques and systems can detect anomalies that the human eye cannot perceive due to increasing refresh rates or frame rates. The techniques and systems can be used to detect audio anomalies and video/audio out of sync issues.

The luminosity fingerprinting or signatures determined herein can provide multi-point measurements of a display screen over time, which greatly reduces the needed storage and datasets for high-resolution, high-frame rate video or graphics. A luminosity signature comprises luminosity measurements for each frame, and over the course of many frames this signature trends towards a unique identity of the content displayed, such as a movie or animation. Contrary to analyzing each frame and pixel, which is computationally intensive, the examples herein reduce the complexity by luminance measurement on multiple points, and the number of points could be increased or decreased based on need. The luminosity signature for each frame provides sufficient information to automatically analyze and report flickers or anomalies, often in real-time instead and avoids lengthy post-processing.

In certain examples, a cost-effective test fixture can be fitted with low-cost processing elements and multiple photoresistors that provide scalable, readily deployable testing for labs, factory, and other test environments. The examples herein can measure randomly-distributed points on a display or screen to perform a luminance probe. By probing multiple points, a test system can generate a distinct pattern for each frame, referred to as a fingerprint or signature. Voltage levels for each photoresistor can be measured in a time domain and processed in a time or frequency domain, such as using bandpass filtering and FFT analysis. Sudden changes in luminosity, corresponding to high-frequency signal content, can indicate flicker or other anomalies. In other examples, the test systems can measure or record stereo audio channel values for each frame as an additional feature.

Further enhanced features include correlation in time of flicker anomalies to operating system log events for a device performing media playback on a display. The test system can provide indications of the flicker anomalies and operating system events contextualized to the flicker anomalies. This can be helpful when debugging or monitoring reboot cycles of a computing device which experiences flicker or other anomalies. Comparing frames using time domain cross-correlation can lead to 80% accuracy and allows the system to gather and process data at scale.

Certain inventive aspects may be appreciated from the foregoing disclosure, of which the following are various examples.

Example 1: A method, comprising obtaining luminosity measurements of a display acquired over a series of frames depicted on the display, processing the luminosity measurements to form a luminosity signature for the series of frames, and detecting transient anomalies for the display based at least on the luminosity signature.

Example 2: The method of Example 1, wherein the luminosity signature comprises multiple samples of the luminosity measurements for each frame over a period of time encompassing the series of frames.

Example 3: The method of Examples 1-2, comprising processing the luminosity signature to identify a reference luminosity signature that corresponds to the series of frames depicted on the display, and comparing at least a portion of the luminosity signature against the reference luminosity signature to determine differences between the reference luminosity signature and the luminosity signature indicating the transient anomalies.

Example 4: The method of Examples 1-3, comprising processing the luminosity signature against the reference luminosity signature to align the luminosity signature with respect to the reference luminosity signature in time according to luminosity events correlated among the luminosity signature and the reference luminosity signature.

Example 5: The method of Examples 1-4, comprising processing frequency components of the luminosity signature to determine frequency components above a threshold frequency selected to signify screen flicker anomalies.

Example 6: The method of Examples 1-5, comprising correlating the transient anomalies to operating system log events for a device generating the series of frames and comprising the display, and providing indications of at least the operating system log events as corresponding to the transient anomalies.

Example 7: The method of Examples 1-6, comprising correlating the transient anomalies to the operating system log events based at least on a trigger event in the operating system log events indicating a graphics rendering start time for the series of frames.

Example 8: The method of Examples 1-7, wherein the luminosity measurements comprise luminosity intensity measurements made at multiple selected locations across the display and correlated in time to frames among the series of frames.

Example 9: The method of Examples 1-8, comprising obtaining audio measurements corresponding to the series of frames and correlating the audio measurements to portions of the luminosity signature; and detecting anomalies in the audio measurements which correspond to the transient anomalies for the display.

Example 10: The method of Examples 1-9, wherein the series of frames depicted on the display corresponds to at least one among media playback on the display, operations within a user interface depicted on the display, or a boot process of a computing system comprising the display.

Example 11: An apparatus, comprising a processing system operatively coupled with one or more computer readable storage media, and program instructions stored on the one or more computer readable storage media. Based at least on being read and executed by the processing system, the program instructions direct the processing system to at least obtain luminosity measurements of a display acquired over a series of frames depicted on the display, process the luminosity measurements to form a luminosity signature for the series of frames, and detect transient anomalies for the display based at least on the luminosity signature.

Example 12: The apparatus of Example 11, wherein the luminosity signature comprises multiple samples of the luminosity measurements for each frame over a period of time encompassing the series of frames.

Example 13: The apparatus of Examples 11-12, comprising program instructions that further direct the processing system to at least process the luminosity signature to identify a reference luminosity signature that corresponds to the series of frames depicted on the display, and compare at least a portion of the luminosity signature against the reference luminosity signature to determine differences between the reference luminosity signature and the luminosity signature indicating the transient anomalies.

Example 14: The apparatus of Examples 11-13, comprising program instructions that further direct the processing system to at least process the luminosity signature against the reference luminosity signature to align the luminosity signature with respect to the reference luminosity signature in time according to luminosity events correlated among the luminosity signature and the reference luminosity signature.

Example 15: The apparatus of Examples 11-14, comprising program instructions that further direct the processing system to at least process frequency components of the luminosity signature to determine frequency components above a threshold frequency selected to signify screen flicker anomalies.

Example 16: The apparatus of Examples 11-15, comprising program instructions that further direct the processing system to at least correlate the transient anomalies to operating system log events for a device generating the series of frames and comprising the display, and provide indications of at least the operating system log events as corresponding to the transient anomalies.

Example 17: The apparatus of Examples 11-16, comprising program instructions that further direct the processing system to at least correlate the transient anomalies to the operating system log events based at least on a trigger event in the operating system log events indicating a graphics rendering start time for the series of frames.

Example 18: The apparatus of Examples 11-17, wherein the luminosity measurements comprise luminosity intensity measurements made at multiple selected locations across the display and correlated in time to frames among the series of frames.

Example 19: The apparatus of Examples 11-18, comprising program instructions that further direct the processing system to at least obtain audio measurements corresponding to the series of frames and correlating the audio measurements to portions of the luminosity signature, and detect anomalies in the audio measurements which correspond to the transient anomalies for the display.

Example 20: A system, comprising a luminosity assembly comprising luminosity sensors configured to acquire luminosity measurements of a display over a series of frames depicted on the display, and a luminosity processor. The luminosity processor is configured to receive the luminosity measurements and process the luminosity measurements to form a luminosity signature for the series of frames, and filtering the luminosity signature according to a bandpass filter having a frequency range that includes a frequency selected to indicate screen flicker anomalies, process the filtered luminosity signature to determine instances of the filtered luminosity signature exceeding a threshold amplitude, and transfer indications of the screen flicker anomalies comprising the instances.

The functional block diagrams, operational scenarios and sequences, and flow diagrams provided in the Figures are representative of exemplary systems, environments, and methodologies for performing novel aspects of the disclosure. The descriptions and figures included herein depict specific implementations to teach those skilled in the art how to make and use the best option. For the purpose of teaching inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations from these implementations that fall within the scope of the disclosed examples. Those skilled in the art will also appreciate that the features described above can be combined in various ways to form multiple implementations. As a result, the invention is not limited to the specific implementations described above, but only by the claims and their equivalents.

The various materials and manufacturing processes discussed herein are employed according to the descriptions above. However, it should be understood that the disclosures and enhancements herein are not limited to these materials and manufacturing processes, and can be applicable across a range of suitable materials and manufacturing processes. Thus, the descriptions and figures included herein depict specific implementations to teach those skilled in the art how to make and use the best options. For the purpose of teaching inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations from these implementations that fall within the scope of this disclosure. Those skilled in the art will also appreciate that the features described above can be combined in various ways to form multiple implementations.