Patent Description:
Currently, the lighting efficiency of display devices (e.g., pixelated displays) has been improving by increasing the brightness per unit power or current. However, display devices have a minimum current requirement to achieve a minimum brightness operational state. This minimum brightness operational state makes it difficult to achieve consistent and well-controlled low-end brightness levels (e.g., dim brightness levels) which are required for night-time operations (e.g., low ambient light conditions). Furthermore, the low-end brightness levels are no longer achievable because the brighter, more efficient displays are unstable at low currents, resulting in poor image qualities or the display not turning on at low currents.

The low performance levels and unstable nature of display devices at low current levels (e.g., low brightness/luminance levels) results in displays having to be operated at higher brightness/luminance levels. These higher luminance levels have been found to be incompatible with night-time operations, as the contrast between the highluminance display and the low ambient light surroundings negatively affect a user's night vision and/or the user's ability to see the real-world. Moreover, displaying aircraft symbology video streams overlaid on top of night-vision video streams may obscure the night vision video stream and/or degrade a user's night-adapted vision. Furthermore, the feasible range for dimming the display device for night operations is limited, as the display devices exhibit low image quality and instability at low brightness levels. In the field of avionics, the highest quality video image is of utmost importance when conducting night-time operations (e.g., low ambient light conditions). Accordingly, the inability of display devices to finely control luminance at low levels for use in lowambient light conditions render them ill-suited for use in many aircraft settings. <CIT> discloses a system for displaying first and second images to a pilot, where the second image is overlaid over the first image, and the system is configured to modify the appearance of the second image to enhance the pilot's ability to discern the first image.

Therefore, there exists a need for a system and method which cure one or more of the shortcomings identified above.

A display system for extending a brightness dimming range of a display substrate according to claim <NUM> is provided.

The appearances of the phrase "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, and embodiments may include one or more of the features expressly described.

As noted previously herein, display devices are often required to produce varying levels of brightness/luminance in different ambient lighting conditions. By way of example, a display device may be required to produce higher brightness/luminance levels during daytime operations (e.g., high ambient light conditions) to maintain sufficient image quality for a user. In these high ambient light conditions, the pilot's helmet mounted display (HMD) as well as the aircraft's head-up displays (HUD) must maintain a brightness and contrast high enough to make the displays visible. Therefore, a high luminance level and efficiency is essential during day time operations.

Conversely, a display device may be required to produce lower brightness/luminance levels during night-time operations (e.g., low ambient light conditions) to both maintain a sufficient image quality for a user and so as not to adversely affect a viewer's night-adapted vision or view of the real-world. It has been found that the contrast between high luminance displays and the low ambient light surroundings during night time operations negatively affect a viewer's night vision or view of the real-world. Moreover, displaying aircraft symbology video streams overlaid on top of night-vision video streams may obscure the night vision video stream and/or degrade a user's night-adapted vision. Therefore, in order to allow pilots to maintain eyesight adapted for night vision and situational awareness of the real-world scene during night time operations, displays with low luminance levels are required.

Taken together, display devices which are capable of maintaining high luminance levels for high ambient light conditions and low luminance levels for low ambient light conditions are required. In particular, such display devices are required in aviation, where eyesight and visibility are of utmost importance.

Accordingly, embodiments of the present disclosure are directed to a display system and method for extending a brightness/luminance dimming range of a display device via image frame manipulation. More particularly, embodiments of the present disclosure are directed to extending a brightness/luminance dimming range of a display device by dropping image frames from a video stream and/or selectively modifying luminance levels of individual image frames. By selectively modifying luminance levels of individual image frames, the system and method of the present disclosure may be configured to extend a luminance dimming range of a display device on a time-based averaging basis. Further embodiments of the present disclosure are directed to generating a composite video stream by performing image frame manipulation on two or more video streams, and combining the two or more video streams.

It is contemplated herein that the image frame manipulation techniques of the present disclosure may enable display devices with improved luminance level dimming ranges. In particular, by adjusting a perceived luminance level (e.g., time-averaged luminance level) of a display substrate on a time-based averaging basis via image frame manipulation, the system and method of the present disclosure may enable display devices to effectively fine-tune luminance levels in both high and low luminance level environments. Moreover, by performing image frame manipulation, embodiments of the present disclosure may enable improved luminance dimming range of a display device while maintaining a minimum current requirement to the display device required for continuous and reliable operation.

<FIG> illustrates a simplified block diagram of a display system <NUM> for extending a brightness dimming range of a display substrate <NUM>, in accordance with one or more embodiments of the present disclosure. The display system <NUM> may include, but is not limited to, a display device <NUM>, a display substrate <NUM>, a controller <NUM>, one or more processors <NUM>, and a memory <NUM>. In embodiments, the system <NUM> may further include a user interface <NUM>, one or more video sources <NUM>, and one or more light sensors <NUM>.

In embodiments, the display device <NUM> may include a display substrate <NUM>. The display device <NUM> may include any display device known in the art including, but not limited to, a head-up display (HUD), a head-mounted display (HMD) a helmetmounted display, a head-worn display (HWD), a vehicle-mounted display (e.g., aircraft cockpit display, automobile display), a mobile device display (e.g., smart phone display, handheld display, smart watch display, and the like). In this regard, while much of the present disclosure is directed to a system <NUM> in the context of an aircraft environment (e.g., aircraft cockpit display, HUD, HMD, HWD, and the like), it is contemplated herein that embodiments of the present disclosure may be applied to display devices <NUM> in contexts other than aircraft environments.

In embodiments, the display substrate <NUM> is configured to display at least one image. For example, the display substrate <NUM> may be configured to display one or more video streams including one or more image frames. For instance, as shown in <FIG>, the display substrate <NUM> may be configured to display a composite video stream including a surrounding environment video stream overlaid with an aircraft symbology video stream.

The display substrate <NUM> may include a pixelated display substrate such that the display substrate includes a plurality of pixels. It is contemplated herein that the display substrate <NUM> may include any display substrate known in the art including, but not limited to, an emissive pixelated display substrate (e.g., OLED), a transmissive pixelated display substrate (e.g., LCD), a reflective pixelated display substrate (e.g., DLP), and the like.

It is noted herein that embodiments of the present disclosure are directed to performing image frame manipulation in order to modify a perceived luminance level of the display substrate <NUM> on a time-based averaging basis. In additional embodiments, the time-based averaging techniques of the present disclosure may be combined with techniques configured to modify the perceived luminance level of the display substrate <NUM> on a spatial-based averaging basis. For example, in embodiments where the display substrate <NUM> includes a pixelated display substrate including one or more pixels, the one or more pixels may be further divided up into sub-pixels. Each pixel and/or sub-pixel of the display substrate may be selectively modified via a sub-pixel drive. In this regard, the sub-pixel drive may be configured to selectively actuate sub-pixels in order to modify the perceived luminance level of the display substrate <NUM> on a spatial-based averaging basis. These spatial-based averaging techniques may be combined with the time-based averaging techniques of the present disclosure to further extend and/or modify a brightness/luminance dimming range of the display substrate <NUM>. A sub-pixel drive configured to modify a perceived luminance level of the display substrate <NUM> on a spatial-based averaging basis is described in <CIT>as inventors.

In embodiments, the display device <NUM> and/or the display substrate <NUM> may be communicatively coupled to a controller <NUM>. The display device <NUM> and the display substrate <NUM> may be communicatively coupled to the controller <NUM> using any wireline or wireless communication technique known in the art. In embodiments, the controller <NUM> may include one or more processors <NUM> and a memory <NUM>. Display system <NUM> may further include a user interface <NUM> communicatively coupled to the controller <NUM>, wherein the user interface <NUM> is configured to display information of display system <NUM> to a user and/or receive one or more input commands from a user configured to adjust one or more characteristics of display system <NUM>.

In some embodiments, the display system <NUM> may further include one or more video sources <NUM>. The one or more video sources <NUM> may include any video sources known in the art configured to acquire images and generate a video stream including, but not limited to, a camera (e.g., video camera), a night vision camera (e.g., night vision video camera), an aircraft aerial reconnaissance camera, and the like. For example, the one or more aircraft video sources <NUM> may include a night vision camera configured to acquire and generate a video stream of the surrounding environment of an aircraft (e.g., surrounding environment video stream).

In additional embodiments, the display system <NUM> may include one or more light sensors <NUM>. The one or more light sensors <NUM> may include any light sensors <NUM> known in the art including, but not limited to, ambient light sensors. For example, the one or more light sensors may include at least one of a photoresistor, a photodiode, a phototransistor, a photocell, a photovoltaic light sensor, a photo diode, a lightdependent sensor, and the like. The one or more light sensors <NUM> may be configured to collect ambient light readings associated with the environment of display system <NUM>. For example, in the context of an aircraft, the one or more light sensors <NUM> may be configured to collect ambient light readings within the cockpit of the aircraft, wherein the ambient light readings are indicative of the amount of ambient light experienced by the pilot of the aircraft at a particular point in time. In this regard, continuing with the same example, the one or more light sensors <NUM> may collect high ambient light readings during the day, and low ambient light readings at night.

The one or more processors <NUM> may be configured to execute a set of program instructions stored in memory <NUM>, the set of program instructions configured to cause the one or more processors <NUM> to carry out one or more steps of the present disclosure. For example, the one or more processors <NUM> of the controller <NUM> may be configured to: acquire a video stream including a plurality of image frames; selectively modify one or more characteristics of one or more image frames of the plurality of image frames to generate a modified video stream; and generate one or more control signals configured to cause the display device <NUM> to display the modified video stream via the display substrate <NUM>. Each of the various steps/functions performed by the one or more processors <NUM> of the controller <NUM> will be discussed in further detail herein.

In embodiments, the controller <NUM> may be configured to acquire a video stream including a plurality of image frames. For example, as shown in <FIG>, the controller <NUM> may be configured to receive a video stream from the one or more video sources <NUM>. For instance, the one or more video sources <NUM> of an aircraft may be configured to acquire images/video to generate a video stream of the surrounding environment, and transmit the surrounding environment video stream to the controller <NUM>. For the purposes of the present disclosure, "surrounding environment video stream," and like terms, may be used to refer to a video stream of the environment within which the display system <NUM> and/or display device <NUM> is operating. In the context of an aircraft, a surrounding environment stream may include a video stream of surrounding airspace when the aircraft is in flight, a video stream of the landscape below and/or surrounding the aircraft when the aircraft is in flight, a video stream of the ground/facility/runway when the aircraft is grounded, and the like. The controller <NUM> may be configured to store the received video stream in memory <NUM>.

In additional and/or alternative embodiments, the controller <NUM> may be configured to "acquire" a video stream by generating a video stream. For example, the one or more processors <NUM> of the controller <NUM> may be configured to generate a symbology video stream indicative of one or more metrics or parameters associated with the display system <NUM>, vehicle (e.g., aircraft), or the like. For example, it is noted herein that aircraft and other automobiles commonly use HUD or HMD displays which display data and information related to the aircraft or automobile including, but not limited to, speed, heading, altitude, engine revolutions per minute (RPM), engine temperature, and the like. In this example, a symbology video stream generated by the controller <NUM> may include a video stream which displays data associated with an aircraft in real-time and/or near-real-time. It is further noted herein that symbology video streams may be overlaid on top of real-world sights to achieve augmented reality (e.g., projected onto a window or face mask), as well as combined and/or overlaid on top of other video streams to achieve virtual reality (e.g., overlaid on top of another video stream, such as a surrounding environment video stream).

The controller <NUM> may additionally and/or alternatively be configured to acquire a video stream from one or more external sources. For example, the controller <NUM> may be configured to receive a video stream transmitted from a terrestrial transmitting device (e.g., airport, base station, military base, terrestrial vehicle), an airborne transmitting device (e.g., satellite, aircraft, drone), and the like. In this regard, the video stream received/generated by the controller <NUM> may include any video stream which is to be displayed via the display device <NUM>.

In embodiments, the controller <NUM> is configured to selectively modify one or more characteristics of one or more image frames of a video stream to generate a modified video stream. The modified video stream may then be stored in memory <NUM>. The controller <NUM> is configured to selectively modify one or more characteristics of one or more image frames of a video stream in order to selectively adjust a time-averaged luminance level of the display substrate <NUM>/modified video stream. For example, the controller <NUM> may be configured to "drop" delete, remove, or replace one or more image frames within a video stream. By way of another example, the controller <NUM> may be configured to selectively modify a luminance level (e.g., brightness level) of more image frames from a video stream. Characteristics of image frames which may be selectively modified by the controller <NUM> may include, but are not limited to, the presence/absence of an image frame, a luminance level of an image frame, frequencies of light included within an image frame, and the like.

Selectively modifying characteristics of image frames within a video stream may be further shown and described with reference to <FIG>.

<FIG> illustrates a flowchart of a method 200a for selectively modifying image frames 204a-204n of a video stream <NUM> via image frame dropping. It is noted herein that the steps of method 200a may be implemented all or in part by display system <NUM>. It is further recognized, however, that the method 200b is not limited to the display system <NUM> in that additional or alternative system-level embodiments may carry out all or part of the steps of method 200a.

As noted previously, the controller <NUM> may receive and/or generate a video stream <NUM> including a plurality of image frames 204a, 204b, 204n. For example, as shown in <FIG>, the controller <NUM> may generate an aircraft symbology video stream <NUM> which is configured to display data associated with an aircraft (e.g., speed, altitude, heating, and the like) in real-time and/or near-real-time. For instance, as an aircraft is in flight, the aircraft symbology video stream <NUM> may be configured to continually update and display the current speed, altitude, and heading of the aircraft.

In embodiments, the controller <NUM> may be configured to perform image frame dropping processes <NUM> on the received/generated video stream <NUM> to generate a modified video stream 208a. In this regard, the modified video stream 208a may include one or more original image frames 204a-204n as well as one or more dropped image frames 210a-210n. The one or more dropped image frames 210a-210n may be formed using any technique known in the art. For example, the controller <NUM> may be configured to replace one or more image frames 204a-204n with black (e.g., dark) image frames to generate the one or more dropped image frames 210a-210n. By way of another example, the controller <NUM> may be configured to drop, delete, or otherwise remove one or more image frames 204a-204n on the video stream <NUM>. For instance, as shown in <FIG>, the controller <NUM> may be configured to drop, delete, remove, or replace every third image frame 204a-204n of the video stream <NUM> such that the modified video stream 208a includes one dropped image frame 210a-210n for every two original image frames 204a-204n.

It is noted herein that the eyes of an ordinary user/viewer (e.g., aircraft pilot) typically are not able to perceive individual image frames of a video stream (e.g., video stream <NUM>, modified video stream 210a). This is particularly true in the context of increasingly high frame rate video streams. Indeed, users are typically only capable of viewing a video stream in the aggregate as a sum total of the individual image frames. In this regard, the luminance level (e.g., brightness) of a display substrate <NUM>, as it is perceived by a user, may be defined as a time-averaged luminance level of the individual image frames of the video stream. In other words, a perceived luminance level of a display substrate <NUM> may be defined as an average luminance level of the individual image frames of the video stream being displayed over a defined time period, where higher perceived luminance levels are indicative of higher brightness, and lower perceived luminance levels are indicative of lower brightness.

By including dropped image frames 210a-210n within the modified video stream 208a, which may appear dark/black, the modified video stream 208a may appear to exhibit a lower perceived luminance level (time-averaged luminance level) when displayed via the display substrate <NUM> as compared to the original video stream <NUM>. In particular, as it is perceived by a user, time-averaging effects while viewing the modified video stream 208a result in a lower "perceived luminance level" (e.g., time-averaged luminance level) as compared to the original video stream <NUM>.

The difference in time-averaged luminance levels (e.g., perceived luminance levels) between the video stream <NUM> and the modified video stream 208a is a function of the ratio of dropped image frames 210a-210n to original (un-dropped) image frames 204a-204n. A higher ratio of dropped image frames 210a-210n to original image frames 204a-204n (e.g., more dropped image frames <NUM>) may result in a modified video stream 208a with a lower time-averaged luminance level, whereas lower ratio of dropped image frames 210a-210n to original image frames 204a-204n (e.g., fewer dropped image frames <NUM>) may result in a modified video stream 208a with a higher time-averaged luminance level as compared to the higher ratio of dropped image frames. It is further noted, however, that any number of dropped image frames <NUM> may result in a lower luminance level as compared to the original video stream. Accordingly, the controller <NUM> may be configured to selectively drop any number of image frames 204a-204n from the video stream <NUM> in order to achieve a modified video stream 208a with a desired/selected time-averaged luminance level.

The controller <NUM> may be further configured to selectively modify image frames <NUM> of a video stream <NUM> to adjust a time-averaged luminance level (e.g., perceived luminance level) of a display substrate <NUM> by selectively modifying luminance levels of individual image frames <NUM> of the video stream <NUM>. This may be further understood with reference to <FIG>.

<FIG> illustrates a flowchart of a method 200b for selectively modifying image frames of a video stream <NUM> via image frame luminance level adjustment, in accordance with one or more embodiments of the present disclosure. It is noted herein that the steps of method 200b may be implemented all or in part by display system <NUM>. It is further recognized, however, that the method 200b is not limited to the display system <NUM> in that additional or alternative system-level embodiments may carry out all or part of the steps of method 200b.

As noted previously, the controller <NUM> may receive and/or generate a video stream <NUM> including a plurality of image frames 204a, 204b, 204n. In embodiments, the controller <NUM> may be configured to perform image frame luminance level adjustment processes <NUM> on the received/generated video stream <NUM> to generate a modified video stream 208b. In this regard, the modified video stream 208b may include one or more original image frames 204a-204n as well as one or more luminance-altered image frames 214a-214n. For example, the controller <NUM> may be configured to adjust the luminance level of one or more image frames 204a-204n on the video stream <NUM>. For instance, as shown in <FIG>, the controller <NUM> may be configured to adjust a luminance level of every other image frame 204a-204n of the video stream <NUM> such that the modified video stream 208b includes one luminance-altered image frame 214a-214n for every original image frame 204a-204n.

As noted previously herein with respect to image frame dropping in <FIG>, image frame luminance level adjustment in <FIG> may effectively adjust (e.g., decrease, increase) the time-averaged luminance level (e.g., perceived luminance level) of the modified video stream 208b displayed on the display substrate <NUM> due to time-averaging effects.

While <FIG> and <FIG> illustrate the controller <NUM> selectively modifying image frames <NUM> by either image frame dropping or luminance level adjustment, this is not to be regarded as a limitation of the present disclosure, unless noted otherwise herein. In this regard, the controller <NUM> may be configured to perform a combination of image frame dropping and luminance level adjustment on various image frames <NUM> of a video stream <NUM> in order to more precisely achieve a desired or selected time-averaged luminance level. For example, it is contemplated herein that dropping a large percentage of image frames <NUM> may cause a user to perceive a "flickering" effect on the display substrate <NUM>. Thus, there may be a practical limit as to how many image frames <NUM> may be dropped completely. However, by performing a combination of image frame dropping and luminance level adjustment, the controller <NUM> may be able to achieve a sufficiently low time-averaged luminance level without introducing a "flickering" effect which is perceptible by a user.

In embodiments, the controller <NUM> may be further configured to generate one or more control signals configured to cause the display device <NUM> to display the modified video stream <NUM> via the display substrate <NUM>. For example, the controller <NUM> may be configured to generate one or more control signals configured to cause the display substrate <NUM> of the display device <NUM> to display the modified video stream 208a illustrated in <FIG>. By way of another example, the controller <NUM> may be configured to generate one or more control signals configured to cause the display substrate <NUM> of the display device <NUM> to display the modified video stream 208b illustrated in <FIG>.

As noted previously herein the controller <NUM> may be configured to selectively modify characteristics of individual image frames <NUM> of a video stream <NUM> in order to selectively modify/adjust a time-averaged luminance level (e.g., perceived luminance level) of the display substrate <NUM> as it displays the modified video stream 208a, 208b. For example, by displaying a modified video stream 208a, 208b, the controller <NUM> may be configured to cause the display device <NUM> to exhibit a lower time-averaged luminance level (e.g., perceived luminance level) as would be the case if the original video stream <NUM> were to be displayed.

Adjusting a luminance level (e.g., brightness) of the display substrate <NUM> via image frame manipulation, as described herein, may enable many advantages over previous techniques. As noted previously herein, a display device <NUM> may be required to produce higher brightness/luminance levels during daytime operations (e.g., high ambient light conditions) to maintain sufficient image quality for a user, as well as lower brightness levels during night-time operations (e.g., low ambient light conditions) to both maintain a sufficient image quality for a user and so as not to adversely affect a viewer's night vision. By selectively modifying individual image frames <NUM> of a video stream <NUM>, the display system <NUM> of the present disclosure may enable the display substrate <NUM> to exhibit high-brightness during high ambient light conditions, as well as lowbrightness during low ambient light conditions. Improvements in the dynamic range of the display substrate <NUM> may be particularly important for some mission profiles, such as covert operations, and black hole approaches to airports, aircraft carriers, or other stealth-type landing zones.

Moreover, as noted previously herein, modern display devices <NUM> typically exhibit a minimum current requirement to achieve a minimum brightness operational state. This minimum brightness operational state makes it difficult to achieve the low-end brightness levels (e.g., low luminance levels) which are required for night-time operations. Accordingly, the display system <NUM> and method of the present disclosure may enable dynamic dimming range improvements of a display substrate <NUM> while simultaneously providing sufficient current to the display device <NUM> to ensure efficient and reliable operation. In particular, by modifying characteristics of individual image frames <NUM>, the controller <NUM> of the display system <NUM> may effectively reduce the time-averaged luminance level of the display substrate <NUM> while not overly restricting the current provided to the display device <NUM>. In this regard, the controller <NUM> may effectively improve the dimming range of the display substrate <NUM> to achieve time-averaged low luminance levels below the minimum brightness level of any single frame, while simultaneously meeting a minimum current requirement to achieve a minimum brightness operational state of the display device <NUM>.

In some embodiments, the display system <NUM> may be configured to adaptively modify the time-averaged luminance level of the display substrate <NUM> in response to changing ambient light readings. As noted previously herein, for optimal performance, a display substrate <NUM> may be operated at high luminance levels during high ambient light conditions (e.g., daytime), and may further be operated at low luminance levels during low ambient light conditions (e.g., at night). In this regard, the controller <NUM> may be configured to adjust a time-averaged luminance level (e.g., perceived luminance level) of the display substrate <NUM> ("display substrate luminance level") in response to one or more collected ambient light readings by selectively modifying one or more characteristics of one or more image frames <NUM>.

For example, at night, the one or more light sensors <NUM> may collect ambient light readings indicating low ambient light conditions (e.g., low ambient light readings). The controller <NUM> may then be configured to selectively modify one or more characteristics of one or more image frames <NUM> of a video stream <NUM> in order to lower the time-averaged luminance level of the display substrate <NUM> in response to the low ambient light reading. For instance, the controller <NUM> may be configured to drop one or more image frames <NUM> to generate one or more dropped image frames <NUM> and/or modify a luminance level of one or more image frames <NUM> to generate one or more luminance-altered image frames <NUM> with decreased luminance levels. By selectively modifying individual image frames <NUM>, the controller <NUM> may be configured to lower the time-averaged luminance level of the display substrate <NUM> based on the low ambient light readings.

By way of another example, during the daytime, the one or more light sensors <NUM> may collect ambient light readings indicating high ambient light conditions (e.g., high ambient light readings). The controller <NUM> may then be configured to selectively modify one or more characteristics of one or more image frames <NUM> of a video stream <NUM> in order to increase the time-averaged luminance level of the display substrate <NUM> in response to the low ambient light reading. For instance, the controller <NUM> may be configure to cease dropping image frames from the video stream <NUM> in order to increase the time-averaged luminance level. Additionally and/or alternatively, the controller <NUM> may be configured to modify a luminance level of one or more image frames <NUM> to generate one or more luminance-altered image frames <NUM> with increased luminance levels.

In embodiments, the controller <NUM> may be configured to selectively alter/drop one or more image frames <NUM> depending on a comparison of collected ambient light readings to ambient light threshold values. For example, ambient light readings above an ambient light threshold value may be associated with a "day time mode" with a high display substrate luminance level, and ambient light readings below the ambient light threshold value may be associated with a "night time mode" with a low display substrate luminance level. For instance, the controller <NUM> may be configured to lower a time-averaged luminance level by dropping frames and/or decreasing a luminance level of one or more image frames <NUM> in response to collected ambient light readings below an ambient light threshold value. Conversely, the controller <NUM> may be further configured to increase a time-averaged luminance level by ceasing to drop frames and/or increasing a luminance level of one or more image frames <NUM> in response to collected ambient light readings above an ambient light threshold value.

While ambient light readings are described as being compared to a single ambient light threshold for a "day time mode" and a "night time mode," this is not to be regarded as a limitation of the present disclosure. In this regard, display system <NUM> may be configured to compare ambient light readings to any number of ambient light thresholds such that the display substrate <NUM> may be operated in a plurality of display "modes. " For example, ambient light readings below a first ambient light threshold may be indicative of a "low brightness mode" or "night time mode," ambient light readings below the first ambient light threshold and below a second ambient light threshold may be indicative of an "intermediate brightness mode," and ambient light readings above the second ambient light threshold may be indicative of a "high brightness mode" or "day time mode.

<FIG> illustrates a flowchart of a method <NUM> for combining modified video streams <NUM> generated via image frame manipulation processes <NUM>, in accordance with one or more embodiments of the present disclosure.

In addition to selectively modifying characteristics of image frames <NUM> within a single video stream <NUM>, the display system <NUM> of the present disclosure may be further configured to generate one or more modified video streams <NUM>, and combine the one or more modified video streams <NUM> with one or more additional video streams in order to generate a composite video stream <NUM>.

It is noted herein that the composite video stream <NUM> may be generated by combining two or more video streams using any techniques known in the art including, but not limited to, overlaying multiple video streams, combining video streams in a "picture-in-picture" combined layout, abutting video streams next to one another, and the like.

For example, as shown in <FIG>, the controller <NUM> may be configured to receive a first video stream 202a. For instance, the one or more video sources <NUM> of the display system <NUM> may be configured to acquire a video stream of the surrounding environment of an aircraft. In this regard, the first video stream 202a may include a surrounding environment video stream 202a which depicts landscapes and other views viewable by a pilot of an aircraft and/or the video sources <NUM>.

Additionally, the controller <NUM> may be configured to receive a second video stream 202b. For instance, the controller <NUM> may be configured to generate/receive a video stream 202b which displays data and information related to the aircraft or automobile including, but not limited to, speed, heading, altitude, engine revolutions per minute (RPM), engine temperature, and the like. In this example, the second video stream 202b may include a symbology video stream 202b which displays data associated with an aircraft in real-time and/or or near-real-time.

Continuing with reference to <FIG>, the controller <NUM> may be configured to carry out one or more image frame manipulation processes <NUM> on the first video stream 202a (e.g., surrounding environment video stream 202a) and the second video stream 202b (e.g., symbology video stream 202b). In this regard, the controller <NUM> may be configured to selectively modify one or more characteristics of one or more image frames <NUM> of the first video stream 202a and/or the second video stream 202b. The one or more image frame manipulation processes <NUM> may include, but are not limited to, image frame dropping processes <NUM> (<FIG>), and image frame luminance level adjustment processes <NUM> (<FIG>).

For example, as shown in <FIG>, the controller <NUM> may be configured selectively adjust a luminance level of one or more image frames <NUM> of the first video stream 202b in order to generate a first modified video stream 208a including one or more luminance-altered image frames <NUM>. Similarly, the controller <NUM> may be configured to selectively drop one or more image frames <NUM> of the second video stream 202b in order to generate a second modified video stream 208b including one or more dropped image frames <NUM>.

In some embodiments, the controller <NUM> may be configured to selectively manipulate image frames of one video stream <NUM> in order to match, or approximately match, a luminance level of another video stream. For example, the controller <NUM> may be configured to drop one or more image frames <NUM> from the first video stream 202a (e.g., surrounding environment video stream 202a) to generate the first modified video stream 208a. The controller <NUM> may then be configured to determine a time-averaged luminance level (e.g., perceived luminance level) of the first video stream 202a (e.g., surrounding environment video stream 202a). Subsequently, the controller <NUM> may be configured to selectively modify one or more characteristics of the second video stream 202b (e.g., symbology video stream 202b) in order to generate the second modified video stream 208b which exhibits an equivalent, or substantially equivalent, time-averaged luminance level as the first modified video stream 208b.

It is contemplated herein that approximately matching luminance levels of video streams which are to be combined may prevent situations in which a heightened luminance level of a symbology video stream obscures a user's ability to view the surrounding environment and/or another video stream displayed on the display substrate <NUM>.

In embodiments, the controller <NUM> may then be further configured to carry out video stream combining processes <NUM> in order to combine the first modified video stream 208a and the second modified video stream 208b to generate a composite video stream <NUM>. The modified video streams 208a, 208b may be combined using any techniques known in the art. For instance, in the context of a surrounding environment video stream (e.g., first modified video stream 208a) and a symbology video stream (e.g., second modified video stream 208b), the two modified video streams 208a, 208b may be combined by overlaying the symbology video stream on top of the surrounding environment video stream. By way of another example, the first modified video stream 208a and the second modified video stream 208b may be combined in a "picture-in-picture" format where the second modified video stream 208b is inlaid within the first modified video stream 208a. By way of another example, the first modified video stream 208a and the second modified video stream 208b may be combined by abutting the modified video streams 208a, 208b adjacent to one another, where the second modified video stream 208b is disposed adjacent to the first modified video stream 208a (e.g., vertical "split screen," horizontal "split screen," and the like). It is further noted herein that the composite video stream <NUM> generated by display system <NUM> may be generated by combining any number of video streams. In another embodiment, the controller <NUM> may be configured to generate one or more control signals configured to cause the display device <NUM> to display the composite video stream <NUM> via the display substrate <NUM>.

It is noted herein that dropping one or more image frames from second video stream 202b (e.g., symbology video stream 202b), while simply lowering the luminance level of image frames within the first video stream 202a (e.g., surrounding environment video stream 202a), the controller <NUM> may lower the "effective frame rate" of the modified symbology video stream 208b with respect to the modified surrounding environment video stream 208a. It is contemplated that night vision video streams (e.g., surrounding environment video stream 202a, modified surrounding environment video stream 208a) may be required to be shown at a high effective frame rate in order to minimize effects of smearing, image ghosting, and motion blur. However, symbology video streams (e.g., symbology video stream 202b, modified symbology video stream 208b) may be shown at a lower effective frame rate, as shown in <FIG>.

In some embodiments, the one or more image frame manipulation processes <NUM> performed on the first video stream 202a and/or the second video stream 202b may be performed in order to achieve a particular time-averaged luminance level of the composite video stream <NUM> displayed on the display substrate <NUM>. For example, the controller <NUM> may receive one or more ambient light readings from the one or more light sensors <NUM>. Based on the received ambient light readings, the controller <NUM> may be configured to determine a desired time-averaged luminance level of the display substrate <NUM> which will optimize a user's ability to view both the display substrate <NUM> and the surrounding real-world environment without adversely affecting a user's night-adapted vision in low ambient light conditions. Upon determining an optimal (e.g., desired) time-averaged luminance level, the controller <NUM> may perform the one or more image frame manipulation processes <NUM> on the first video stream 202a and/or the second video stream 202b in order to generate the composite video stream <NUM> which exhibits the desired time-averaged luminance level.

It is noted herein that the controller <NUM> may continually adjust and modify the one or more image frame manipulation processes <NUM> performed on the first video stream 202a and/or the second video stream 202b over time in response to changing ambient light conditions. In this regard, the one or more steps/functions carried out by the controller <NUM> on the video streams <NUM> may change and evolve over time.

Generally referring to <FIG>, a display substrate <NUM> displaying combined video streams 220a-220c are shown and described. In particular, <FIG> illustrate combined video streams 220a-220c generated by overlaying a second video stream 202b (e.g., symbology video stream 202b) on top of a first video stream 202a (e.g., surrounding environment video stream 202a). However, as noted previously herein, a combined video stream <NUM> may be generated by combining two or more video streams using any techniques known in the art including, but not limited to, overlaying multiple video streams, combining video streams in a "picture-in-picture" combined layout, abutting video streams next to one another, and the like. Accordingly, the overlay techniques shown in <FIG> are provided solely as examples, and are not to be regarded as limiting, unless noted otherwise herein.

<FIG> illustrates a display substrate <NUM> displaying a composite video stream 220a, in accordance with one or more embodiments of the present disclosure. In particular, the composite video stream 220a may include an un-modified first video stream 202a (e.g., surrounding environment video stream 202a) and an un-modified second video stream 202b (e.g., symbology video stream 202b). As shown in <FIG>, the symbology video stream 202b may be overlaid on top of the surrounding environment video stream.

The surrounding environment video stream 202a and the symbology video stream 202b illustrated in <FIG> may be un-modified in that the controller <NUM> has not dropped image frames and/or dimmed luminance level of image frames within the respective video streams 202a, 202b (e.g., no image frame manipulation processes <NUM>). In this regard, each of the surrounding environment video stream 202a and the symbology video stream 202b may exhibit a "full" or high luminance level. Such high luminance levels may be used in the context of high ambient light conditions, and in conjunction with high ambient light readings collected by the one or more light sensors <NUM>.

In low ambient light conditions, maintaining the surrounding environment video stream 202a and/or the symbology video stream 202b at a high time-averaged luminance level may obscure the other video stream and/or inhibit a user's (e.g., pilot's) ability to view the real-world surroundings. For example, maintaining the symbology video stream 202b at a high luminance level may obstruct the user's ability to see the surrounding environment video stream 202a, as well as adversely affect the user's night-adapted vision and which inhibits the user's ability to see the real-world surroundings. In this regard, the controller <NUM> may be configured to dim the symbology video stream 202b, as shown in <FIG>.

<FIG> illustrates a display substrate <NUM> displaying a composite video stream 220b generated by performing image frame manipulation processes <NUM> on one or more video streams <NUM> of the composite video stream 220b, in accordance with one or more embodiments of the present disclosure.

More particularly, the composite video stream 220b may include an un-modified surrounding environment video stream 202a and a modified symbology video stream 208b. The modified symbology video stream 208b may have been generated by performing one or more image frame manipulation processes <NUM> (e.g., image frame dropping, image frame luminance level dimming) on the un-modified symbology video stream 202a illustrated in <FIG>. In lowering the time-averaged luminance level of the modified symbology video stream 208b, the controller <NUM> may effectively lower the time-averaged luminance level of the composite video stream 220b, and thus improve a user's ability to view the display substrate <NUM> in low ambient light conditions.

Extremely low ambient light conditions may require even lower time-averaged luminance levels of the display substrate <NUM>. For example, during covert operations and/or black hole approaches, the controller <NUM> may be configured to lower the time-averaged luminance level of the display substrate <NUM> by selectively modifying image frames of the surrounding environment video stream 202a and the symbology video stream 202b, as shown in <FIG>.

<FIG> illustrates a display substrate <NUM> displaying a composite video stream 220c generated by performing image frame manipulation processes <NUM> on one or more video streams <NUM> of the composite video stream 220c, in accordance with one or more embodiments of the present disclosure.

More particularly, the composite video stream 220c may include a modified surrounding environment video stream 208a and a modified symbology video stream 208b. The modified surrounding environment video stream 208a and the modified symbology video stream 208b may have been generated by performing one or more image frame manipulation processes <NUM> (e.g., image frame dropping, image frame luminance level dimming) in order to lower the time-averaged luminance level of the display substrate <NUM>. In lowering the time-averaged luminance level of the modified surrounding environment video stream 208a and the modified symbology video stream 208b, the controller <NUM> may effectively lower the time-averaged luminance level of the composite video stream 220c, and thus improve a user's ability to view the display substrate <NUM> in extremely low ambient light conditions.

It is noted herein that the one or more components of display system <NUM> may be communicatively coupled to the various other components of display system <NUM> in any manner known in the art. For example, the display substrate <NUM>, the controller <NUM>, the one or more processors <NUM>, the memory <NUM>, the user interface <NUM>, the one or more video sources <NUM>, and/or the one or more light sensors <NUM> may be communicatively coupled to each other and other components via a wireline (e.g., copper wire, fiber optic cable, and the like) or wireless connection (e.g., RF coupling, IR coupling, WiFi, WiMax, Bluetooth, <NUM>, <NUM>, <NUM> LTE, <NUM>, and the like).

In one embodiment, the one or more processors <NUM> may include any one or more processing elements known in the art. In this sense, the one or more processors <NUM> may include any microprocessor-type device configured to execute software algorithms and/or instructions. In one embodiment, the one or more processors <NUM> may consist of a desktop computer, mainframe computer system, workstation, image computer, parallel processor, a field-programmable gate array (FPGA), multi-processor system-on-chip (MPSoC), or other computer system (e.g., networked computer) configured to execute a program configured to operate the display system <NUM>, as described throughout the present disclosure. It should be recognized that the steps described throughout the present disclosure may be carried out by a single computer system or, alternatively, multiple computer systems. In general, the term "processor" may be broadly defined to encompass any device having one or more processing elements, which execute program instructions from memory <NUM>. Moreover, different subsystems of the display system <NUM> (e.g., display device <NUM>, user interface <NUM>, video source <NUM>, light sensors <NUM>) may include one or more processor or logic elements suitable for carrying out at least a portion of the steps described throughout the present disclosure. Therefore, the above description should not be interpreted as a limitation on the present disclosure but merely an illustration.

The memory <NUM> may include any storage medium known in the art suitable for storing program instructions executable by the associated one or more processors <NUM>. For example, the memory <NUM> may include a non-transitory memory medium. For instance, the memory <NUM> may include, but is not limited to, a read-only memory (ROM), a random-access memory (RAM), a magnetic or optical memory device (e.g., disk), a magnetic tape, a solid-state drive and the like. It is further noted that memory <NUM> may be housed in a common controller housing with the one or more processors <NUM>. In an alternative embodiment, the memory <NUM> may be located remotely with respect to the physical location of the processors <NUM> and controller <NUM>. In another embodiment, the memory <NUM> maintains program instructions for causing the one or more processors <NUM> to carry out the various steps described through the present disclosure.

In another embodiment, the controller <NUM> is coupled to a user interface <NUM>. In another embodiment, the user interface includes a display and/or a user input device. For example, the display device may be coupled to the user input device by a transmission medium that may include wireline and/or wireless portions. The display device of the user interface <NUM> may include any display device known in the art. The display device of the user interface <NUM> may include the display device <NUM> or additional and/or alternative display devices. For example, the display device may include, but is not limited to, a liquid crystal display (LCD), an organic light-emitting diode (OLED) based display, a CRT display, and the like. Those skilled in the art should recognize that a variety of display devices may be suitable for implementation in the present invention and the particular choice of display device may depend on a variety of factors, including, but not limited to, form factor, cost, and the like. In a general sense, any display device capable of integration with a user input device (e.g., touchscreen, bezel mounted interface, keyboard, mouse, trackpad, and the like) is suitable for implementation in the present invention.

The user input device of the user interface <NUM> may include any user input device known in the art. For example, the user input device may include, but is not limited to, a keyboard, a keypad, a touchscreen, a lever, a knob, a scroll wheel, a track ball, a switch, a dial, a sliding bar, a scroll bar, a slide, a handle, a touch pad, a paddle, a steering wheel, a joystick, a bezel input device, or the like. In the case of a touchscreen interface, those skilled in the art should recognize that a large number of touchscreen interfaces may be suitable for implementation in the present invention. For instance, the display device may be integrated with a touchscreen interface, such as, but not limited to, a capacitive touchscreen, a resistive touchscreen, a surface acoustic based touchscreen, an infrared based touchscreen, or the like. In a general sense, any touchscreen interface capable of integration with the display portion of a display device is suitable for implementation in the present invention. In another embodiment, the user input device may include, but is not limited to, a bezel mounted interface.

<FIG> illustrates a flowchart of a method <NUM> for extending a brightness dimming range of a display substrate <NUM>, in accordance with one or more embodiments of the present disclosure. It is noted herein that the steps of method <NUM> may be implemented all or in part by system <NUM>. It is further recognized, however, that the method <NUM> is not limited to the system <NUM> in that additional or alternative system-level embodiments may carry out all or part of the steps of method <NUM>.

In a step <NUM>, a first video stream including a plurality of image frames is acquired. For example, as shown in <FIG>, the controller <NUM> may receive a surrounding environment video stream 202a including a plurality of image frames <NUM>. The surrounding environment video stream 202a may be acquired by one or more video sources <NUM> communicatively coupled to the controller <NUM>.

In a step <NUM>, a second video stream including a plurality of image frames is acquired. For example, as shown in <FIG>, the controller <NUM> may be configured to generate a symbology video stream 202b including a plurality of image frames <NUM>. The symbology video stream 202a may depict data and information related to the aircraft or automobile including, but not limited to, speed, heading, altitude, engine revolutions per minute (RPM), engine temperature, and the like. In this regard, the symbology video stream 202b may display data associated with an aircraft in real-time and/or or near-real-time.

In a step <NUM>, one or more characteristics of one or more image frames of the first video stream are selectively modified to generate a first modified video stream. For example, the controller <NUM> may be configured to perform one or more image frame manipulation processes <NUM> on the surrounding environment video stream 202a to generate a modified surrounding environment video stream 208a. For instance, the controller <NUM> may be configured to drop one or more image frames <NUM> from the surrounding environment video stream 202a and/or adjust a luminance level of one or more image frames <NUM> of the surrounding environment video stream 202a. It is noted herein that performing one or more image frame manipulation processes <NUM> may effectively adjust a time-averaged luminance level (e.g., perceived luminance level) of the modified surrounding environment video stream 202a.

In a step <NUM>, one or more characteristics of one or more image frames of the second video stream are selectively modified to generate a second modified video stream. For example, the controller <NUM> may be configured to perform one or more image frame manipulation processes <NUM> on the symbology video stream 202b to generate a modified symbology video stream 208b. For instance, the controller <NUM> may be configured to drop one or more image frames <NUM> from the symbology video stream 202b and/or adjust a luminance level of one or more image frames <NUM> of the symbology video stream 202b.

While method <NUM> is shown and described as selectively modifying image frames <NUM> of both the surrounding environment video stream 202a and the symbology video stream 202b, this is not to be regarded as limiting, unless noted otherwise herein. In this regard, it is contemplated that the controller <NUM> may be configured to modify any number of video streams. For example, in some instances, the controller <NUM> may perform image frame manipulation processes <NUM> only on the symbology video stream 202b. By way of another example, in other instances, in some instances, the controller <NUM> may perform image frame manipulation processes <NUM> only on the surrounding environment video stream 202a.

In a step <NUM>, the first modified video stream and the second modified video stream are combined. As noted previously herein, the composite video stream <NUM> may be generated by combining two or more video streams using any techniques known in the art including, but not limited to, overlaying multiple video streams, combining video streams in a "picture-in-picture" combined layout, abutting video streams next to one another, and the like. For example, the controller <NUM> may be further configured to carry out video stream combining processes <NUM> in order to combine the modified surrounding environment video stream 208a and the modified symbology video stream 208b to generate a composite video stream <NUM>. For instance, the modified symbology video stream 208b may be overlaid on top of the modified surrounding environment video stream 208a.

Claim 1:
A display system (<NUM>) for extending a brightness dimming range of a display substrate (<NUM>), comprising:
one or more video sources (<NUM>); wherein the one or more video sources are configured to generate a first video stream (202a) of a surrounding environment of an aircraft;
a display device (<NUM>), the display device including a display substrate (<NUM>) configured to display at least one image; and
a controller (<NUM>) communicatively coupled to the display substrate, the controller including one or more processors (<NUM>) configured to execute a set of program instructions stored in a memory (<NUM>), the set of program instructions configured to cause the one or more processors to:
generate a second video stream (202b); wherein the second video stream is a symbology video stream configured to display data associated with the aircraft;
selectively adjust a luminance level of one or more image frames (<NUM>) of the first video stream (202a) in order to generate a first modified video steam (208a) including one or more luminance-altered image frames (<NUM>);
selectively drop one or more image frames (<NUM>) of the second video stream (202b) in order to generate a second modified video stream (208b) including one or more dropped image frames (<NUM>); wherein the controller (<NUM>) is configured to replace the one or more image frames with black image frames to generate the one or more dropped image frames (<NUM>);
combine the first modified video stream (208a) and the second modified video stream (208b) to generate a composite video stream (<NUM>); and
generate one or more control signals configured to cause the display device to display the composite video stream via the display substrate.