Patent Description:
Analog NV systems function by receiving low levels of light and intensifying the received light using an image intensifier. The image intensifier has a photocathode that emits electrons in response to incident photons. The emitted electrons are accelerated through a vacuum tube and directed towards a microchannel plate that amplifies the signal by multiplying the number of electrons. The multiplied electrons then strike a phosphor screen, and, via the phenomenon of luminescence, the phosphor screen emits photons in response to radiant energy (e.g., the electrons). The luminescent light from the phosphor screen is coupled through a series of optics to the user. For example, the luminescent light may be coupled through an inverting fiber optic to an eyepiece where the user can view the illuminated phosphor screen, thus allowing the user to see the objects.

Analog NV systems can include an overlay display that transmits a direct-view, intensified image through the overlay display and emits display light representing a display image from the overlay display to thereby generate a combined image with the display image superimposed over the direct-view, intensified image. The overlay display can be used to convey various information to the user, such as temperatures, distances, indicators marking objects, situational awareness messages, messages from other users, etc..

A challenge of adding an overlay display to an analog NV system is that this addition can increase the size, weight, and power of the analog NV system. For example, if each pixel of the display image is determined by an external controller and the entire display image is updated each time the frame is updated (i.e., at the frame rate), then considerable power will be consumed by the communications interfaces between the external controller and the overlay display because each pixel value will be communicated via the communications interfaces during each period of the frame rate. Accordingly, improved analog NV systems and overlay displays are desired that more efficiently control and update the display image.

From <CIT> a night vision system is known. The document discloses in particular an image intensifier tube and a method of manufacturing such tube. The image intensifier tube comprises doped electron multipliers and addressable electron emitters that generate multiplied secondary electrons and addressed electron emissions from a backside surface of one or more transmission mode secondary electron (TMSE) image intensifiers shown as a display within the tube. In other words, the night vision systems comprise an electronically addressable display within the analog image intensifier tube itself. Symbology and images of the same or different wavelength modalities are sent to one or more semiconductor-based electron multipliers within the analog image intensifier tube.

One embodiment illustrated herein includes an optical device that includes a semiconductor chip configured to receive first light of an underlying image and superimpose second light of a display image over the underlying image, according to the appended claims.

As discussed above, the size, weight, and power of an analog night vision (NV) system may be reduced by using a more efficient overlay display. For example, the NV system can be improved by using more efficient techniques to determine and update the pixel values in the overlay display image and communicate data indicating the pixel values to be displayed from an external controller. For example, rather than determining the display image and pixel values using an external controller and communicating those pixel values from the external controller using a high-speed parallel communication interface, the pixel values are controlled by a specialized frame driver, which is either on the same chip as the overlay display or at least on the same PC board.

The frame driver includes a local frame buffer containing a memory array that represents the pixel values of each pixel in a fixed order. Updating the display image is then performed by locally reading these pixel values out at desired frame rate. The frame rate may be separately adjustable relative to the rate at which the values in the frame buffer are updated/changed. Further, the communication bandwidth required for updating the image can be reduced by only communicating those pixel values that change relative to the previous frame. Thus, the local frame buffer reduces the volume of information to be communicated from (and to) an external controller.

In certain embodiments, the change in pixel values can be determined locally, as opposed to being communicated from an external controller. For example, the overlay display may include an array of detectors, and information or symbols being displayed may be determined based on the signals detected by the array of detectors. Rather than sending the detected signals to an external controller and then receiving the information or symbols to be displayed from the external controller, a local processor can process the detected signals and generate therefrom the information or symbols to be displayed. The local processor may be on the same chip as the overlay display or at least on the same PC board. Examples of information or symbols derived from detected signals may include, e.g., an outline designating an object of interest, which was determined based on edge detection or change detection performed on an acquired image (i.e., the detected signals), a text message derived from a modulated infrared light (e.g., infrared (IR) beacon modulated using pulse frequency code modulation). Additional examples are discussed below.

Generally, each of the techniques described herein reduces the amount of information needed from an external controller to determine pixel values in the display image. Without these techniques, an NV system having an integrated overlay display tends to draw a lot of electrical power when the overlay display uses a high-speed parallel data interface to update the entire pixel array at the specified frame rate (e.g., <NUM>) regardless of whether the value of a particular pixel changes from frame to frame.

For many applications using an overlay display, the underlying image over which the overlay display is superimposed is of primary importance. Accordingly, in some embodiments, the overlay image is only superimposed part of the time (e.g., much of the time, the pixel array will be held to zero, i.e., off) or is only superimposed over a small fraction of the pixels in the overlay display (e.g., a small percentage of pixel have non-zero pixel values). This enables a user to see the underlying image as clearly as possible with as few obstructions as possible. For NV systems, enabling the user to see the underlying image can be important when the underlying image is of primary significance and the overlay display serves a secondary function as an augmented reality accentuator.

As mentioned above, using a high-speed parallel data interface to update the entire pixel array at the specified frame rate (e.g., <NUM>) may be inefficient when the display image is sparse (e.g., few pixel values are non-zero) and/or mostly static (e.g., the pixel values change infrequently). This inefficiency is especially problematic in NV systems due to the premium on size, weight, and power. For example, in a covert operation in which a war fighter uses a head-mounted, battery-powered NV system, bad results may occur if the battery is exhausted before the covert operation is complete (e.g., a war fighter should not need to change batteries in the middle of a fire fight, or have to carry batteries rather than other necessities). Further, weight is important because the head mounted NV system applies torque to the user's neck, potentially causing fatigue and injury unless the weight is kept reasonably low.

To reduce size, weight, and power, the embodiments disclosed herein reduce (or eliminate) the data required from an external controller to determine the pixel values of a display image. For example, the embodiments disclosed herein avoid the requirement for a high-speed parallel data interface that updates the entire pixel array at the specified frame rate. One technique to reduce the interface data rate is to update only the value of those pixels that change from one frame to the next. Another technique is to determine some or all the pixel values using a local processor. A third technique is to use a high-level language to minimize the amount to data to be communicated across the data interface (e.g., providing text as a character string), and then a local processor converts the received data in the high-level language to low-level pixel values.

These techniques are advantageous for many reasons. As discussed above, these techniques reduce the size, weight, and power of the NV system. Additionally, these techniques may reduce electromagnetic interference due to the communications interface.

Referring now to <FIG>, a non-limiting example of a NV system is illustrated. In particular, <FIG> illustrate a PVS - <NUM> NV system <NUM>. In the example illustrated, the NV system <NUM> includes a housing <NUM>, which houses an image intensifier module <NUM>. The NV system <NUM> further includes an objective <NUM> which receives light from an environment. The objective <NUM> includes optics such as lenses, waveguides, and/or other optical components for receiving and transmitting light to the image intensifier module <NUM>. The NV system <NUM> further includes an eyepiece <NUM>. The eyepiece <NUM> includes optics for focusing images created by the NV system <NUM> into the eye of the user.

<FIG> illustrates the image intensifier module <NUM>, according to one example. The image intensifier module <NUM> includes an image intensifier <NUM> without an overlay display. The light from the image intensifier module <NUM> is captured by the eyepiece <NUM> and directed to the user.

The image intensifier module <NUM> receives the input light <NUM>, which has been transmitted through the objective <NUM> to the image intensifier module <NUM>. The input light <NUM> may be dim light from a nighttime environment that would be challenging to see with the naked eye.

The objective directs the input light <NUM> into the image intensifier <NUM>. The image intensifier <NUM> includes functionality for amplifying the received image so that the amplified image is sufficiently bright to be readily seen by the user. In the illustrated embodiment, this amplification is accomplished using a photocathode <NUM>, a microchannel plate <NUM>, and a phosphor screen <NUM>. The photocathode <NUM> absorbs incident photons and outputs electrons in response. The electrons may pass through an optional ion barrier film <NUM>. Electrons from the photocathode <NUM> are accelerated across an electric potential and transmitted through the microchannel plate <NUM> to multiply the number of electrons. The multiplied electrons then strike a phosphor screen <NUM>, converting the energy from electrons to photons via the phenomenon of luminescence. The phosphor screen <NUM> thus converts the radiant energy of the multiplied electrons to luminescent light. Accordingly, the phosphor screen <NUM> glows due to electrons from the microchannel plate <NUM> striking the phosphor screen <NUM>, creating an intensified image that represents the image of the input light <NUM>. A fiber-optic element <NUM> carries the intensified light <NUM> to the overlay display <NUM>.

The overlay display <NUM> generates display light <NUM>, which is superimposed with the intensified light <NUM>. For example, the overlay display <NUM> includes functionality for displaying information to a user. Such information may include graphical content, including text, images, superimposed thermal image data and the like. Additional details regarding certain embodiments of the NV system <NUM> and the overlay display <NUM> are provided in <CIT>, titled "Backside Etch Process for Transparent Silicon Oxide Technology.

<FIG> illustrates an example of an image in which an overlay display <NUM> superimposes text and other graphical symbols over an amplified image of a nightscape that includes trees and clouds. As discussed above, the overlay display <NUM> includes functionality for displaying information to a user. Such information may include graphical content, including text, images, superimposed thermal image data and the like. The overlay display <NUM> outputs display light <NUM> which can be sent to the eyepiece. Thus, an image such as that illustrated in <FIG> is presented to the user in the NV system <NUM>.

<FIG> and <FIG> illustrate example embodiments of the overlay display <NUM>. For example, the overlay display <NUM> may include several active silicon areas <NUM> function as circuit components, such a field effect transistors (FETs). In certain embodiments, one plane of circuits elements may be provided as display-control circuitry <NUM>, and another plane of circuits elements may be provided as data-handling circuitry <NUM>. The display-control circuitry <NUM> may control respective pixels of the overlay display <NUM>. In certain embodiments, the overlay display <NUM> may be a digital display having a certain pixel density. Each pixel has one or more transistors controlling one or more emitters <NUM> (e.g., the emitters <NUM> may be organic light emitting diodes (OLEDs)). Additional details regarding the overlay display <NUM> are provided in <CIT>.

<FIG> illustrates a cross-section of a part of the overlay display <NUM>. In certain non-limiting embodiments, the overlay display <NUM> may include active silicon areas, which are illustrated as active silicon islands <NUM> (e.g., native silicon islands). The active silicon islands <NUM> can be used to fabricate transistors, such as MOSFETs by doping the silicon (Si) with spatially varying concentrations of donor and acceptor atoms. Further, the MOSFETs may be fabricated using intermetal and dielectric layers <NUM> that include insulators (e.g., oxides and dielectrics) and metal traces <NUM>. In certain embodiments, the MOSFETs may provide (but are not limited to providing) logic functions and/or control functions.

<FIG> illustrates the overlay display <NUM> superimposing a display image over the intensified image. For example, a display image is generated by outputting the display light <NUM>. In <FIG>, the intensified light <NUM> enters the overlay display <NUM> from the bottom, passes through the oxide <NUM> and then through the other layers before exiting the overlay display <NUM> through the cover glass <NUM>. The display light <NUM> is generated in the emitter <NUM>, and, like the intensified light <NUM>, the display light <NUM> exits through the cover glass <NUM>. After exiting through the cover glass <NUM>, both the display light <NUM> and the intensified light <NUM> are transmitted to the eyepiece <NUM> of the NV system <NUM>, and then to the user.

Whereas the pixels (i.e., Si island <NUM>, metal traces <NUM>, and emitters <NUM> in the emitter stack <NUM>) substantially attenuate the intensified light <NUM>, transparent regions between the pixels are at least partially transparent to the intensified light <NUM>. Accordingly, the intensified light <NUM> is transmitted through the transparent regions between the pixels of the overlay display <NUM>. In contrast, the active Si islands <NUM> and the metal traces <NUM> substantial block the intensified light <NUM>.

In both <FIG> and <FIG>, the circuit components are fabricated in more than one plane, whereas in other embodiments (not shown) the circuit components may be fabricated in a single plane. Different functions may be performed by circuitry in different planes. For example, <FIG> illustrates a certain embodiment of the overlay display <NUM> in which a first set of fabrication layers implement the display-control circuitry <NUM> (e.g., circuitry to drive the emitters <NUM> and generate display light <NUM>). A second set of fabrication layers implement the data-handling circuitry <NUM>. Thus, the display-control circuitry <NUM> and the data-handling circuitry <NUM> are respectively fabricated in separate circuitry planes.

<FIG> illustrates a top-down view of a portion of an overlay display <NUM> in which the opaque regions (e.g., regions including the active Si islands <NUM> and metal traces <NUM>) are configured with a transparent region <NUM> between the opaque regions. The active Si islands <NUM> and metal traces <NUM> substantially block the intensified light <NUM>, but the intensified light <NUM> is transmitted through the transparent region <NUM> between the Si islands <NUM> and metal traces <NUM>. Metal traces called column lines <NUM> and row lines <NUM> run between the pixels, conveying signals addressed to the respective pixels. These lines are also opaque regions.

<FIG> illustrates an embodiment of an overlay display system <NUM>. The display system <NUM> includes an overlay display <NUM> having an emitter array <NUM> that generates a display image by emitting display light <NUM> in accordance with pixel values stored in the frame buffer <NUM>. For example, the emitter array <NUM> may be a two-dimensional (2D) array of pixels (e.g., the emitters <NUM> illustrated in <FIG>). The emitter array <NUM> may also include the driving electronics (e.g., FETs) in the display-control circuitry <NUM> that drive current through the emitters <NUM> to generate the display light <NUM>. The display image is updated at a first frame rate. The display image emitted from the overlay display <NUM> is updated when pixel values are transferred from the frame buffer <NUM> to the emitter array <NUM>.

The frame buffer <NUM> may be a DMA (direct memory access) hardware readout device that contains a memory array representing the pixel values of each pixel, and the memory array may represent the pixel values in a fixed order. The pixel values may be stored at dedicated memory addresses in the memory array. The display image may be updated periodically at the predetermined first frame rate by transferring the pixel values of each pixel from the frame buffer <NUM> to the emitter array <NUM>.

In a separate process, the pixel values stored in the frame buffer <NUM> are updated at a predetermined second frame rate. Thus, the frame rate and process of updating the frame buffer <NUM> may be independent of the process of updating the emitter array <NUM>. The pixel values stored in the frame buffer <NUM> are updated by the processor <NUM>. The updating of the pixel values stored in the frame buffer <NUM> may proceed more slowly than the first frame rate. Further, many of these pixel values stored in the frame buffer <NUM> may remain the same from frame to frame while only a few pixel values change between frames. Accordingly, the rate at which new values are written to the frame buffer <NUM> may be independent of and decoupled from the first frame rate. Additionally, the amount of data and data rate required to update pixel values stored in the frame buffer <NUM> may be much less than the amount and rate of data transferred between the frame buffer <NUM> and the emitter array <NUM>.

The overlay display <NUM> may optionally include a detector array <NUM> (e.g., a 2D array of the detectors <NUM> illustrated in <FIG>). Like the frame buffer <NUM>, a detector buffer <NUM> may function as a buffer by storing pixel values. The detector buffer <NUM> stores detector signals read from respective detectors in the detector array <NUM>. The detected values are stored at dedicated memory addresses in the detector buffer <NUM>. The reading of the detected signal values on to the detector buffer may proceed at a third frame rate, and pixel values stored on the detector buffer <NUM> may be transferred to the processor <NUM> at a fourth frame rate, which may or may not be the same as the third frame rate. That is, the processor <NUM> may access/read the detected signals stored in the detector buffer <NUM> independently of the third frame rate.

Additionally, processor <NUM> may directly access the detected signals from one or more of the detectors in the array <NUM>. For example, the processor <NUM> may synchronize one or more of the above-noted frame rates with a periodicity of an external light source having a time varying intensity. By synchronizing with the external light source, the detector may avoid beat patterns or other artifacts in the detected signal that may result in a glitchy appearance.

A synchronization signal, which in one embodiment may be at <NUM>, may be used to determine the second frame rate based on the detected signals from the detector array <NUM>. The detector array <NUM> may include signal conditioning circuitry (e.g., transimpedance amplifiers/buffers charge integrators, etc.) or analog to digital conversion circuitry provided in the data-handling circuitry, for example.

The detected signals may include light signals that have been encoded with data, and the processor <NUM> may decode the data from the detected signals received via the detector array <NUM>. For example, infrared (IR) beacons or LiDAR signals may be transmitted within the field of view of the NV system. The light from an IR beacon may be modulated to enable blue force tracking. Blue force tracking may be provided by the light from the IR beacon indicating that the IR beacon represents a friend, as opposed to a foe.

Additionally or alternatively, the modulated light may be encoded with data representing text or other information. The processor <NUM> may decode the message encoded in the received signal from the detector array <NUM> to generate the represented text, and this text may then be displayed within a designated field in the display image, as illustrated in <FIG>, which is discussed below.

The display system <NUM> may include a memory <NUM>, an inertial navigation system (INS) <NUM>, an input/output (I/O) interface <NUM>, a transceiver <NUM>, and an antenna <NUM>.

In certain embodiments, the display system <NUM> can include a processor <NUM> coupled to a memory <NUM>. The processor <NUM> may be a simple controller. Alternatively, the processor may be CPU (e.g., an ARM architecture CPU such as the Cortex A53 by ARM Inc. or a Snapdragon <NUM> by Qualcomm, Inc). In certain embodiments, the processor <NUM> may be an on-board, intelligent controller that is on a same PC board as the overlay display.

The display system <NUM> may have an antenna <NUM> that is connected to a transceiver <NUM> coupled to the processor <NUM>. The transceiver <NUM>, the processor <NUM>, and the memory <NUM> may be used for network communications.

The INS <NUM> of the display system <NUM> may include an GPS device, gyroscopes, accelerometers, magnetometers, or other sensors for position sensing and/or inertial navigation. The GPS device may be coupled to the processor <NUM> and used for determining time, location coordinates, and orientation coordinates, which may be displayed in the display image using the emitter array <NUM>, as illustrated in <FIG>.

The processor <NUM> can be any programmable microcontroller, microprocessor, microcomputer or chips that can be configured by software instructions (applications) to perform a variety of functions, including functions of various embodiments described herein. The processor <NUM> may be field programable gate array (FPGA), an application specific integrated circuit (ASIC), or other circuitry configured to perform instructions on sets of data and store results.

Software applications can be stored in the memory <NUM> before they are accessed and loaded into the processor <NUM>. The processor <NUM> can include or have access to the memory <NUM> sufficient to store the software instructions. The memory <NUM> can also include an operating system (OS). The memory <NUM> may include FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, magnetic storage drive, or any type of non-transitory computer readable medium.

Additionally, the memory <NUM> can be a volatile or nonvolatile memory, such as flash memory, or a mixture of both. For the purposes of this description, a general reference to memory refers to all memory accessible by the processor <NUM>, including memory <NUM>, removable memory plugged into the display system <NUM>, and memory within the processor <NUM> itself, including a secure memory.

The display system <NUM> can also include an input/output (I/O) interface <NUM> to receive and transmit signal to peripheral devices and sensors, or to communicate with an external controller. For example, the I/O interface <NUM> may be a high-speed parallel interface that receives an updated frame each period of the frame rate. Preferably, the I/O interface <NUM> is a serial interface, and only a small amount of data is needed to update the frame of the display image per period of the frame rate. The I/O interface <NUM> may include an I/O bus and a physical port, such as a universal serial bus (USB) port, or small computer system interface (SCSI) port, or other physical digital communicans port.

For example, a significant amount (or all) of the information needed to update the frame may be obtained from other sources than the I/O interface <NUM>. In certain embodiments, the overlay display includes symbols and text representing the position and orientation of the NV system, blue force tracking information, outline or other designators indicating objects of interest. The symbols and text representing the position and orientation of the NV system may be generated using signals from the INS <NUM>, without requiring any data received from the I/O interface <NUM>. The blue force tracking information may be obtained using signal obtained from the detector array <NUM>, without requiring any data received from the I/O interface <NUM>. The outline or other designators indicating objects of interest may be obtained using signal obtained from the detector array <NUM>, without requiring any data received from the I/O interface <NUM>. For example, edge detection and/or change detection signal/image processing methods may be used to determine an object moving relative to a stationary background. Accordingly, the information content and pixel values to be displayed via the emitter array <NUM> may be determined without requiring any data received via the I/O interface <NUM>.

In certain embodiments, the I/O interface <NUM> may not be needed to receive data that is used to determine pixel values for the emitter array <NUM>, and the I/O interface <NUM> may be omitted from the display system <NUM>.

The transceiver <NUM> and antenna <NUM> may be configured to receive and transmit wireless signals, including, e.g., Bluetooth (BT) signals, Bluetooth Low Energy (BLE) signals, cellular signals, WiFi signals (e.g., IEEE <NUM> standard), Zigbee signals, or other wireless communication protocol. An external controller may communicate information to/from the processor via the I/O interface <NUM> or via the transceiver <NUM> and antenna <NUM>.

As described in greater detail below, various embodiments of the display system <NUM> use one or more of the following techniques to reduce (or eliminate) the communication bandwidth required between an external controller and the processer <NUM> to update pixel values to be displayed by the emitter array <NUM>. In certain embodiments, the display system <NUM> reduces the communication bandwidth by using a high-level language to communicate the display information at the symbol/character level, rather than the pixel level. For example, the emitter array <NUM> may include predefined symbol/character fields within the display area, in which fields the emitter array <NUM> displays text, ASCII characters, or other predefined symbols/characters in the predefined symbol/ character fields. The external controller communicates digital values representing the characters or symbols rather than communicating the pixel values. Then the processor <NUM> determines the pixel values based on the received digital values representing the characters or symbols.

In certain embodiments, the display system <NUM> reduces the communication bandwidth by communicating only the pixel values that have change from their values in the previous frame. For example, in a typical application, most of the pixel values of the display will be zero (e.g. turned off) to better allow the underlying, intensified image to be unobscured by the superimposed display image. Accordingly, the communication bandwidth may be reduced by communicating only pixel values for those pixels that change between frames.

Further, in some application, the display image changes by panning (e.g., spatial translating) one or more parts of the display image from one location to another within the display image. For example, the display image may outline a moving vehicle or target, and this outline may be spatially translated to another location to within the display image as the vehicle moves relative to the background. Additionally, the outline of a moving vehicle or target may be translated within the display image as the orientation and field of view of the NV system <NUM> changes. Thus, even if the pixel values of the outline will remain unchanged from frame to frame, their location within the frame may change from frame to frame. That is, in certain embodiments, the pixel values in the display image will be the same, but will be spatially translated between frames. Thus, the display system <NUM> may reduce the communication bandwidth by communicating one or more motion vectors describing how the pixel values are to be translated, rather than communicating new pixel values. For example, the communicated information from the external controller may include motion vectors and/or pixel values for a difference image, similar to image compression protocols like JPEG, H. <NUM>, MPEG1, MPEG2, and MPEG4.

The communication bandwidth required for sending detected images from the processor <NUM> to an external controller can similarly be reduced by sending pixel values (and/or motion vectors) for pixels of the detected image that have changed/moved from frame to frame. For example, the detected image acquired by the detector array <NUM> may have substantial redundant information from frame to frame. Accordingly, the information of the detected image may be reduced by sending only pixel values for those pixel values that change between frames. Additionally, the communications of detected images to the external controller may include motion vectors and/or pixel values for a difference image, similar to image compression protocols like JPEG, H. <NUM>, MPEG1, MPEG2, and MPEG4.

In certain embodiments, the display system <NUM> reduces the communication bandwidth to the external controller by compressing the data. For example, entropy coding, variable length coding, and other data compression techniques may be used to reduce the required bandwidth.

In certain embodiments, the display system <NUM> reduces the communication bandwidth to the external controller by locally determining the information to be represented in the display image, rather than relying on an external controller to provide the information to be represented.

In certain embodiments, the display system <NUM> reduces the communication bandwidth to the external controller by receiving the information to be represented in the display image through other channels such as through the detector array <NUM>, the INS <NUM>, or the transceiver <NUM> and antenna <NUM>, rather than receiving the information to be represented from an external controller. Additionally or alternatively, the information to be represented in the display image may be derived from the information received through other channels. For example, the display image may include a compass representing the orientation and/or geographical coordinates derived from data from the INS <NUM>, as illustrated in <FIG>.

The display system <NUM> may include light source such as an eye safe laser (e.g., a laser source transmitting light having a wavelength of <NUM>), and the light source may be used for ranging and detection of a surrounding environment. The light source may be used for communication by encoding a message or data on the emitted light.

In certain embodiments, the frame buffer <NUM> may be part of a specialized 'frame driver' chip that is mounted on the same PC board (PCB) as the overlay display <NUM>. The frame buffer <NUM> may provide a local, high-speed, parallel interface between the processor <NUM> and the frame buffer <NUM>. For example, the frame buffer <NUM> may function as a simple DMA (direct memory access) hardware readout device. And the overlay display <NUM> may accept the pixel array data as a bulk write without individual headers. Further, in certain embodiments, the local frame buffer <NUM> may contain a memory array that represents the brightness/intensity values of each pixel in a fixed order. The display image emitted by the emitter array <NUM> is then simply read out of the local frame buffer <NUM> at a desired frame rate. The frame rate for the display image may be separately adjustable, relative to the frame rate at which the pixel values are updated in the frame buffer <NUM>.

In certain embodiments, the frame buffer <NUM> is periodically (or nonperiodically/asynchronously) updated with new pixel values to be displayed, which can be received from the processor <NUM>, and the new pixel values to be displayed may be determined by the processor <NUM> based on display information received via communications from an external controller. For example, the I/O interface <NUM> may have one or more low to medium speed serial communications interfaces that enables the external controller to communicate display information via a high-level language. This communication of display information from the external controller may be performed by sending a text string (e.g., a data structure including an array of alphanumeric characters or ASCII characters). The process may then determine which pixel values should be set in the frame buffer <NUM> to display the text string at a pre-determined location within the display image. For example, for the character "H," the <NUM>-bit ASCII binary value "<NUM>" (i.e., decimal value "<NUM>") may be communicated via the I/O interface <NUM>. Then the processor <NUM> may determine a series of pixel values (e.g., ranging from a minimum of <NUM> to a maximum of <NUM>), as illustrated in <FIG>, to represent the character "H," and these pixel values may be written to those memory cells in the frame buffer <NUM> that correspond to the designated position within the display image at which the character "H" is to be displayed. In this non-limiting example, sending a single <NUM>-bit ASCII code rather than sending <NUM>-bit codes for each of the of the <NUM> pixel values decreases the communication data size by a factor of more than <NUM>. In other example, the decrease in communication data size (i.e., communication bandwidth) may be even larger. Thus, transmitting display information via a high-level language may greatly ease the interface requirements.

Various predefined parameters such as a font type, a font size, an anchor coordinate (e.g., the x,y values of the top left pixel for a character string), a font color, a brightness, and a contrast, may be used by the processor when determining the pixels values. These parameters may be sent together with the text message, or the parameters may have been sent previously. For example, the parameters may be set as default parameters, or they may have been set in accordance with the current or previous communications from the external controller.

As discussed above, when updating the frame of the display image, not all pixel values will change relative to the previous frame. Accordingly, the processor <NUM> may determine which of the pixel values are changed when updating the frame, and the processor <NUM> updates the pixel values in the frame buffer <NUM>, in an intelligent way, to limit the power draw and make it scalable to the amount of change being requested by the external controller. The processor <NUM> may update the frame buffer <NUM> with only those pixel values that have changed from frame to frame and lock in the new frame content. The processor <NUM> may also function as a shadow register to avoid the display image having a glitchy appearance.

In certain embodiments, the detector array <NUM> acquires frames of detected intensities of the intensified light <NUM>. Like the frames of the display image, many pixel values in the frames of the detected images of the intensified light <NUM> do not change from one frame to the next. Thus, it may be inefficient to communicate all the pixel values from the detected frames to the external controller, rather than sending only the pixel values of those pixel values that have changed. For example, this more efficient communication may be performed by calculating a difference image relative to a previous frame, and using variable length coding, run length coding, and/or entropy encoding to encode the difference image before communicating the encoded and compressed difference image to the external controller. Similar processes can be used to more efficiently encode and transmit display frame data and detected frame data to enable more efficient communications to and from the external controller. For example, the frames acquired by the detector array <NUM> can be processed similar to the frames of the display image by the processor <NUM> communicating only what changed in the frame acquired by the detector array <NUM> (e.g., a detected underlying tube display).

In addition to decreasing the communication bandwidth from the processor <NUM> to the external controller, determining and isolating the pixel values that change between detected frames (i.e., frames acquired from the detector array <NUM>) can be used advantageously to provide auto display-over-background contrast adjustments while maintaining a low power draw, as discussed below.

In certain embodiments, the local processor <NUM> has an increased degree of control and autonomy and functionality that otherwise have been provided by the external controller. For example, the local processor <NUM>, rather than the external controller, provides a frame-sync signal to control a timing of when frames are displayed on the emitter array <NUM> and/or another frame-sync signal to control when frames are acquired from the detector array <NUM>. The increased autonomy of the processor <NUM> provides decoupling of the processor <NUM> from the.

In certain embodiments, the local processor <NUM> controls timing of detector pixel array <NUM>. This control may be used to enable all-optical communications for the NV system <NUM> and for programming and/or information provision to the NV system <NUM>. The all-optical communications may provide for communications that are not jammable or visually detectable by others (e.g., covert communications). For example, the all-optical communications may be realized using a covert eye-safe wavelength (e.g., lasers transmitting light at <NUM> that is modulated to send information). One or more detectors in the detector array <NUM> may detect the intensified light <NUM> that has been modulated according to a message or data, for example. For the application of blue-force tracking, the received light used to generate the intensified light <NUM> may be infrared (IR) light that has been modulated with a pulse frequency code indicating that the source of the light is a friend, rather than a foe. Alternatively or additionally, the received light may be modulated with a text message or other data. The detector array <NUM> may then generate a time-series of detected values that is demodulated to recover the message. In certain embodiments, the recovered the message may be displayed as text in the display image. In certain embodiments, the recovered the message may include coordinates of a friend's location. The recovered message may include instructions to modify how the NV system operates. For example, all optical communications may be used to update or upgrade the firmware of the NV system.

In certain embodiments, the external controller may communicate with the NV system via wireless communications. As discussed above, the local processor <NUM> may provide functionality that would otherwise be performed by the external controller, thereby reducing the amount of information to be communicated from the external controller. Reducing the amount of information and communication bandwidth to and from the external controller reduces the power draw and reduces the interface pin count, decreasing the system size and complexity of the communication interface. Accordingly, the communication interface may be realized with as few as <NUM> traces/pins. Additionally or alternatively, the communication interface may be realized using Bluetooth (BT), Bluetooth Low Energy (BLE), ANT Wireless (ANT), cellular wireless communications, WiFi (e.g., IEEE <NUM> standard), Zigbee wireless communications.

In certain embodiments, the processor <NUM> may receive an instruction in a high-level language regarding what symbols to display in the display image. The processor <NUM> may receive, via the communication interface (e.g., the I/O interface <NUM>, the detector array <NUM>, or the transceiver <NUM>) data indicating symbols to be represented within the display image. The processor <NUM> may determine the pixel values for the display image based on the symbols indicated in the data. Then the processor <NUM> may update the pixel values stored in the frame buffer <NUM> based on the determined pixel values to be represented within the display image.

The memory <NUM> may store information relating pixel values to respective symbols (e.g., a look-up-table indexed by the respective symbols that outputs an array of pixel values used to represent the given symbol). The symbols to be represented within the display image may include one or more types of symbols including, e.g., alphanumeric characters, ASCII characters, pictograms, icons, a reticle, a compass, and an outline.

<FIG> illustrates an underlying image (e.g., an intensified image from an analog NV system), and <FIG> illustrates a display image superimposed over the underlying image. In <FIG>, a text box <NUM> is illustrated with the text "Text Here" provided within the text box. In the display image, the text "Text Here" may be displayed without displaying an outline of the text box. The text box may be defined by specifying the coordinates of the box <NUM> (e.g., the upper left corner of the text box), the font, the font size, the font color, and the brightness and/or contrast. The high-level instructions may include a font, a font size, an anchor coordinate, a font color, an array of alphabetic characters to be displayed, a brightness, and a contrast of the symbols to be displayed.

In <FIG>, inertial navigation information <NUM> is illustrated, and a reticle <NUM> and tracking information <NUM> are also illustrated. Some of the illustrated symbols may be determined locally by the processor <NUM> without requiring external information being provided by an external controller.

In certain embodiments, the protocol for determining pixel values within the text box <NUM> may use a simple high-level instruction providing the text or symbol to be inserted via an insertion command. The insertion command may be listed as a 'block of pixels' (e.g., a font character) with a specified origin for the symbol's origin position (e.g. at pixel <NUM>,<NUM>). A symbol library may be pre-installed and stored in the memory <NUM>. Then the pixel locations may be recalled from memory by calling the stored data as command line packet. Additionally, the insertion command may specify an overall size and pixel value (e.g., scaled as a block from the default max size & brightness). Accordingly, the block can have a relative shape and brightness variation in accordance with instructions (e.g., user preferences). To provide error checking and/or correction, the insertion command may include a checksum or error correction option.

In certain embodiments, the insertion command may be provided for a block of pixels as small as a single pixel. More generally, the insertion command may be provided for a block of any shape or configuration. Further, the insertion command may be built up with added commands or by creating a new block. This could be done in real time. Generally, the use of this type of the insertion command provides data compression that keeps the overall communication bandwidth low. By keeping the overall communication bandwidth low, the pixel value processing may be performed, e.g., by simple microcontrollers rather than by an FPGA, which consumes more power.

In certain embodiments, feedback from the detector array <NUM> is used to adjust an intensity of the display image to be balanced relative to the intensified image. For example, if the intensified image is much dimmer than the display image, then the intensified image may be difficult to see because it is overwhelmed by the brighter display image. Thus, the intensity of the display image may be adjusted to be commensurate with the average (local) intensity of the intensified image. For example, when the intensified image is dim, the intensity of the display image is reduced accordingly to also be dim. Similarly, when the intensified image is bright, the intensity of the display image is increased accordingly to also be bright. This intensity adjustment of the display image may be global (e.g., the entire display image) or local (e.g., local regions of the display image may be adjusted brighter or dimmer depending on the local intensity of the intensified image). The local memory <NUM> may store an auto-contrast, brightness-adjustment control table that balances the intensity of the display image relative to the detected intensity of the intensified image, which is detected via the detector array <NUM>.

Additionally, even when equal currents are used to drive respective emitters, different emitters in the emitter array <NUM> may emit with different intensities due to non-uniform manufacturing defects resulting in non-uniform emission efficiencies. Accordingly, the emission efficiencies of the different emitters of the emitter array <NUM> may be calibrated, and a correction factor stored in the local memory <NUM>. Additionally or alternatively, the detection efficiencies of the different detectors of the detector array <NUM> may be calibrated, and a correction factor stored in the local memory <NUM>. That is, the memory <NUM> may store a non-uniformity correction(s), which is a fixed array to correct for display brightness and/or detector efficiency variations across the image plane.

Additionally, as discussed above, the memory <NUM> may store the auto-contrast, brightness-adjustment control table, which may be dynamically updated locally. Given the brightness/intensity adjustments together with the detector/emitter efficiency correction factor that are performed locally via the memory <NUM> and the processor <NUM> rather than via an external controller, the communication bandwidth with the external controller may be reduced and the communication interface may be simplified. For example, the external controller may only be needed to supply the information that is conveyed in the display image-not the hardware level controls.

In certain embodiments, the processor <NUM> may provide a local sync functionality. For example, the signals from the detector array <NUM> may be processed to determine periodic brightness fluctuations in the intensified light <NUM> resulting from the background. Based on the signals from the detector array <NUM>, a sync may be determined to synchronize the frame rate to the periodic background fluctuations (e.g., the display image frame rate and/or detector array frame rate may be synchronized to the periodic background fluctuations). The local sync functionality may provide automatic gating from a supply of an electron tube used to intensify the light. The electron tube intensifies the light received from the environment by accelerating electrons, which are generated when the received light strikes a photocathode, across an electric potential provided by the electron tube before the accelerated electrons pass through a microchannel plate, multiplying the number of electrons through interactions with the side walls of the microchannel plate. The local sync functionality helps to avoids beat patterns, which appear as pulsing of the intensified image.

In certain embodiments, the sync functionality may be advantageously used to realize additional functionality. For example, the sync functionality may be used with pulse frequency code IR beacons located with friendly users (e.g., aviation or ground). This additional functionality would help to identify unknown people/entities and/or identify their position relative to the NV system <NUM>. Thus, the sync functionality may offer enhanced blue-force spatial awareness (SA) in GPS denied areas.

In certain embodiments, the signals from the detector array <NUM> (e.g., together with the INS <NUM>) may be used to provide feedback for quasi-inertial and/or passive visual image processing of point positions to generate a map of known points within line-of-sight of the NV system <NUM> (e.g., known points within line-of-sight may be provided by IR beacons transmitting known pulse frequency codes). The provided feedback may be used together the GPS data (while in an area with access to GPS signals) to train and improve blue-force tracking accuracy to better account for beacon movement characteristics in real time. Some embodiments may rely on only non-GPS data but use GPS data as feedback to be trained while in areas with access to GPS signals, such that later when in a GPS-denied area the performance will be improved when using only non-GPS data. Thus, an improved algorithm can be learned with improved GPS-denied blue-force tracking accuracy. Similarly, algorithms may also be improved for gyro drift compensation via image processing based on the signals from the detector array <NUM>.

In certain embodiments, the display system <NUM> may enable passive ranging. Additionally, augmented reality icons, such as the tracking information <NUM>, may be used. Based on passive ranging, the display system <NUM> may determine whether the object indicated by a given augmented reality icon is located inside or outside a same building as the NV system <NUM>. Based on this determination, the symbol used as the augmented reality icon may change to indicate the object's relative location (e.g., in or out the same building as the NV system <NUM>). Thus, symbology indicators may change based on information of a user's field of view (FOV) in real time.

In certain embodiments, the NV system <NUM> may include a covert, eye-safe range finder (e.g., a frequency modulated or pulsed laser emitting light having a wavelength of <NUM>). The signals from the range finder may be used to improve the accuracy of blue-force tracking, especially in GPS denied areas. For example, the signals from the range finder may be used to provide range information that supplements the angular information determined from the signals from the detector array <NUM>.

In the examples above it should be noted that although not shown various alternatives can be implemented. For example, in any of the embodiments illustrated, a backside fill may be used or may be omitted. Alternatively, or additionally, while the active areas have been shown as being substantially square in nature, it should be appreciated that the active areas may be rectangular or other appropriate shapes.

The discussion above refers to a number of methods and method acts that may be performed.

Claim 1:
An optical device comprising:
an eyepiece (<NUM>); an objective (<NUM>); and an image intensifier module (<NUM>);
wherein the objective (<NUM>) is configured to receive first light from an environment and to transmit the first light to the image intensifier module (<NUM>); the image intensifier module (<NUM>) comprises an image intensifier (<NUM>) configured to amplify and output the received first light, and an overlay display system (<NUM>); the overlay display system comprising:
- a semiconductor chip (<NUM>) configured to generate second light based on pixel values of a display image; wherein the semiconductor chip (<NUM>) comprises a bottom surface configured to receive the output amplified first light and a top surface configured to output the second light and the amplified first light, and an array of pixels comprising light emitters (<NUM>) arranged with transparent regions between them so such that light passing through the bottom surface can pass to the top surface, and wherein the light emitters (<NUM>) are configured to generate the second light with an intensity emitted based on respective pixel values of a display image;
a frame buffer (<NUM>) configured to the store pixel values in a computer readable memory, and configured to communicate, at a first frame rate, the pixel values to the light emitters (<NUM>); and
a processor (<NUM>) configured to update, at a second frame rate, pixel values stored in the frame buffer (<NUM>);
wherein the eyepiece (<NUM>) comprises optics adapted to focus the light from the semiconductor chip into the eye of a user.