Synchronized camera and lighting system

A system, a camera, and a method. The system may include a light device and a camera. The light device may be configured to emit light having a lighting frequency defined by on states and off states of the light device. The camera may be in view of the emitted light. The camera may have an image acquisition frequency configured to capture all images during the on states of the light device.

BACKGROUND

Cameras are becoming more prevalent in aircraft to allow better visibility to cabin crew, enhance security, and protect airlines from out-of-context social media mobile phone video clips. Light emitting diodes (LEDs) are currently the preferred lighting choice for use in aircraft. LEDs are typically used in aircraft with light intensities less than 100% of the LEDs full intensity capability; such diminished light intensities are typically achieved by quickly pulsing the LEDs between on states (where the LEDs are powered) and off states (where the LEDs receive no power or less power than the on states) by utilizing pulse-width modulation (PWM) frequencies for the LEDs. Flashing of LEDs at the PWM frequencies is imperceptible to humans as the flashing of the LEDs occurs quicker than the flicker fusion frequency of human vision so that a consistently lit environment is perceived. Such flashing of the LEDs, however, can cause problems with cameras taking photos or video inside the aircraft cabin as the camera refresh rate can be out-of-synch with the LED flashes. Currently, camera systems onboard aircraft are susceptible to flicker artifacts, rolling shutter (e.g., aliasing), and black frames (e.g., where LEDs are off during image acquisition) caused by camera image acquisition capturing some images when the LEDs are in off states. Even worse, as the LEDs dim to a low percentage (e.g., in an on state for 10% a PWM cycle and then in an off state for 90% of the PWM cycle), the likelihood of acquiring an image while the LEDs are in an on state is low, despite the fact that human eyes may perceive constant light.

SUMMARY

In one aspect, embodiments of the inventive concepts disclosed herein are directed to a system. The system may include a light device and a camera. The light device may be configured to emit light having a lighting frequency defined by on states and off states of the light device. The camera may be in view of the emitted light. The camera may have an image acquisition frequency configured to capture all images during the on states of the light device.

In a further aspect, embodiments of the inventive concepts disclosed herein are directed to a camera. The camera may include an image sensor and a processor. The image sensor may be in view of emitted light from a light device, the emitted light having a lighting frequency defined by on states and off states of the light device. The image sensor may have an image acquisition frequency and may be configured to capture images. The processor may be communicatively coupled to the image sensor. The processor may be configured to control the image sensor such that the image sensor captures all images during the on states of the light device.

In a further aspect, embodiments of the inventive concepts disclosed herein are directed to a method. The method may include controlling, by a processor, an image sensor of a camera, the image sensor in view of emitted light from a light device, the emitted light having a lighting frequency defined by on states and off states of the light device, the image sensor having an image acquisition frequency. The method may also include capturing, by the image sensor of the camera, all images during the on states of the light device

DETAILED DESCRIPTION

Broadly, embodiments of the inventive concepts disclosed herein are directed to a system, apparatus, and method for synchronizing an image acquisition frequency and timing thereof of a camera with a lighting frequency and timing thereof of at least one light device so that the camera captures all images during on states of the at least one light device. For example, a lighting system (e.g., an aircraft cabin lighting system) may have the lighting system's pulse width modulation (PWM) frequency and timing synchronized (e.g., aligned) with a camera's image acquisition frequency and timing so that there are no flicker artifacts, rolling shutter (e.g., aliasing), or black frames (e.g., where light emitting diodes (LEDs) are off during image acquisition). Some embodiments may include a lighting controller sending a synch signal (e.g., a synch line) to a camera so as to cause the camera to acquire images during a leading edge of a PWM cycle when the lights are in an on state. Similarly, some embodiments may include a camera sending a synch signal (e.g., a synch line) to a lighting controller so as to cause lights to be in a leading edge of a PWM cycle (when the lights are in an on state) when the camera is scheduled to acquires images.

Embodiments may include connecting a camera with a lighting system to cause alignment with the camera's image acquisition and on states of light devices. In some embodiments, a lighting controller may output a synch signal to a camera to cause synchronization of light devices with the camera, while in other embodiments, the camera may output a synch signal to a lighting controller to cause synchronization of light devices with the camera. Embodiments may allow the camera to align the camera's frames with on states of light devices so as to avoid image flicker and other artifacts, and embodiments may maintain bright and clear images even when the environment appears dimly lit to humans because images are acquired during an on state portion (e.g., a leading on state edge) of a PWM flash. Due to such synchronization, an aircraft cabin may always be consistently and brightly lit during a camera's image acquisitions even when the cabin appears dimly lit to humans. Such synchronization can be performed with reading lights, ceiling lights, sidewall lights, and/or other feature lights to maintain a coherent, camera friendly environment. Additionally, some embodiments may include infrared (IR) LEDs configured to emit IR light and an IR camera configured to capture IR images to allow night vision of the cabin, such as when the cabin lighting is in night mode.

Additionally, in some embodiments, if a camera's image acquisition frequency is too slow (e.g., less than 100 Hertz (Hz)), a frequency multiplying method can be applied. For example, the camera can be configured to perform a synchronized image acquisition for one out of every two PWM flashes, one out of every three PWM flashes, one out of every four PWM flashes, etc. This keeps the frequency of the PWM flashes above the flicker fusion frequency of human vision and allows for the synchronization of an image acquisition frequency and timing thereof of a camera with a lighting frequency and timing thereof of a light device so that the camera captures all images during on states of the at least one light device, even though the image acquisition frequency is less than (e.g., ½ of, ⅓ of, ¼ of, . . . or 1/n of (where n is an integer)) the lighting frequency.

Referring now toFIGS. 1-4, an exemplary embodiment of a system (e.g., a vehicular system (e.g., an aircraft system)) implemented in a vehicle (e.g., an aircraft102) according to the inventive concepts disclosed herein. WhileFIGS. 1-4exemplarily depict an aircraft system implemented on the aircraft102, some embodiments may be implemented as any suitable vehicular system (e.g., an automobile system, a train system, and/or a ship system) implemented in any suitable vehicle (e.g., an automobile, a train, and/or a ship).

The aircraft102may include at least one lighting controller104, at least one light device (e.g., at least one LED light device106), at least one camera108, at least one IR camera110, and at least one computing device, some or all of which may be communicatively coupled at any given time. While the aircraft102exemplarily includes elements as shown, in some embodiments, one or more of the elements of the aircraft102may be omitted, or the aircraft102may include other elements.

The light devices (e.g., the LED light devices106, which may include LEDs) may be configured to emit light having a lighting frequency defined by on states and off states of the light devices. That is, the light devices may repeat cycles of being in an on state followed by being in an off state. In some embodiments, the lighting frequency is a PWM frequency. The lighting frequency may be adjustable, such as by the lighting controller104, so as to cause lighting to appear dimmer or brighter to humans. In some embodiments, the light devices may have synchronized lighting frequencies such that the on states and off states are synchronized for all of the light devices. In some embodiments, the light devices may be controlled by the lighting controller104. For example, the PWM frequency (e.g., so as to dim or brighten the perceived light) and a timing thereof (e.g., by shifting a phase of lighting cycles) may be controlled by the lighting controller104. In some embodiments, some of the light devices may be implemented as IR light devices (e.g., IR LED light devices) and/or may include IR lights (e.g., IR LEDs). For example, the light devices may include LED light devices106and separate IR LED light devices. For example, light bars (or any other light assemblies, such as reading lights) may include the LED light devices106and the IR LED light devices, where each light bar is part of a red green blue white IR (RGBW(IR)) string such that the camera108and/or the camera110may be configured to capture images across the visible and IR spectrum.

The lighting controller104may include at least one processor202, memory204, and at least one storage device206, as well as other components, equipment, and/or devices commonly included in a computing device, some or all of which may be communicatively coupled at any given time, as shown inFIG. 2. The processor202may be implemented as any suitable processor, such as a general purpose processor, a field-programmable gate array (FPGA), and/or an image processor. For example, the lighting controller104may be configured to control the lighting frequency (e.g., so as to dim or brighten the perceived light) and a timing thereof (e.g., by shifting a phase of lighting cycles) of the light devices.

In some embodiments, the lighting controller104may be configured to obtain an image acquisition frequency and a timing (e.g., a phase at a point in time of the image acquisition cycle) thereof from the camera108and to synchronize the lighting frequency of the light devices with the image acquisition frequency of the camera108such that the camera108captures all images during the on states (e.g., during leading edges of the on states) of the light devices. The image acquisition frequency and the timing thereof may be obtained as a synch signal (e.g., a synch line).

In some embodiments, the lighting controller104may be configured to obtain and/or determine a lighting frequency and a timing (e.g., a phase at a point in time of the lighting cycle) thereof of the light devices and to output the lighting frequency and the timing thereof of the light devices to the camera108. The lighting frequency and the timing thereof may be output as a synch signal (e.g., a synch line). As such, the camera108may be configured to receive the lighting frequency and the timing thereof from the lighting controller104and to synchronize the image acquisition frequency of the camera108with the lighting frequency such that the camera108captures all images during the on states (e.g., during leading edges of the on states) of the light devices.

In some embodiments, the lighting controller104may be configured to synchronize the lighting frequency with the image acquisition frequency by adjusting (e.g., increasing or decreasing) the lighting frequency to be the same as or an n (where n is an integer greater than or equal to 2) multiple of the image acquisition frequency and/or by adjusting a timing (e.g., by shifting a phase of the lighting cycle) of each lighting cycle.

The processor202may be configured to run various software applications or computer code stored (e.g., maintained) in a non-transitory computer-readable medium (e.g., memory204and/or storage206) and configured to execute various instructions or operations. For example, the processor202may be configured to perform any or all of the operations of the lighting controller104. The lighting controller104may be implemented as any suitable controller or computing device (e.g.,112). In some embodiments, the lighting controller104may be implemented as a cabin lighting controller. Additionally, for example, the lighting controller104and/or the processor202may be implemented as special purpose computers or special purpose processors configured (e.g., programmed) to execute instructions for performing any or all of the operations disclosed throughout. In some embodiments, the system ofFIG. 1may include any suitable number of lighting controllers104. In some embodiments, the computing device112may be implemented as a lighting controller such that the lighting controller104may be omitted.

While the lighting controller104exemplarily includes elements as shown, in some embodiments, one or more of the elements of the lighting controller104may be omitted, or the lighting controller104may include other elements.

The camera108(e.g., a digital camera) may include at least one image sensor302, at least one processor304, memory306, at least one storage device308, and at least one light synch sensor310, as well as other components, equipment, and/or devices commonly included in a digital camera, some or all of which may be communicatively coupled at any given time, as shown inFIG. 3. The image sensor302may be implemented as any suitable image sensor. The processor304may be implemented as any suitable processor, such as a general purpose processor, a field-programmable gate array (FPGA), and/or an image processor. The camera108may be positioned in view of emitted light from the light devices. The camera108may have an image acquisition frequency, which may be configured to capture all images during the on states of the light devices. The image acquisition frequency may be adjustable, and the processor304may be configured to control the image acquisition frequency and a timing (e.g., e.g., a phase of the image acquisition cycle) thereof. For example, the processor304may be configured to shift a phase of the image acquisition cycle to change a timing. For example, the processor304may be configured to cause the image sensor to capture images. In some embodiments, the camera108may be configured to capture images in the visible spectrum and the IR spectrum; for example, the camera108may include the image sensor302and an IR image sensor (e.g.,402) or the image sensor302may be configured to capture images in the visible spectrum and the IR spectrum.

In some embodiments, the processor304may be configured to output an image acquisition frequency and a timing (e.g., a phase at a point in time of the image acquisition cycle) thereof to the lighting controller104such that the lighting controller104can synchronize the lighting frequency of the light devices with the image acquisition frequency of the camera108so that the camera108captures all images during the on states (e.g., during leading edges of the on states) of the light devices. The image acquisition frequency and the timing thereof may be output as a synch signal (e.g., a synch line).

In some embodiments, the processor304may be configured to obtain (e.g., receive) a lighting frequency and a timing (e.g., a phase at a point in time of the lighting cycle) thereof of the light devices from the lighting controller104and synchronize the image acquisition frequency with the lighting frequency such that the camera108captures all images during the on states of the light devices. The lighting frequency and the timing thereof may be obtained as a synch signal (e.g., a synch line).

In some embodiments, the processor304may be configured to obtain and/or determine a lighting frequency and a timing (e.g., a phase at a point in time of the lighting cycle) thereof of the light devices. For example, the processor304may be configured to obtain and/or determine a lighting frequency and a timing thereof based on data received from the light synch sensor310, which may be configured to detect the lighting frequency and the timing thereof. Based on the lighting frequency and the timing thereof, the processor may be configured to synchronize the image acquisition frequency with the lighting frequency such that the camera108captures all images during the on states (e.g., during leading edges of the on states) of the light devices.

In some embodiments, the image acquisition frequency may be an initial image acquisition frequency. The processor304may be configured to determine that the lighting frequency of the light devices is less than the initial image acquisition frequency, to reduce a frequency of the initial image acquisition frequency to a second image acquisition frequency, and to synchronize the second image acquisition frequency with the lighting frequency such that the camera108captures all images during the on states (e.g., during leading edges of the on states) of the light devices.

In some embodiments, the processor304may be configured to synchronize the image acquisition frequency with the lighting frequency by adjusting (e.g., increasing or decreasing) the image acquisition frequency to be the same as or a 1/n multiple of the lighting frequency and/or by adjusting a timing (e.g., by shifting a phase of the image acquisition cycle) of each image acquisition cycle.

The processor304may be configured to run various software applications or computer code stored (e.g., maintained) in a non-transitory computer-readable medium (e.g., memory306and/or storage308) and configured to execute various instructions or operations. For example, the processor304may be configured to perform any or all of the operations of the camera108. Additionally, for example, the processor304may be implemented as a special purpose processor configured (e.g., programmed) to execute instructions for performing any or all of the operations disclosed throughout. In some embodiments, the system ofFIG. 1may include any suitable number of cameras108. In some embodiments, the processor114of the computing device112may be configured to perform, at least in part, the functionality of the processor304.

The light synch sensor310may be configured to detect the lighting frequency and the timing thereof. In some embodiments, the light synch sensor310may include a processor (e.g., a controller (e.g., a microcontroller)) configured to measure the lighting frequency (e.g., a PWM frequency) and the timing thereof of sensed light such that the light synch sensor310can determine the lighting frequency and the timing thereof without being communicatively coupled to the lighting controller104(if the system includes a lighting controller). In some embodiments, the light synch sensor310may be configured to measure the lighting frequency and the timing thereof for visible light and/or IR light. The light synch sensor310may be communicatively coupled to the processor304of the camera108and configured to output detected (e.g., measured) lighting frequency and the timing thereof to the processor304of the camera108. In some embodiments, the light synch sensor310is implemented in or on the camera108; however, in some embodiments, the light synch sensor310may be implemented at any suitable position in a system (e.g., in a vehicle cabin) where the light synch sensor310is in view of light emitted from the light devices. For example, the light synch sensor310may be implemented separately from the camera108.

In some embodiments, a processor (e.g., the processor304and/or a controller of the light synch sensor310) may be configured to execute software such that the processor is configured to determine an appropriate operating mode of a plurality of operating modes (e.g., bright operating mode, medium operating mode, and dim operating mode) for the camera108, such as by locking on all light flashes (e.g., LED light flashes), locking on alternating light flashes, or recurrently locking on one of a series of light flashes (e.g., one of every n light flashes). As such, in some embodiments, the camera108may be retrofitted, without modifying or replacing legacy lighting systems, so as to achieve a synchronized camera and lighting system.

While the camera108exemplarily includes elements as shown, in some embodiments, one or more of the elements of the camera108may be omitted, or the camera108may include other elements.

The IR camera110(e.g., a digital IR camera) may include at least one IR image sensor402, at least one processor404, memory406, at least one storage device408, and at least one light synch sensor (not shown), as well as other components, equipment, and/or devices commonly included in a digital IR camera, some or all of which may be communicatively coupled at any given time, as shown inFIG. 4. The image sensor302may be implemented as any suitable IR image sensor. The IR camera110may be implemented similarly as and have functionality similarly to the camera108, except that the IR camera110is configured to capture IR images. For example, similar to the camera108, an IR image acquisition frequency of the IR camera110may be synchronized with an IR lighting frequency of IR light devices such that the camera110captures all IR images during the on states (e.g., during leading edges of the on states) of the IR light devices. Additionally, for example, the IR camera110may be configured to capture IR images when the lighting frequency of the light devices is less than the image acquisition frequency.

While the IR camera110exemplarily includes elements as shown, in some embodiments, one or more of the elements of the IR camera110may be omitted, or the IR camera110may include other elements.

In some embodiments, the IR camera110may be omitted. For example, the camera108may be configured to sense light in the IR spectrum and capture IR images. For example, the camera108may have sensitivity to IR light and be configured to capture black and white images based on sensed IR light.

The computing device112may include at least one processor114, at least one memory116, and at least one storage device118, as well as other components, equipment, and/or devices commonly included in a computing device, some or all of which may be communicatively coupled. The processor114may be implemented as any suitable processor, such as a general purpose processor, an FPGA, and/or an image processor. For example, the computing device112may be configured to control the lighting controller104, the camera108, and the IR camera110. For example, the processor114may be configured to receive user input data to change a dimness or a brightness of the light devices, and the processor114may output an instruction to the lighting controller104to change the dimness or brightness of the light devices. Likewise, the lighting controller104and/or the camera108may respond such that the lighting frequency and the image acquisition frequency remain aligned, such as disclosed throughout. The processor114may be configured to run various software applications or computer code stored (e.g., maintained) in a non-transitory computer-readable medium (e.g., memory116and/or storage118) and configured to execute various instructions or operations. The computing device112may be implemented as any suitable computing device. In some embodiments, the computing device112is implemented as a vetronics computing device (e.g., an avionics computing device) in a vehicle, such as the aircraft102, an automobile, or a train. Additionally, for example, the computing device112or the processor114may be implemented as special purpose computers or special purpose processors configured (e.g., programmed) to execute instructions for performing any or all of the operations disclosed throughout. In some embodiments, the system ofFIG. 1may include any suitable number of computing devices112.

While the computing device112exemplarily includes elements as shown, in some embodiments, one or more of the elements of the computing device112may be omitted, or the computing device112may include other elements.

Referring now toFIG. 5, an exemplary graph of an LED lighting cycle at 100% LED power output compared to image acquisition cycles (e.g., camera frame acquisition cycles) of a camera is shown. As shown inFIG. 5, the LED is at 100% power output such that the LED is always in an on state. The image acquisitions (as shown by peaks inFIG. 5) occur while the LED is in an on state such that the camera captures all images during an on state of the LED.

Referring now toFIG. 6, an exemplary graph of LED lighting cycles at 50% LED power output compared to image acquisition cycles (e.g., camera frame acquisition cycles) of a camera, according to inventive concepts disclosed herein, is shown. As shown inFIG. 6, the LED is at 50% power output such that the LED is operating half the time in on states during the peaks and operating half the time in off states during the valleys. The image acquisition frequency is aligned with the light frequencies such that the image acquisitions (as shown by peaks inFIG. 6) occur during the leading edge of the on state of each LED lighting cycle. Each image acquisition occurs while the LED is in an on state such that the camera captures all images during on states of the LED so as to avoid flicker and other artifacts. An advantage of synchronizing the image acquisitions to being aligned with leading edges of on states of the LED may be that the image acquisition remains synchronized with the on states of the light devices even if a duration of each on state is reduced (e.g., so as to dim the light devices).

Referring now toFIG. 7, exemplary overlaid graphs of LED lighting cycles at various LED power outputs compared to various image acquisition cycles of a camera, according to inventive concepts disclosed herein, are shown. The top graph shows an LED lighting cycle at 100% LED power output compared to image acquisition cycles of a camera configured for capturing 60 frames per second, similar toFIG. 5. The second graph shows LED lighting cycles at 50% LED power output compared to image acquisition cycles of a camera configured for capturing 60 frames per second, similar toFIG. 6. The third graph shows LED lighting cycles at 5% LED power output compared to image acquisition cycles of a camera configured for capturing 60 frames per second. The fourth graph shows LED lighting cycles at 2.5% LED power output compared to image acquisition cycles of a camera configured for capturing 30 frames per second. With respect to the fourth graph, as the LED dims to a low level and approaches the camera's image acquisition duration, the mode of the LED system may changes from a PWM mode to a pulse-frequency modulation (PFM) mode, which lengthens the lighting cycle (by increasing the amount of time between on states) to achieve a lower dim level while still allowing each image acquisition to fully occur during an on state of the LED. For second, third, and fourth graphs, the image acquisition frequency is aligned with the lighting frequency such that the image acquisitions (as shown by peaks inFIG. 6) occur during the leading edge of the on state of each LED lighting cycle.

Referring now toFIG. 8, an exemplary embodiment of a method800according to the inventive concepts disclosed herein may include one or more of the following steps. Additionally, for example, some embodiments may include performing one more instances of the method800iteratively, concurrently, and/or sequentially.

A step802may include controlling, by a processor, an image sensor of a camera, the image sensor in view of emitted light from at least one light device, the emitted light having a lighting frequency defined by on states and off states of the at least one light device, the image sensor having an image acquisition frequency.

A step804may include capturing, by the image sensor of the camera, all images during the on states of the at least one light device.

Further, the method800may include any of the operations disclosed throughout.

As will be appreciated from the above, embodiments of the inventive concepts disclosed herein may be directed to a method, a lighting system, and a camera, wherein the lighting system is synchronized with the camera such that the camera captures all images during on states of the lighting system.

As used throughout and as would be appreciated by those skilled in the art, “at least one non-transitory computer-readable medium” may refer to as at least one non-transitory computer-readable medium (e.g., memory116, memory204, memory306, memory406, storage118, storage206, storage308, storage408, or a combination thereof; e.g., at least one computer-readable medium implemented as hardware; e.g., at least one non-transitory processor-readable medium, at least one memory (e.g., at least one nonvolatile memory, at least one volatile memory, or a combination thereof; e.g., at least one random-access memory, at least one flash memory, at least one read-only memory (ROM) (e.g., at least one electrically erasable programmable read-only memory (EEPROM)), at least one on-processor memory (e.g., at least one on-processor cache, at least one on-processor buffer, at least one on-processor flash memory, at least one on-processor EEPROM, or a combination thereof), or a combination thereof), at least one storage device (e.g., at least one hard-disk drive, at least one tape drive, at least one solid-state drive, at least one flash drive, at least one readable and/or writable disk of at least one optical drive configured to read from and/or write to the at least one readable and/or writable disk, or a combination thereof), or a combination thereof).

In the present disclosure, the methods, operations, and/or functionality disclosed may be implemented as sets of instructions or software readable by a device. Further, it is understood that the specific order or hierarchy of steps in the methods, operations, and/or functionality disclosed are examples of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the methods, operations, and/or functionality can be rearranged while remaining within the scope of the inventive concepts disclosed herein. The accompanying claims may present elements of the various steps in a sample order, and are not necessarily meant to be limited to the specific order or hierarchy presented.