Patent Description:
<CIT> is directed to detecting video recapture of an image rendered by an image rendering device in which light intensity is controlled through periodic variation of the light intensity of a light source.

<CIT> is directed to an image control apparatus that includes a flicker detecting section configured to detect a flicker component of a light source and a controlling section configured to control timing of imaging according to a detection result by the flicker detection section.

<CIT> is directed to an imaging apparatus in which flicker is prevented.

One implementation of the present disclosure is a video camera system as set out in claim <NUM>.

Another implementation of the present disclosure is a method as set out in claim <NUM>. Preferably, the method further comprises: calculating, by the camera controller, the luminance parameter to include an average luminance of each video frame. It is also preferred that the method comprises: generating, by the camera controller, the plurality of luminance amplitude values mapped to the plurality of frequencies based on a plurality of video frames corresponding to a duration of time, the duration of time including at least one of a predetermined duration of time and a duration of time received via a user input. Additionally, it is preferred that the light received by the sensor includes at least one of light outputted by a light emitting diode (LED) electric light unit and light emitted by an LED electric light unit and reflected by an object.

Another implementation of the present disclosure is a camera controller as set out in claim <NUM>. Preferably, the camera controller comprises instructions that cause the one or more processors to: generate the plurality of luminance amplitude values mapped to the plurality of frequencies based on a plurality of video frames corresponding to a duration of time, the duration of time including at least one of a predetermined duration of time and a duration of time received via a user input.

Those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the devices and/or processes described herein, as defined solely by the claims, will become apparent in the detailed description set forth herein and taken in conjunction with the accompanying drawings.

The present disclosure generally relates to the field of video camera systems, and more particularly to systems and methods of eliminating video flicker caused by light emitting diode (LED) luminaire duty cycling to maintain brightness and control power consumption. Referring generally to the figures, systems and methods in accordance with the present disclosure can effectively address video flicker resulting from the use of LED luminaires. Existing luminaires, such as fluorescent luminaires, flicker in line with the frequency of the mains power supply (e.g., based on the corresponding cyclical fluctuation in current received by the luminaires), this frequency typically being <NUM> or <NUM>. As such, existing video camera systems may use a predetermined flicker setting based on knowing the frequency to be either <NUM> or <NUM> to avoid the effects of flicker in the captured video stream, such as to execute frame capture during an on part of the cycle of the flickering luminaires. However, LED luminaires do not have to be, and often are not, tied to the frequency of the mains power supply. Rather, flicker and other effects associated with changes in the light outputted by the LED luminaires may result from operation of a controller of the LED luminaires, such as a pulse width modulation (PWM) controller, that drives the LED luminaires to achieve a specific brightness. This does not have to be synchronized to the frequency of the mains power supply, and in fact, the luminaires may be powered by other modalities, such as Power over Ethernet (PoE) or direct current (DC) power. This can result in significant flicker in the video that cannot be eliminated by existing <NUM>/<NUM> settings. In addition, as the location in which the video camera system is operating may have various LED luminaires of different manufacturers and/or being controlled in different manners, it may be difficult to identify a predetermined setting that would universally eliminate flicker from all the LED luminaires (e.g., by simply selecting a different frequency than <NUM> or <NUM>).

The present solution can detect and eliminate flicker from LED electric light units by calculating specific parameters regarding the light outputted by the LED electric light units, and appropriately adapting camera operation such as by adapting a shutter rate. For example, systems and methods in accordance with the present disclosure can calculate an average luminosity for each video frame, determine a periodicity and/or frequency of the flicker based on the average luminosity, and adapt a shutter rate of the camera responsive to the periodicity.

Referring to <FIG> among others, an environment <NUM> in which a video camera system <NUM> operates is depicted. The environment <NUM> includes one or more electric light units <NUM>. The one or more electric light units <NUM> may include luminaires. For example, the one or more electric light units <NUM> can include one or more LED units. The one or more electric light units <NUM> may also include fluorescent luminaires.

The one or more electric light units <NUM> output light to illuminate one or more objects <NUM>. The video camera system <NUM> includes one or more video capture devices <NUM>, which can receive light from a respective field of view (including light received directly from the one or more electric light units <NUM> and/or reflected by the one or more objects <NUM>), and generate image data (e.g., one or more image frames; a stream of images; video data) based on the received light. In some embodiments, the one or more electric light units <NUM> may output light characterized by flicker or other changes in brightness as a function of time, which may in turn be captured by the one or more video capture devices <NUM> and represented in the image and data that the one or more video capture devices <NUM> generate.

Referring now to <FIG>, a video camera system <NUM> is depicted. The video camera system <NUM> can incorporate features of the video camera system <NUM> described with reference to <FIG>. The video camera system <NUM> includes at least one video capture device <NUM>. The at least one video capture device <NUM> includes a sensor <NUM>, a shutter <NUM>, and a frame processor <NUM>. The sensor <NUM> detects image and/or video data based on light received by the sensor <NUM>, and outputs the image and/or video data (e.g., outputs a signal representing the image and/or video data) to the frame processor <NUM>. The sensor <NUM> can including various sensor devices, such as charge coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), or N-type metal-oxide-semiconductor (NMOS) devices. The shutter <NUM> can be opened and/or closed responsive to a received shutter control signal in order to selectively permit light to be received at the sensor <NUM>. The frame processor <NUM> can generate one or more video frames <NUM> (e.g., image frames including a plurality of pixels) representative of the image and/or video data received from the sensor <NUM>. The frame processor <NUM> can use a clock to assign a timestamp to each generated video frame <NUM>, the timestamp representing a time of capture of the corresponding video frame <NUM>.

The frame processor <NUM> can output the one or more video frames <NUM> (e.g., as a video stream) to a video processor <NUM>. The video processor <NUM> can perform various video processing functions on the received video frames <NUM>, such as to encode the received video frames <NUM> into a compressed format.

The video camera system <NUM> includes a camera controller <NUM> that receives the one or more video frames <NUM> from the frame processor <NUM> and can control operation of the at least one video capture device <NUM> (e.g., control operation of the shutter <NUM>) based on the one or more video frames <NUM>, such as to maintain brightness, control power consumption, and/or eliminate flicker.

The camera controller <NUM> includes a processor <NUM> and memory <NUM>. The processor <NUM> can be a general purpose or specific purpose processor, an application specific integrated circuit (ASIC), one or more field programmable gate arrays (FPGAs), a group of processing components, or other suitable processing components. The processor <NUM> can execute computer code or instructions stored in memory <NUM> or received from other computer readable media (e.g., CDROM, network storage, a remote server, etc.).

The memory <NUM> can include one or more devices (e.g., memory units, memory devices, storage devices, etc.) for storing data and/or computer code for completing and/or facilitating the various processes described in the present disclosure. Memory <NUM> can include random access memory (RAM), read-only memory (ROM), hard drive storage, temporary storage, non-volatile memory, flash memory, optical memory, or any other suitable memory for storing software objects and/or computer instructions. Memory <NUM> can include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. Memory <NUM> can be communicably connected to processor <NUM> via camera controller <NUM> and may include computer code for executing (e.g., by processor <NUM>) one or more processes described herein. When processor <NUM> executes instructions stored in memory <NUM>, processor <NUM> generally configures the camera controller <NUM> to complete such activities.

The camera controller <NUM> includes a frame parameter calculator <NUM>. The frame parameter calculator <NUM> calculates at least one parameter <NUM> regarding the video frame <NUM>. The frame parameter calculator <NUM> can calculate at least one of a brightness parameter and a luminance parameter of the video frame <NUM>. For example, the frame parameter calculator <NUM> can calculate an average luminance of the video frame <NUM>, such as by identifying a luminance corresponding to each pixel of the video frame <NUM> and averaging the identified luminances. The frame parameter calculator <NUM> may calculate the average luminance as a weighted average, such as by applying greater weights to certain pixels of the video frame <NUM> relative to other pixels of the video frame <NUM>. The frame parameter calculator <NUM> can output the at least one parameter <NUM> together with the timestamp assigned to the video frame <NUM>.

The camera controller <NUM> includes a frequency domain analyzer <NUM>. The frequency domain analyzer <NUM> can receive the at least one parameter <NUM> (and assigned timestamp) from the frame parameter calculator <NUM>, and execute a frequency analysis based on the at least one parameter <NUM>. The frequency domain analyzer <NUM> can execute the frequency analysis for a duration of time which may be predetermined or adjusted based on user input. In some embodiments, the frequency domain analyzer <NUM> executes the frequency analysis to generate a plurality of luminance amplitude values mapped to corresponding frequencies <NUM> at which those luminance amplitude values occur for video frames <NUM> during the duration of time. For example, given a duration of time of eight seconds and a frame rate of the video frames <NUM> of twenty four frames per second, the frequency domain analyzer <NUM> can receive a plurality of video frames <NUM> from the duration of time (e.g., the one hundred ninety two frames of the eight seconds, each having an average luminance value), and generate a data structure mapping, to each of a plurality of frequencies, a corresponding luminance amplitude value.

The camera controller <NUM> includes a control signal generator <NUM>. The control signal generator <NUM> can receive the plurality of luminance amplitude values mapped to corresponding frequencies <NUM> from the frequency domain analyzer <NUM>, and generate a control signal <NUM> to control operation of the at least one video capture device <NUM> using the control signal <NUM>. For example, the camera controller <NUM> can generate the control signal <NUM> to control a shutter speed of the shutter <NUM> of the at least one video capture device <NUM> to be a target shutter speed. The camera controller <NUM> can use the control signal <NUM> to set the target shutter speed to a value that reduces or eliminates luminance variations while maximizing maximum luminance for the resulting video.

In some embodiments, the control signal generator <NUM> generates the target shutter speed to be a value that corresponds to a highest expected luminance for the video frames <NUM> based on the plurality of luminance amplitude values mapped to corresponding frequencies <NUM>. For example, the control signal generator <NUM> can generate the target shutter speed to correspond to a frequency of mapped to the highest luminance amplitude value, or to a multiple or a fraction (e.g., divisor) thereof. For example, the control signal generator <NUM> can generate the target shutter speed to be equal to the frequency of the highest luminance amplitude value or a multiple or fraction thereof (e.g., if frequency of highest luminance amplitude value is <NUM>, the shutter speed can be set to <NUM>/<NUM>, a multiple thereof, such as <NUM>/<NUM>, or a fraction thereof, such as <NUM>/<NUM>).

In some embodiments, the control signal generator <NUM> modifies the targets shutter speed based on at least one of a maximum shutter speed and a minimum shutter speed. For example, the control signal generator <NUM> can generate an initial target shutter speed based on the plurality of luminance amplitude values mapped to corresponding frequencies <NUM>, compare the initial target shutter speed to the at least one of the maximum shutter speed and the minimum shutter speed, adjust the initial target shutter speed to be equal to the maximum shutter speed if the initial target shutter speed is greater than the maximum shutter speed, and/or adjust the initial target shutter speed to be equal to the minimum shutter speed if the initial target shutter speed is less than the minimum shutter speed. The control signal generator <NUM> can then generate the control signal <NUM> using the adjusted target shutter speed.

The control signal generator <NUM> can iteratively and/or continuously adjust the target shutter speed based on factors representative of flicker in the video frames <NUM>. For example, as the shutter speed of the shutter <NUM> of each video capture device <NUM> is modified (based on control signal <NUM>), the plurality of luminance amplitude values mapped to corresponding frequencies <NUM> will change. As such, the control signal generator <NUM> can vary the target shutter speed over a plurality of iterations until an iteration condition is satisfied. The iteration condition may be based on a maximum value of a parameter representative of flicker, such as a change or variance of the plurality of luminance amplitude values, so that the control signal generator <NUM> can vary the target shutter speed until the parameter representative of flicker is less than the maximum value.

The control signal generator <NUM> can generate the target shutter speed to minimize flicker (e.g., change in luminance). For example, the control signal generator <NUM> can determine that flicker has not been eliminated based on determining that a change in luminance across cycles (e.g., between consecutive video frames <NUM>; between each iteration of calculating the plurality of luminance amplitude values mapped to corresponding frequencies <NUM>), and adjust the target shutter to minimize the change in luminance and/or reduce the change in luminance to be less than a threshold change in luminance. For example, if there are multiple LED electric light units that have differing flicker frequencies or PWM control schemes, it may be difficult to consistently eliminate flicker; as such, the control signal generator <NUM> can still control the shutter speed to reduce flicker as much as possible.

In some embodiments, the video camera system <NUM> includes a user interface <NUM>. The user interface <NUM> can receive user input and present information regarding operation of the video camera system <NUM>, such as the target shutter speed and the video frames <NUM>. The user interface <NUM> may include one or more user input devices, such as buttons, dials, sliders, or keys, to receive input from a user. The user interface <NUM> may include one or more display devices (e.g., OLED, LED, LCD, CRT displays), speakers, tactile feedback devices, or other output devices to provide information to a user.

In some embodiments, the video camera system <NUM> includes a communications circuit <NUM>. The camera controller <NUM> can use the communications circuit <NUM> to communicate with remote entities (including one or more of the at least one video capture device <NUM> if the camera controller <NUM> is remote from the one or more of the at least one video capture device <NUM>). The communications circuit <NUM> can include wired or wireless interfaces (e.g., jacks, antennas, transmitters, receivers, transceivers, wire terminals, etc.) for conducting data communications with various systems, devices, or networks. For example, the communications circuit <NUM> can include an Ethernet card and port for sending and receiving data via an Ethernet-based communications network. The communications circuit <NUM> can include a WiFi transceiver for communicating via a wireless communications network. The communications circuit <NUM> can communicate via local area networks (e.g., a building LAN), wide area networks (e.g., the Internet, a cellular network), and/or conduct direct communications (e.g., NFC, Bluetooth). In some embodiments, the communications circuit <NUM> can conduct wired and/or wireless communications. For example, the communications circuit <NUM> can include one or more wireless transceivers (e.g., a Wi-Fi transceiver, a Bluetooth transceiver, a NFC transceiver, a cellular transceiver).

Referring now to <FIG>, a method <NUM> of operating a video camera system is depicted. The method <NUM> can be performed using various systems and devices described herein, including the video camera system <NUM>.

At <NUM>, a video frame is received. The video frame may be received by a camera controller. The video frame may be received from at least one video capture device. The video frame may have an assigned timestamp indicating a time at which the video frame was captured. A plurality of video frames may be received over time.

At <NUM>, a parameter regarding the video frame is calculated by the camera controller. The parameter may include at least one of a brightness parameter and a luminance parameter of the video frame. For example, the camera controller can calculate the parameter to include an average luminance by identifying a luminance corresponding to each pixel of the video frame and averaging the identified luminances.

At <NUM>, a frequency analysis is executed based on the parameter regarding the video frame by the camera controller. The frequency analysis can be executed based on a plurality of values of the parameter regarding the video frame for a duration of time. The camera controller can execute the frequency analysis to generate a plurality of luminance amplitude values mapped to corresponding frequencies at which those luminance amplitude values occur for video frames during the duration of time.

At <NUM>, an initial target shutter speed is determined based on the frequency analysis by the camera controller. Generating the initial target shutter speed can include identifying at least one of a frequency mapped to a highest luminance amplitude value of the plurality of luminance amplitude values, a multiple of the frequency mapped to the highest luminance amplitude value, and a fraction of the frequency mapped to the highest luminance amplitude value.

At <NUM>, the initial target shutter speed is compared to at least one of a maximum shutter speed and a minimum shutter speed by the camera controller. If the initial shutter speed is greater than the maximum shutter speed or less than the minimum shutter speed, then at <NUM>, the camera controller can adjust the target shutter speed. For example, the camera controller can adjust the target shutter speed to be equal to the maximum shutter speed if the initial target shutter speed is greater than the maximum shutter speed, and can adjust the target shutter speed to be equal to the minimum shutter speed if the initial target shutter speed is less than the maximum shutter speed.

At <NUM>, the camera controller transmits a control signal generated based on the target shutter speed to the video capture device to cause the at least one video capture device to operate at the target shutter speed. For example, the camera controller can use the control signal to cause a shutter of the at least one video capture device to operate at the target shutter speed.

At <NUM>, the camera controller can evaluate a flicker parameter, such as change in luminance, to determine whether the flicker has been eliminated. For example, the camera controller can compare the flicker parameter to a maximum flicker parameter threshold and determine that flicker has been eliminated responsive to the flicker parameter being less than the maximum flicker parameter threshold. Responsive to determining that flicker has been eliminated, the camera controller can maintain operation at the target shutter speed, such as by discontinuing transmission of the control signal, continuing to transmit the control signal at the target shutter speed, and/or continuing to evaluate the flicker parameter (e.g., <FIG> depicts the camera controller continuing to evaluate the flicker parameter). Responsive to determining that flicker has not been eliminated, the camera controller can adjust the target shutter speed.

The construction and arrangement of the systems and methods as shown in the various exemplary embodiments are illustrative only. Modifications may be made in the design, operating conditions and arrangement of the exemplary embodiments without departing from the scope of the invention as defined by the appended claims.

References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms. References to at least one of a conjunctive list of terms may be construed as an inclusive OR to indicate any of a single, more than one, and all of the described terms. For example, a reference to "at least one of 'A' and 'B'" can include only 'A', only 'B', as well as both 'A' and 'B'. Such references used in conjunction with "comprising" or other open terminology can include additional items.

The present disclosure contemplates methods, systems and program products on any machine-readable media for accomplishing various operations. The embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Embodiments within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

Claim 1:
A video camera system (<NUM>), comprising:
a video capture device (<NUM>) comprising:
a sensor (<NUM>) configured to receive light;
a shutter (<NUM>) configured to be opened and/or closed to selectively permit the light to be received at the sensor (<NUM>) based on a shutter speed; and
a frame processor (<NUM>) configured to generate a plurality of video frames (<NUM>) based on the light received at the sensor (<NUM>); and
a camera controller (<NUM>) configured to receive the plurality of video frames, the camera controller (<NUM>) comprising:
a frame parameter calculator (<NUM>) configured to calculate luminance parameters of the plurality of frames;
a frequency domain analyzer (<NUM>) configured to execute a frequency domain analysis based on the luminance parameters to generate a plurality of luminance amplitude values mapped to a plurality of corresponding frequencies; and
a control signal generator (<NUM>) configured to:
generate a target shutter speed based on the plurality of luminance amplitude values mapped to the plurality of corresponding frequencies, the target shutter speed comprising a target value corresponding to a frequency assigned to the highest luminance value of the plurality of luminance values, a multiple of the target value, or a fraction of the target value,
determine whether the target shutter speed is at least one of greater than or equal to a maximum shutter speed and less than or equal to a minimum shutter speed, and at least one of:
(<NUM>) adjust the target shutter speed to be equal to the maximum shutter speed if the target shutter speed is greater than the maximum shutter speed; and
(<NUM>) adjust the target shutter speed to be equal to the minimum shutter speed if the target shutter speed is less than the minimum shutter speed;
transmit a control signal based on the adjusted target shutter speed to cause the shutter (<NUM>) to operate at the adjusted target shutter speed; and
evaluate a change in luminance corresponding to the plurality of video frames (<NUM>) and readjust the target shutter speed responsive to the change in luminance being greater than a change in luminance threshold, to cause the change in luminance to be less than the change in luminance threshold.