Automatic control of location-registered lighting according to a live reference lighting environment

According to the disclosed embodiments, a three-dimensional (3D) reference environment having one or more environmental lighting reference (ELR) sources is determined by a controller. Also, characteristics of the one or more ELR sources are tracked by the controller. Then, a lighting environment generated by the one or more ELR sources in the 3D reference environment is replicated by the controller by dynamically and automatically controlling one or more physical light sources that illuminate a physical subject according to the characteristics of the one or more ELR sources.

TECHNICAL FIELD

The present disclosure relates generally to lighting control, and, more particularly, to the automatic control of location-registered lighting according to a reference lighting environment.

BACKGROUND

During film production, among other related practices, lighting of objects is of particular importance. This can be especially true when utilizing special effects techniques, such as chroma key compositing, including the use of a “green screen,” where a simulated three-dimensional space is reproduced to virtually surround the physical objects. Typically, the lighting equipment on-set consists of large, generally un-connected groups of lights. The lights, or light arrays, may be set up individually to provide illumination to the physical objects, which may add significant time to the overall shoot and can consume the majority of hours each day on-set. Not surprisingly, this can add dramatic costs to production and can severely limit the time available to film.

SUMMARY

According to one or more embodiments of the disclosure as described in greater detail below, a three-dimensional (3D) reference environment having one or more environmental lighting reference (ELR) sources is determined. Also, characteristics of the one or more ELR sources are tracked. Then, a lighting environment generated by the one or more ELR sources in the 3D reference environment is replicated by dynamically and automatically controlling one or more physical light sources that illuminate a physical subject according to the characteristics of the one or more ELR sources.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Chroma key compositing, or color keying, is a special effects technique for compositing, e.g., layering, two images or video feeds together according to color hues, or “chroma range.” The technique is used post-production to remove a background, e.g., “green screen,” from the subject of a photo or video. In particular, the technology involves a color range, such as green (or any other uniform and distinct color), in the “top” layer that is made transparent, revealing another image behind. Ideally, no part of the subject being filmed or photographed should duplicate the background color.

Filming with a green screen allows a person or object, e.g., the subject, to be efficiently separated from the studio environment and instead be integrated into a background shot of a different time or place, thereby creating a “composite.” The technique further allows different elements to be shot and combined in unique ways and allows people or objects to be added to an environment that would be too costly or resource-intensive to actually film. Instead, the environment may be created entirely using a computer or as artwork. Use of the green screen is especially common in the newscasting, motion picture, and videogame industries, where a physical subject may be filmed while positioned in front of a large computer-generated imagery (CGI) backdrop, though in actuality, it is a large mono-colored background. The desired result is for the subject and background environment to appear to have been photographed or filmed at the same time and place with the same camera.

Achieving a realistic composite requires accurately coordinating several aspects of the image of the subject and the image of the background. For example, the subject and background should be viewed from consistent perspectives and should exhibit the same imaging properties, such as color balance, sharpness or blurriness, brightness response curves, lens flare, and noise. Manipulating these characteristics such that they are commensurate in both the subject and background is critical for creating an authentic-looking scene.

In addition to the above concerns, a key component in generating a realistic composite involves consistent lighting of the subject and background. That is, the subject should exhibit the same lighting parameters, e.g., shading, highlights, shadows, indirect illumination, and the like, that would have been exhibited if actually present within the background environment. For instance, consider an actor being composited into a concert hall. In this case, the actor's illumination should appear to derive primarily from the concert hall's stage lighting. The actor's illumination should also change in accordance with the changing stage lights, as the characteristics of the stage lighting may be programmed to frequently vary in direction, color, intensity, etc. A myriad of other factors may further affect the illumination of the actor, including the actor's movements, background light obstructions (which may cause shadows), whether the shot is wide or close-up, reflections in the scene, and so forth.

As noted above, current lighting systems are controlled either manually or by a lighting control system, such as digital multiplex (DMX). These lighting systems most often require the input of a human, either in real-time or via a programmed lighting sequence. On a typical film set, for instance, highly skilled artisans may be required to manually place and color each light. Other lighting arrays can be controlled using a video input; however, they do not react dynamically as their background locations, e.g., distance, rotation, elevation, etc., change with respect to the environment or lighting conditions they are seeking to replicate. Moreover, these programmable light arrays are not coordinated in any particular manner, nor are they able to be integrated to a live digital set and composite.

Automated Lighting Reproduction for Live-Action Compositing

The techniques herein are directed to automatic control of location-registered lighting according to a live reference lighting environment. In particular, a real-time, spatially reactive lighting array may be powered according to a real-time environmental lighting reference. As a result, digital lighting scenarios may be automatically replicated in the practical world.

Specifically, according to one or more embodiments of the disclosure as described in detail below, a three-dimensional (3D) reference environment having one or more environmental lighting reference (ELR) sources is determined by a controller. Also, characteristics of the one or more ELR sources are tracked by the controller. Then, a lighting environment generated by the one or more ELR sources in the 3D reference environment is replicated by the controller by dynamically and automatically controlling one or more physical light sources that illuminate a physical subject according to the characteristics of the one or more ELR sources.

Illustratively, the techniques described herein may be performed by hardware, software, and/or firmware. In addition, the techniques herein may be treated as extensions to existing standards for digital communication networks, e.g., DMX, or other communication techniques suitable for controlling environmental lighting and effects, and as such, may be processed by similar components understood in the art that execute those techniques, accordingly.

FIG. 1illustrates an exemplary schematic diagram of a location-registered lighting system automatically controlled according to a live reference lighting environment. As shown inFIG. 1, the automated lighting system100includes a configuration of light arrays110, DMX controller120, animation control module130, light-emitting diodes (LEDs)140, color process box150, and controller160. Each of the above components of the automated lighting system100may be interconnected via suitable communication links, as described further below. It should be noted that the configuration depicted inFIG. 1is for illustration purposes only and should not be treated as limiting the disclosed embodiments to the particular depicted configuration. In particular, though light arrays110and LEDs140are shown as both being present and separate, a system100may only comprise light arrays110, or only LEDs140, or any combination/arrangement of light arrays and LEDs as desired.

The light arrays110may include a grouping of one or more lighting instruments in any suitable arrangement, such as a panel. For example, the light arrays110may be a Fresnel, which typically consists of a metal housing, a reflector, a lamp assembly, and a Fresnel lens. The lighting instruments of the light arrays110may include any type of illumination device suitable for lighting a typical film set, including, for example, LEDs, fluorescent bulbs, strobe lights, tungsten lights, hydrargyrum medium-arc iodide (HMI) lights, and the like.

The light arrays110may be in communication with the DMX controller120via a DMX link125. Notably, although the exemplary system100illustratively utilizes DMX (e.g., DMX controller120and DMX link125), it should be understood that any suitable digital communication technique used to control stage lighting and effects may be utilized, and DMX is merely used for demonstration purposes only.

A DMX network, such as a DMX512 network, may employ a multi-drop bus topology with nodes strung together, e.g., a “daisy chain.” The DMX network may consist of a single DMX controller, e.g., DMX controller120, and one or more slave devices. For example, a lighting console may be employed as the DMX controller120, while the slave devices may include, for example, light arrays110, as well as other devices such as dimmers, fog machines, and the like. Thus, in addition to being in communication with the light arrays110, the DMX controller may further be in communication with additional devices not shown inFIG. 1.

The DMX controller120may transmit lighting control signals to the light arrays110using the DMX link125, which may be a suitable DMX cable, for example. Alternatively, lighting control signals may be wirelessly transmitted from the DMX controller120to the light arrays110. In this case, the DMX controller120may be equipped with a wireless transmitter, while strategically placed receivers may be positioned near the light arrays110to convert the wireless signal back to a conventional DMX wired network signal. The lighting control signals from the DMX controller120may control the lighting of the light arrays110. For example, the lighting control signals may be operable to adjust the intensity, color, direction, and so forth, of the light produced by the light arrays110. Moreover, the lighting control signals may be sent at the command of the controller160via the animation control module130, as described further below.

In response to receiving lighting control signals, the light arrays110may transmit status signals to the animation control module130via the data link135. The data link135may be any suitable communication link, including wired, wireless, and so forth. In particular, the status signals may indicate the positioning, e.g., location, angle, etc., of each of the light arrays110. This way, the animation control module130may be aware of the positioning of each of the light arrays110and LEDs140, thus allowing for all of the light sources in the system100to be effectively coordinated.

The animation control module130may be a computer graphics generation platform, such as Autodesk Maya™, which is generally used to generate three-dimensional assets for use in film, television, game development and architecture. The animation control module130(alternatively referred to as a “3D reference environment determining module”) may be utilized to determine a three-dimensional (3D) reference environment having one or more environmental lighting reference (ELR) sources. The 3D reference environment may, for example, be any simulated (e.g., computer-generated) environment or any real-life environment. The ELR sources may represent light sources—either real or computer-generated—in the 3D reference environment. For instance, where a beach setting is the 3D reference environment, a first ELR source may represent the sun, while a second ELR source may represent sunlight reflecting off nearby buildings. Then, a physical subject (e.g., actor, object, etc.) may be recorded and/or superimposed in a replication (e.g., staged environment) of the 3D reference environment using post-production techniques known in the art. In doing so, the physical subject should be illuminated in accordance with physical light sources that emulate the ELR sources in the 3D reference environment.

In the case that the 3D reference environment is computer-generated, the 3D reference environment may be generated by the animation control module130, or it may be previously generated, archived, and subsequently accessed by the animation control module130. By having knowledge of the 3D reference environment, the animation control module130may determine and track characteristics of the ELR source(s) therein, including, for example, the ELR source's direction, color, location, intensity, and so forth. Notably, the ELR source characteristics may constantly change in conjunction with the changing 3D reference environment (e.g., a virtual camera panning around the reference environment). Thus, using the knowledge of the 3D reference environment, as well as the ELR source(s) therein, the animation control module130may dynamically and automatically control the light arrays110and LEDs140, e.g., by adjusting their position, direction, intensity, and so forth, so as to replicate the ELR sources in the physical world. It should be noted that the dynamic control of the light arrays110and LEDs140to replicate the ELR sources may alternatively, or additionally, be performed by the controller160.

Additionally, the animation control module130may transmit DMX control signals to the DMX controller120(which may then transmit lighting control signals to the light arrays110, as described above) and may receive status signals from the light arrays110and LEDs140via the data links135. Furthermore, the animation control module130may transmit LED control signals to the LEDs140so as to adjust the intensity, color, direction, and so forth, of the light produced by the LEDs. The LED control signals may be transmitted from the animation control module130via the HDMI link145. Although the HDMI link145is illustratively included in the system100, it should be understood that any suitable link may be used for the animation control module130to communicate with the LEDs140, including, for example, a high-definition serial digital interface (HD-SDI) link. Also, the signals transmitted from the animation control module130may optionally traverse the color process box150before reaching the desired destination device.

In response to receiving LED control signals, the LEDs140may transmit status signals to the animation control module130via the data link135, in a similar manner as the light arrays110. In particular, the status signals may indicate the positioning, e.g., location, angle, etc., of each of the LEDs140. Thus, as explained above, the animation control module130may be aware of the positioning of each of the light arrays110and LEDs140, thus allowing for all of the light sources in the system100to be effectively coordinated.

The controller160may effectively manage the automated lighting system100via commands sent to the animation control module130. The controller160may utilize a “prelink” in its communications with the animation control module130in order to link or store frequently used programs for subsequent program executions, thereby reducing overall operation time and increasing system efficiency. It should be understood that although the controller160is depicted as separate from the DMX controller120and animation control module130, the controller160may be implemented as an internal device in either of these components or incorporate the functions of DMX controller120and/or animation control module130therein. Furthermore, the illustrated arrangement of the automated lighting system100inFIG. 1, particularly with respect to the controller160, is for demonstration purposes only and should not be treated as limiting the disclosed techniques to the illustrated configuration.

According to one or more embodiments disclosed herein, the automated lighting system100may dynamically and automatically replicate the lighting of a three-dimensional (3D) reference environment. For example, in a scenario where an actor is superimposed into a 3D CGI concert hall, as referenced above, the primary light source in the virtual concert hall may be stage lights positioned in front of the actor. Thus, physical light arrays in the automated lighting system100, such as the light arrays110, LEDs140, and/or other lighting devices included in the system, can be automatically and dynamically controlled in real-time, e.g., by the controller160, DMX controller120, animation control module130, and/or other suitable components in the automated lighting system100, so as to replicate the illumination generated by the virtual stage lights in the CGI concert hall.

Furthermore, the light arrays110, LEDs140, and/or other lighting devices in the automated lighting system100may change dynamically without any human control input (other than physical location) to reflect their changing positions in the 3D environmental space. This may be accomplished, in part, by location data streaming from the light arrays110and LEDs140to the animation control module, allowing for accurate coordination of the physical light sources and 3D reference space. Moreover, using the data transmitted between the light sources and DMX controller120/animation control module130, the automated lighting system100may convert the location of registered positions of the light arrays110and LEDs140into lighting control inputs in real-time.

As a result, the physical lights, on a film set, for example, can reflect the relevant lighting of their position in the digital environment regardless of whether the lights are moving, the environment is moving, or both. That is, the physical space registration, e.g., tracking, of the light arrays110, LEDs140, and so forth, may correspond to the digital coordinate sets of the environmental lighting reference source, e.g., virtual light source in the 3D environmental space. Therefore, the automated lighting system100may provide for an efficient, integrated system that utilizes a digital set to drive real-time lighting in the physical world.

As a specific example, assume a beach setting as illustrated inFIG. 2. In the illustrative 3D reference environment200, an actor (e.g., “physical subject”)210is located on a beach next to an ocean and surrounded by a number of ELR sources2201-2203. In this example, the ELR sources include lights from a nearby building2201, the sun2202, and sunlight reflecting off the ocean2203. Any variation of ELR sources may exist in the 3D reference environment200, and the ELR sources2201-2203are shown for demonstration purposes only.

As would be expected, the actor210is illuminated by each of the ELR sources2201-2203in the 3D reference environment200in a manner particular to the current state of each respective ELR source. In the techniques herein, the lighting sources, e.g., their color, intensity, etc., may be detected, such as through a camera located within the reference environment taking images of the scene (e.g., 360-degrees worth), or using any other light detection device. By detecting the illumination provided by the physical sources (or digital sources in a computer-generated environment), the techniques herein may convert those environmental lighting sources into a staged environment, as described herein.

Notably, as the characteristics of the ELR sources2201-2203change, e.g., changes in interior lighting sources, weather, the time of day, etc., or even based on scene changes (e.g., filming an actor to be superimposed on a beach, and then quickly changing the scene to allow filming an actor to be superimposed within a concert hall), the illumination of the actor210also changes correspondingly. Thus, when considering a staged environment (e.g., while shooting a film), in order to effectively and believably replicate the lighting environment caused by the ELR sources2201-2203in the 3D reference environment200, the techniques herein disclose a procedure by which one or more physical light sources that illuminate a physical subject may be dynamically and automatically controlled according to the characteristics of the ELR sources2201-2203.

Accordingly, based on the 3D reference environment200, a staged environment may be created having one or more physical light sources that can illuminate a physical subject emulating the ELR sources2201-2203, as if the physical subject were actually in the 3D reference environment200. To this point, as shown inFIG. 3, a staged environment300including an actor310and one or more physical light sources3201-3203may be created to replicate the lighting environment generated by the one or more ELR sources2201-2203in the 3D reference environment200. This may be accomplished by dynamically and automatically controlling the one or more physical light sources3201-3203that illuminate the physical subject310according to the (often varying) characteristics of the one or more ELR sources2201-2203. In essence, the lights of the staged environment300are configured to create lighting of the subject (e.g., actor) based on the lighting the subject would have received had it been located in the reference environment (e.g., standing at the beach).

In the staged environment300, the physical light sources3201-3203may be any variety of light sources suitable for stage lighting, such as an LED panel, for example. InFIG. 3, the physical light sources3201,3202, and3203illustratively correspond to the ELR sources2201,2202, and2203of the 3D reference environment200, respectively. However, separate physical light source units are not necessarily required, as a single lighting unit may be sufficient to replicate multiple ELR sources. Also, a physical light source is not necessarily required to correspond only to one ELR source. Thus, the depiction of the physical light sources3201-3203and ELR sources2201-2203do not limited the disclosed embodiments, but are for demonstration purposes only.

As described above, the characteristics of the ELR sources2201-2203may be tracked over a period of time, which may be predetermined prior to the tracking. Also, the tracking of characteristics may be performed according to a predetermined schedule. These characteristics, such as position, intensity, color, and so forth, may change throughout the period of time. For example, in the beach setting illustrated as the 3D reference environment200, the position of the sun2202will be lowered toward the horizon as sun is setting. Moreover, as the sun sets, the color of the sun2202may change from an intense white to a softer orange or red. As the characteristics of the light emitted from the sun2202change, so too would the light reflecting off the ocean water2203. Moreover, while the building lights2201may be turned down or completely off while the sun2202is shining, after the sun sets, the building lights2201may be illuminated and may emit a bright fluorescent light.

As such, by tracking the characteristics of the ELR light sources2201-2203in the 3D reference environment200(which may be real or computer-generated), the resultant lighting environment may be replicated in the staged environment300by dynamically and automatically controlling the one or more physical light sources3201-3203that illuminate the physical subject310accordingly. For example, to replicate an early afternoon at the beach200, the physical light sources3202and3203, which may correspond to the sun2202and reflections2203, respectively, may be positioned at the appropriate angle/height and may emit a bright whitish light onto the actor310. Conversely, to replicate a night at the beach200, the physical light source3201, which may correspond to the building lights2201, may emit a bright fluorescent light onto the actor310, whereas the physical light sources3202and3203may be turned off completely. Alternatively, the physical light sources3202and3203may emit a soft whitish light to simulate moonlight. By dynamically and automatically controlling the physical light sources3201-3203, as described above, a realistic and believable lighting environment may be achieved in a staged environment. Effectively, therefore, the system herein converts detected or determined light from a reference environment, into light source controls for use within a staged environment to produce the same lighting effect on a subject within that environment.

Furthermore, the physical light sources3201-3203in the staged environment300can be tracked in physical space so they are registered to a corresponding location in the 3D reference environment200. Therefore, the physical light sources3201-3203may respond to the controller (e.g., controller120and/or160) based on where it is located physically on the set. For instance, if a light (e.g., ELR light source) were moved from one side of the 3D reference environment200to another (e.g., from the building lights-side (2201) to the sun-side (2202)), a corresponding physical light source320in the staged environment300may modify its lighting characteristics (in the manner described above) according to its physical location on the set. That is, the physical light source320may respond to the controller by modifying its lighting characteristics using additional knowledge of its own physical location on set. As a result, each of the physical light sources3201-3203may perform in a manner unique to their respective locations. The location of the physical light sources3201-3203may be communicated to the controller so that the controller is aware of the location of each light source in the staged environment300. The communication may be achieved in a variety of ways, such as using an RF locator or other communication means illustrated inFIG. 1, for example.

Optionally, a green screen330(or other similar technique) may be utilized in the staged environment300and positioned behind the actor310in order to further create a believable and realistic scene. Moreover, one or more cameras340may be positioned so as to record the actor310, who is being illuminated by the dynamically and automatically controlled physical light sources3201-3203.

FIG. 4illustrates an example simplified procedure for automatic control of location-registered lighting according to a live reference lighting environment. The procedure400may start at step405, and continues to step410, where, as described in greater detail above, the location of registered positions of the physical illumination instruments may be converted into lighting control inputs in real-time, according to a simulated environmental space.

At step410, a 3D reference environment having one or more ELR sources is determined. A “3D reference environment determining module,” e.g., animation control module130, may be utilized to determine the 3D reference environment, as well as the ELR sources therein. The 3D reference environment may, for example, be any simulated environment (e.g., CGI, generated by the animation control module130) or any real-life environment. The ELR sources may represent light sources—either real or computer-generated—in the 3D reference environment.

At step415, characteristics of the one or more ELR sources are determined/tracked. The characteristics of the one or more ELR sources may be determined based on various sensing/imaging devices, such as camera images of the environment, illumination detectors, etc., and may occur over a period of time. That is, these characteristics, such as position, intensity, color, and so forth, may change throughout the period of time.

Then, at step420, a lighting environment generated by the one or more ELR sources in the 3D reference environment is replicated by dynamically and automatically controlling one or more physical light sources that illuminate a physical subject according to the characteristics of the one or more ELR sources. By dynamically and automatically the physical light sources, as described above, a realistic and believable lighting environment may be achieved in a staged environment.

The simplified procedure400illustratively ends at step425. The techniques by which the steps of procedure400may be performed, as well as ancillary procedures, parameters, and apparatuses performing the same, are described in detail above. It should be noted that certain steps within procedure400may be optional, and the steps shown inFIG. 4are merely examples for illustration. Certain other steps may be included or excluded as desired. Further, while a particular order of the steps is shown, this ordering is merely illustrative, and any suitable arrangement of the steps may be utilized without departing from the scope of the embodiments herein.

The techniques described herein, therefore, provide for devices and procedures which may be advantageously utilized during scenarios involving chroma key compositing, such as film production. In particular, the disclosed techniques and devices greatly simplify on-set workflow, and as a result, dramatically save time required to set up initial lighting and change the lighting, as well as saving costs in lighting rentals and on-set labor needed for manually placing and adjusting lights for production. Even more, the disclosed techniques and devices may provide the possibility of creating more realistic and dynamic lighting scenarios.

While there have been shown and described illustrative embodiments that provide for automatic control of location-registered lighting according to a live reference lighting environment, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the embodiments herein, with the attainment of some or all of their advantages. For instance, it is expressly contemplated that the components and/or elements described herein may be embodied as non-transitory computer readable media on a computer readable medium containing executable program instructions executed by a processor, controller or the like. Examples of the computer readable mediums include, but are not limited to, ROM, RAM, compact disc (CD)-ROMs, magnetic tapes, floppy disks, flash drives, smart cards and optical data storage devices. The computer readable recording medium can also be distributed in network coupled computer systems so that the computer readable media is stored and executed in a distributed fashion, e.g., by a telematics server or a Controller Area Network (CAN). Additionally, it is understood that a number of the devices and procedures herein may be executed by at least one controller. The term “controller” refers to a hardware device that includes a memory and a processor. The memory is configured to store program instructions, and the processor is specifically configured to execute said program instructions to perform one or more processes which are described further below. Accordingly this description is to be taken only by way of example and not to otherwise limit the scope of the embodiments herein. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the embodiments herein.