METHOD, SYSTEM AND APPARATUS FOR VISUAL EFFECTS

A method, apparatus or system to produce visual effects includes processing a first video signal from a first camera providing video of an object, tracking information from tracking a movement of the object, and a second video signal including information representing at least one of a reflection on the object and a lighting environment of the object to produce a rendered signal representing video in which the tracked object is replaced in real time with a virtual object having one or more of the reflection and the lighting environment of the tracked object.

TECHNICAL FIELD

The present disclosure involves a method, system and apparatus for creating visual effects for applications such as linear, interactive experiences, augmented reality or mixed reality.

BACKGROUND

Any background information described herein is intended to introduce the reader to various aspects of art, which may be related to the present embodiments that are described below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light.

Creating visual effects for applications such as film, interactive experiences, augmented reality (AR) and/or mixed reality applications may involve replacing a portion of an image or video content captured in a real-world situation with alternative content. For example, a camera may be used to capture a video of a particular model of automobile. However, a particular use of the video may require replacing the actual model automobile with a different model while retaining details of the original environment such as surrounding or background scenery and details. Modern image and video processing technology permits making such modifications to an extent that the resulting image or video with the replaced portion, e.g., the different model automobile, may appear at least somewhat realistic. However, creating a sufficient degree of realism typically requires significant post-processing effort, i.e., in a studio or visual effects facility, after the image or video capture has been completed. Such effort may include an intensive and extensive manual effort by creative personnel such as graphic artists or designers with a substantial associated time and cost investment.

In addition to the cost and time required by post processing, adding realism by post processing presents numerous challenges during the initial image or video capture. For example, because effects are added later to create the final images or video, a camera operator or director cannot see the final result while they are behind the camera capturing images or video. That is, a cameraman or director cannot see what they are actually shooting with respect to the final result. This presents challenges with regard to issues such as composition and subject framing There may be a lack of understanding, or an inaccurate understanding, as to how the subject fits into the final scene. Guesswork is required to deal with issues such as the effect or impact of surrounding lighting conditions on the final result, e.g., is the subject properly lit? Thus, there is a need to be able to visualize in-camera the result of editing the actual video with the augmentation process.

SUMMARY

In general, an embodiment comprises a method or system or apparatus providing visualization of photorealistic effects in real time during a shoot.

In accordance with an aspect of the present principles, an embodiment comprises producing visual effects incorporating in real time information representing reflections and/or lighting from one or more sources using an image sensor.

In accordance with another aspect of the present principles, an embodiment comprises producing visual effects for film, interactive experiences, augmented reality or mixed reality including capturing and incorporating in real time reflections and/or lighting from one or more sources using an image sensor.

In accordance with another aspect of the present principles, an embodiment comprises producing visual effects for film, interactive experiences, augmented or mixed reality including capturing and incorporating in real time lighting and/or reflections from one or more sources using at least one of a light sensor and an image sensor.

In accordance with another aspect of the present principles, an embodiment comprises producing visual effects for film, interactive experiences, augmented reality including capturing and incorporating in real time lighting and/or reflections from one or more sources using one or more sensors locationally distinct from a camera providing a video feed or image information to be augmented to produce augmented reality content.

In accordance with another aspect of the present principles, an embodiment comprises producing visual effects such as mixed reality including capturing and incorporating in real time lighting and/or reflections from one or more sources using one or more sensors locationally distinct from a camera providing a video feed to a wearable device worn by a user whose vision is being augmented in mixed reality.

In accordance with another aspect of the present principles, an embodiment comprises a method including receiving a first video feed from a first camera providing video of an object; tracking the object to produce tracking information indicating a movement of the object, wherein a camera array including a plurality of cameras is mounted on the object; receiving a second video signal including video information corresponding to a stitching together of a plurality of output signals from respective ones of the plurality of cameras included in the camera array, wherein the second video signal captures at least one of a reflection on the object and a lighting environment of the object; and processing the first video feed, the tracking information, and the second video signal to generate a rendered signal representing video in which the tracked object has been replaced in real time with a virtual object having reflections and/or a lighting environment of the tracked object.

In accordance with another aspect of the present principles, an embodiment of apparatus comprises one or more processors configured to receive a first video signal from a first camera providing video of an object; track the object to produce tracking information indicating a movement of the object, wherein a camera array including a plurality of cameras is mounted on the object; receive a second video signal including video information corresponding to a stitching together of a plurality of output signals from respective ones of the plurality of cameras included in the camera array, wherein the second video signal captures at least one of a reflection on the tracked object and a lighting environment of the tracked object; and process the first video signal, the tracking information, and the second video signal to generate a rendered signal representing video in which the tracked object has been replaced in real time with a virtual object having at least one of the reflection and the lighting environment of the tracked object.

In accordance with another aspect of the present principles, an embodiment of a system comprises a first camera producing a first video signal providing video of an object; a camera array including a plurality of cameras mounted on the object and having a first processor processing a plurality of output signals from respective ones of the plurality of cameras included in the camera array to produce a second video signal representing a stitching together of the plurality of output signals, wherein the second video signal includes information representing at least one of a reflection on the object and a lighting environment of the object; a second camera tracking the object and producing tracking information indicating a movement of the object; and a second processor processing the first video signal, the tracking information, and the second video signal to generate in real time a rendered signal representing video in which the tracked object has been replaced in real time with a virtual object having at least one of the reflection and the lighting environment of the object.

In accordance with another aspect, any embodiment as described herein may include the tracked object having one or more light sources emitting light from the tracked object that matches the color, directionality and intensity of the light emitted from the virtual object.

In accordance with another aspect, any embodiment as described herein may include a sensor and calculating light and/or reflection maps for one or more virtual objects locationally distinct from the sensor or a viewer, e.g., a camera or a user.

In accordance with another aspect, any embodiment as described herein may include communication of lighting and/or reflection information from one or more sensors using a wired and/or a wireless connection.

In accordance with another aspect, any embodiment as described herein may include modifying the lighting of a virtual object in real time using sampled real-world light sources rather than vice versa.

In accordance with another aspect of the present principles, an embodiment comprises photo-realistically augmenting a video feed from a first camera, such as a hero camera, in real time by tracking an object with a singular camera or multiple camera array mounted on the object to produce tracking information, capturing at least one of reflections on the object and a lighting environment of the tracked object using the single camera or array, stitching outputs of a plurality of cameras included in the camera array in real time to produce a stitched video signal representing reflections and/or lighting environment of the object, communicating the stitched output signal to a processor by a wireless and/or wired connection, wherein the processor processing the video feed, the tracking information, and the stitched video signal to generate a rendered signal representing video in which the tracked object has been replaced in real time with a virtual object having reflections and/or a lighting environment matching that of the tracked object.

In accordance with another aspect of the present principles, any embodiment described herein may include generating a positional matrix representing a placement of the virtual object responsive to the tracking information and generating the rendered signal including the virtual object responsive to the positional matrix.

In accordance with another aspect of the present principles, tracking an object in accordance with any embodiment described herein may include calibrating a lens of the first camera using one or more fiducials that are affixed to the tracked object or a separate and unique lens calibration chart.

In accordance with another aspect of the present principles, any embodiment as described herein may include processing the stitched output signal to perform image-based lighting in the rendered signal.

In accordance with another aspect of the present principles, an embodiment comprises a non-transitory computer readable medium storing executable program instructions to cause a computer executing the instructions to perform a method according to any embodiment of a method as described herein.

It should be understood that the drawings are for purposes of illustrating exemplary aspects of the present principles and are not necessarily the only possible configurations for illustrating the present principles. To facilitate understanding, throughout the various figures like reference designators refer to the same or similar features.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail to avoid obscuring the present disclosure in unnecessary detail.

All examples and conditional language recited herein are intended for instructional purposes to aid the reader in understanding the principles of the disclosure and are to be construed as being without limitation to such specifically recited examples and conditions.

In general, an embodiment in accordance with the present principles comprises a method or system or apparatus providing visualization of photorealistic effects in real time during a shoot. Visual effects, photorealistic effects, virtual objects and similar terminology as used herein are intended to broadly encompass various techniques such as computer-generated images or imagery (CGI), artist's renderings, images of models or objects that may be captured or generated and inserted or included in scenes being shot or produced. Shooting visual effects provides directors and directors of photography a visualization challenge. When shooting, directors and directors of photography need to know how virtual elements will be framed up, whether they are lit correctly, what can be seen in the reflections. An aspect of the present principles involves addressing the described problem.

An exemplary embodiment of a system and apparatus in accordance with the present principles is shown inFIG. 1. InFIG. 1, video signal VIDEO IN is received from a camera such as a so-called “hero” camera which captures video of the activity or movement of an object that is the subject of a particular shoot. Signal VIDEO IN includes information that enables tracking the object, hereinafter referred to as the “tracked object”. Tracking may be accomplished in a variety of ways. For example, the tracked object may include various markings or patterns that facilitate tracking. An exemplary embodiment of tracking is explained further below in regard toFIG. 5.

Also inFIG. 1, a signal REFLECTION/LIGHTING INFORMATION is received. This signal may be generated by one or more sensors or cameras, i.e. an array of sensors or cameras, arranged in proximity to or on the tracked object as explained in more detail below. Signal REFLECTION/LIGHTING INFORMATION provides a representation in real time of reflections on the tracked object and/or the lighting environment of the tracked object. A processor120receives the tracking information from TRACKER110, signal VIDEO IN, and signal REFLECTION/LIGHTING INFORMATION and processes these inputs to perform real time rendering and produce a rendered output signal. The real time rendering operation performed by processor120includes replacing in real time the tracked object in the video of signal VIDEO IN with a virtual object such that the output or RENDERED VIDEO signal represents in real time the virtual object moving in the same or substantially the same manner as the tracked object and visually appearing to have the surroundings, reflections and lighting environment of the tracked object. Thus, signal RENDERED VIDEO inFIG. 1provides a signal suitable for visual effects on linear film and augmented and/or mixed reality.

In more detail,FIG. 2shows the features ofFIG. 1and illustrates camera230, e.g., a hero camera, a tracked object250and an array240of one or more sensors or cameras241to244. Object250may be moving and camera230captures information enabling tracking of object250as described below. Cameras or sensors242to244are illustrated in phantom indicating that they are optional. Also, although the exemplary embodiment of array240is illustrated as including one to four cameras or sensors, array240may include more than four cameras or sensors. Typically, an increased number of cameras or sensors may improve the accuracy of the reflections and lighting information. However, additional cameras or sensors also increases the amount of data that must be processed in real time.

FIG. 6shows an exemplary embodiment of aspects of the exemplary systems ofFIGS. 1 and 2. InFIG. 6, one or more cameras such as in array240ofFIG. 2are shown arranged in a frame or container310intended to be mounted to or in proximity to the tracked object. Various types of cameras or sensors may be used of which professional quality cameras such as those from RED Digital Cinema are an example. Lens330provides image input to camera array240. Lens330may be a “fisheye” type lens enabling 360-degree panoramic capture of the surroundings by array240to ensure complete and accurate capture of reflection and lighting environment information of the tracked object. Image or video information from array240is communicated to a processor such as processor120inFIG. 1orFIG. 2via a connection such as wired connection350in the exemplary embodiment illustrated inFIG. 6. Other embodiments may implement connection350using wireless technology such as that based on WiFi standards well known to one skilled in the art along with or in place of wired connection350shown inFIG. 6.

Turning now toFIG. 3, an exemplary method in accordance with the present principles is illustrated. InFIG. 3, an exemplary method produces output signal RENDERED OUPUT providing a version of a video feed produced by video capture, e.g., by a hero camera, at step310. The video feed produced by video capture at step310represents an object that is tracked, i.e., a tracked object, by the camera. The tracked object includes a camera or sensor array such as array240described above that provides for capturing reflections and/or the lighting environment of the tracked object at step330. Signal RENDERED OUTPUT represents a version of the video feed produced at step310augmented in real time to replace the tracked object with a virtual object appearing photo-realistically in the environment of the tracked object. The video feed produced at step310from a first camera, such as a hero camera is processed at step320to generate tracking information. Tracking the object and generating tracking information at step320may comprise calibrating a lens of the camera producing the video feed, e.g., a hero camera, using fiducials that are affixed to the tracked object.

An exemplary embodiment of the processing involved is described in more detail below. As described above, the lighting environment and reflections information produced at step330may include a plurality of signals produced by a corresponding plurality of cameras or sensors, e.g., by an array of a plurality of cameras mounted on the object. Each of the camera signals may represent a portion of the lighting environment or reflections on the tracked object. At step340, the content of the multiple signals are combined or stitched together in real time to produce a signal representing the totality of reflections and/or lighting environment of the tracked object. At step350, a processor performs real time rendering to produce an augmented video output signal RENDERED OUTPUT. The processing at step350comprises processing the video feed produced at step310, the tracking information produced at step320and the stitched reflections/lighting signal produced at step340to replace the tracked object in the video feed with a virtual object having reflections and/or a lighting environment matching that of the tracked object. In accordance with an aspect of the present principles, an embodiment of the rendering processing occurring at step350may comprise producing a positional matrix representing a placement of the virtual object responsive to the tracking information and generating signal RENDERED OUTPUT including the virtual object responsive to the positional matrix. In accordance with another aspect, processing at step350may comprise processing the stitched output signal to perform image-based lighting in the rendered signal. In accordance with another aspect, the stitching process at step340may occur in a processor in the tracked object such that step340is locationally distinct, i.e., in a different location, from the camera generating the video feed at step310and from the processing occurring at step320and350. If so, step340may further include communicating the stitched signal produced by step340to the processor performing real-time rendering at step350. Such communication may occur by wire and/or wirelessly.

FIG. 4illustrates another exemplary embodiment of a system or apparatus in accordance with the present principles. InFIG. 4, block430illustrates an embodiment of a tracked object which includes a plurality of cameras CAM1, CAM2, CAM3and CAM4generating the above-described plurality of signals representing reflections and or lighting environment of the tracked object, a stitch computer for stitching together the plurality of signals produced by the plurality of cameras to produce a stitched signal, and a wireless transmitter capable of transmitting a high definition (HD) stitched signal in real time wirelessly to unit420. The plurality of cameras CAM1, CAM2, CAM3and CAM4may correspond to camera array240described above and may be configured and mounted in an assembly such as that shown inFIG. 6which may be mounted to a tracked object. Also inFIG. 4, unit420includes the hero camera producing the video feed, a wireless receiver receiving the stitched signal produced and wirelessly transmitted from unit430, and a processor performing operations in real time including tracking as described above and compositing of the virtual object into the video feed to produce the rendered augmented signal. Unit420may also include a video monitor for displaying the rendered output signal, e.g., to enable the person operating the hero camera to see the augmented signal and evaluate whether framing, lighting etc. are as required. Unit420may further include a wireless high definition transmitter for wirelessly communicating the augmented signal to unit410where a wireless receiver receives the augmented signal and provides it to another monitor for viewing of the augmented signal by, e.g., a client or director, to enable real time evaluation of the visual effects incorporated into the augmented signal. An exemplary embodiment of a processor suitable for providing the function of processor120inFIG. 1 or 2, the real-time rendering at step350inFIG. 3, and the processor included in unit420ofFIG. 4may be a processor providing capability such as provided by a video game engine. An exemplary embodiment of a tracking function suitable for generating tracking information as described above in regard to tracker function110inFIGS. 1 and 2and generating tracking information at step320ofFIG. 3is described further below.

FIG. 5illustrates an exemplary embodiment of a planar target that may be mounted in various locations of the tracked object to enable tracking. Each one of a plurality of targets has a unique identifying pattern and accurate two-dimensional corners. Such targets enable fully automatic detection for tracking. For example, images of the targets in the video feed may be associated with time and coordinate data, e.g., GPS data, captured along with the video feed. In addition to tracking, other potential use cases provided by a plurality of such targets include pose estimation and camera calibration. An exemplary embodiment of targets such as that shown inFIG. 5is a target provided by AprilTag. Other approaches to tracking may also be used such as light-based tracking, e.g., lighthouse tracking by Valve.

FIG. 7illustrates an example of the multiple signals produced by camera or sensor array240ofFIGS. 1 and 2and the result of stitching the signals together to produced a stiched signal, e.g., at step340of the method shown inFIG. 3. Images720,730,740, and750inFIG. 7correspond to images captured by an exemplary camera array including four cameras. Each of images720,730,740, and750correspond to images or video captured by a respective one of the four cameras included in the camera array. Image710illustrates the result of stitching to produce a stitched signal incorporating the information from all four of images720,730,740and750. In accordance with an aspect of the present principles, the stitching occurs in real time.

FIG. 8illustrates an exemplary embodiment in accordance with the present principles. InFIG. 8, the vehicle in the right lane of the highway corresponds to a tracked object. The vehicle in the left lane provides the hero camera mounted on the boom extending in front of the vehicle in the left lane. Although unclear from the image inFIG. 8, a camera array such as array240in a configuration such as the exemplary arrangement shown inFIG. 6is mounted at the top center of the vehicle in the right lane, i.e., the tracked object. Reflection and lighting information signals produced by that camera array are stitched in real time by a processor on the tracked object and the resulting stitched signal is transmitted wirelessly to the vehicle in the left lane. Processing capability in the vehicle in the left lane processes the signal from the hero camera mounted on the boom and the stitched signal received from the tracked object to produce a rendered augmented signal in real time as described herein. This enables, for example, a director riding in the vehicle in the left lane to view the augmented signal on a monitor in the vehicle in the left lane and see in real time the appearance of the virtual object in the real-world surroundings of the tracked object including the photorealistic reflection and lighting environment visual effects produced as described herein.

In accordance with another aspect of the present principles, the tracked object may include one or more light sources to emit light from the tracked object. For example, the desired visual effects may include inserting a virtual object that is light emitting, e.g., a reflective metal torch with a fire on the end. If so, one or more lights or light sources, e.g., an array of lights or light sources, may be included in the tracked object. Light from such light sources that is emitted from the tracked object is in addition to any light reflected from the tracked object due to light incident on the tracked object from the lighting environment of the tracked object. The lights or light sources included in a tracked object may be any of various types of light sources, e.g., LEDs, incandescent, fire or flames, etc. If multiple lights or light sources are included in the tracked object, e.g., in an array of light sources, for an application then more than one type of light source may be included, e.g., to provide a mix of different colors, intensities, etc. of light. An array of lights would also enable movement of the lighting from the tracked object, e.g., a sequence of different lights in the array turning on or off, and/or flickering such as for a flame. That is, an array of lights may be selected and configured to emit light from the tracked object that matches the color, directionality, intensity, movement and variations of these parameters of the light emitted from the virtual object. Having the tracked object emit light that matches that emitted from the virtual object further increases the accuracy of the reflection and lighting environment information captured by an array of sensors or cameras240described above, thereby increasing the realism of the augmented signal including the virtual object. As an example,FIG. 9illustrates an exemplary embodiment of a tracked object310shown in more detail inFIG. 10. Also inFIG. 9, following processing of image signals in accordance with the present principles, a virtual object920replaces the tracked object in the rendered image produced on a display device. The exemplary tracked object shown inFIG. 10includes one or more targets320and a fisheye lens330such as those described above in regard toFIGS. 5 and 6. Enlarged images of the exemplary tracked object and the virtual object shown inFIGS. 9 and 10are shown inFIGS. 11 and 12.FIG. 13depicts light1310being emitted by virtual object920. As described above, in accordance with the present principles, such effects may be produced with enhanced realism in the rendered image by including one or more light sources in the tracked object.

It is to be appreciated that the various features shown and described are interchangeable, that is a feature shown in one embodiment may be incorporated into another embodiment.

Although embodiments which incorporate the teachings of the present disclosure have been shown and described in detail herein, the present description illustrates the present principles. It will thus be appreciated that those skilled in the art can readily devise many other varied embodiments that still incorporate these teachings. Having described embodiments which are intended to be illustrative and not limiting, it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments disclosed which are within the scope of the disclosure.

The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (“DSP”) hardware, peripheral interface hardware, memory such as read-only memory (“ROM”) for storing software, random access memory (“RAM”), and non-volatile storage, and other hardware implementing various functions as will be apparent to one skilled in the art.

Herein, the phrase “coupled” is defined to mean directly connected to or indirectly connected with through one or more intermediate components. Such intermediate components may include both hardware and software-based components.

It is to be understood that the teachings of the present principles may be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof. For example, various aspects of the present principles may be implemented as a combination of hardware and software. Moreover, the software may be implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. The machine may be implemented on a computer platform having hardware such as one or more central processing units (“CPU”), a random-access memory (“RAM”), and input/output (“I/O”) interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

It is to be further understood that, because some of the constituent system components and methods depicted in the accompanying drawings may be implemented in software, the actual connections between the system components or the process function blocks may differ depending upon the manner in which the present principles are programmed Given the teachings herein, one of ordinary skill in the pertinent art will be able to contemplate these and similar implementations or configurations of the present principles.

Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present principles are not limited to those precise embodiments, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present principles. All such changes and modifications are intended to be included within the scope of the present principles as set forth in the appended claims.