Patent Publication Number: US-2021185297-A1

Title: Multi-spectral volumetric capture

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of priority under 35 U.S.C. § 119(e) of co-pending U.S. Provisional Patent Application No. 62/947,684, filed Dec. 13, 2019, entitled “Multi-Spectral Volumetric Capture.” The disclosure of the above-referenced application is incorporated herein by reference. 
    
    
     BACKGROUND 
     Field 
     The present disclosure relates to video capture of a subject, and more specifically, to video capture using a combination of multi-spectral infrared and color cameras. 
     Background 
     Green or blue screen is often used to capture motions of a subject and the motions are later composited with custom backgrounds in special effects. However, avoiding using a green screen can be useful because green screen capture imposes restrictions on lighting of the subject and can also introduce “color bleeding” of the green background onto the subject. 
     SUMMARY 
     The present disclosure provides for video capture of a subject using a combination of multi-spectral infrared (IR) and color cameras combined with an IR pattern. 
     In one implementation, a system for video capture of a subject is disclosed. The system includes: a first IR camera, for capturing video data of the subject; a second IR camera, for capturing the video data of the subject; a color camera, for capturing the video data of the subject; a post, where the first IR camera, the second IR camera, and the color camera are attached to the post, and where the color camera is positioned between the first IR camera and the second IR camera; at least one IR light source for illuminating the subject; and a processor connected to the first IR camera, the second IR camera, and the color camera, wherein the processor is configured to: generate depth solve data for the subject using data from the first IR camera and the second IR camera; generate projected color data by using data from the color camera to project color onto the depth solve data; and generate final capture data by merging the depth solve data and the projected color data. 
     In another implementation, a method for video capture of a subject is disclosed. The method includes: generating depth solve data for the subject using data from a first IR camera and a second IR camera; generating projected color data using data from a color camera to project color onto the depth solve data; and generating final capture data by merging the depth solve data and the projected color data, where the color camera is positioned between the first IR camera and the second IR camera. 
     In another implementation, a non-transitory computer-readable storage medium storing a computer program to capture video of a subject is disclosed. The computer program including executable instructions that cause a computer to: generate depth solve data for the subject using data from a first IR camera and a second IR camera; generate projected color data using data from a color camera to project color onto the depth solve data; and generate final capture data by merging the depth solve data and the projected color data, wherein the color camera is positioned between the first IR camera and the second IR camera. 
     Other features and advantages should be apparent from the present description which illustrates, by way of example, aspects of the disclosure. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The details of the present disclosure, both as to its structure and operation, may be gleaned in part by study of the appended drawings, in which like reference numerals refer to like parts, and in which: 
         FIG. 1  is a block diagram of a video system for video capture in accordance with one implementation of the present disclosure; 
         FIG. 2  is a flow diagram of a method for video capture of a subject in accordance with one implementation of the present disclosure; 
         FIG. 3A  is a representation of a computer system and a user in accordance with an implementation of the present disclosure; and 
         FIG. 3B  is a functional block diagram illustrating the computer system hosting a video application in accordance with an implementation of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     As described above, a conventional green screen capture imposes restrictions on lighting of the subject and can also introduce “color bleeding” of the green background onto the subject. 
     Certain implementations of the present disclosure provide systems and methods for processing video data. In one implementation, a video system captures video data for a subject and environment and background, using a combination of multi-spectral IR and color cameras combined with an IR pattern. This system captures volumetric data without using a green screen to separate the subject from the background with depth. This makes it possible to light the subject freely without the concern for the background. 
     After reading the below descriptions, it will become apparent how to implement the disclosure in various implementations and applications. Although various implementations of the present disclosure will be described herein, it is understood that these implementations are presented by way of example only, and not limitation. As such, the detailed description of various implementations should not be construed to limit the scope or breadth of the present disclosure. 
     Features provided in implementations can include, but are not limited to, one or more of the following items: (a) a point triangulation of depth using structure from motion tie points; (b) a combination of infrared and color cameras; (c) cameras are placed in a specific order: IR, color, IR, where the left and right IR cameras are used for depth solve with contribution of two or more IR cameras; (d) IR LED floodlights are used to illuminate subject of IR depth capture to create matte; (e) the center color camera (which captures visible spectrum light data) is used to project color on to the IR camera depth solve; (f) a minimum of one group of three cameras is used, but extensible to N groups of three cameras depending on the subject; and (g) an IR absorbing pigment is used to block the IR light returning to the sensor. 
     In one implementation, the video system is used in a video production or studio environment and includes one or more cameras for image capture, one or more sensors, and one or more computers to process the camera and sensor data. By using structure for motion with a combination of IR and color cameras, the system captures volumetric data with complete freedom for lighting and exposure of the captured subject without using a green screen. Avoiding using a green screen is useful because green screen capture imposes restrictions on lighting of the subject and can also introduce “color bleeding” of the green background onto the subject. 
       FIG. 1  is a block diagram of a video system  100  for video capture in accordance with one implementation of the present disclosure. In the illustrated implementation of  FIG. 1 , the video system  100  includes a first IR camera  110 , a second IR camera  114 , a color camera  112 , a post  140 , at least one IR light source  120 ,  122 , and a processor  130 . 
     In one implementation, the first IR camera  110 , the second IR camera  114 , and the color camera  112  are configured to capture video data of a subject  104 . Two IR cameras are used for depth solves. Further, in one implementation, the first IR camera  110 , the second IR camera  114 , and the color camera  112  are all attached to a post  140  such that the color camera  112  is positioned between the first IR camera  110  and the second IR camera  114 . In another implementation, the color camera  112  is positioned between the first IR camera  110  and the second IR camera  114  without having to attach to the post  140 , but positioned with other means such as being attached together or positioned on a table. In a further implementation, the IR light sources  120 ,  122  are configured to illuminate the subject  104 . 
     In one implementation, the processor  130  is connected to the first IR camera  110 , the second IR camera  114 , and the color camera  112 . The processor  130  is configured to: (a) generate depth solve data  111 ,  115  for the subject using data from the first IR camera  110  and the second IR camera  114 ; (b) generate projected color data  113  by using data from the color camera  112  to project color onto the depth solve data  111 ,  115 ; and (c) generate final capture data  132  by merging the depth solve data  111 ,  115  and the projected color data  113 . The processor  130  uses data from all the cameras to generate the final capture data. Further, the processor  130  uses data from all the cameras to generate the final capture data. 
     In a further implementation, the video system  100  includes at least one more color camera and at least two more IR cameras to capture video data of the subject  104 . In another implementation, the cameras are arranged in groups of three cameras to form a system of nodes, with each node having two IR cameras and a color camera. 
     In a further implementation, the video system  100  also includes a laser  124  attached to the post  140  for projecting a laser pattern on the subject  104 . In one implementation, an 830 nm laser is projected from each camera node position to project a laser pattern on the subject, which helps with contrast and depth capture solve for the two IR cameras. 
     In a further implementation, each of the IR cameras  110 ,  114  includes a filter that removes visible light. In one implementation, the filter is a 700-715 nm high-cut filter. In one implementation, the video system  100  also includes IR floodlights (e.g., 850 nm LED floodlights) positioned around the subject  104  to illuminate the subject  104  with IR. This allows a luminance matte and constant edge around the subject  104  to separate the background  102  from the captured subject  104  to assist with depth generation. In one implementation, the background  102  is configured with an IR blocking paint to create a “black hole” behind the subject  104  that is being captured to keep tie points (which are features identified in two or more images and selected as reference points) for depth generation. The video system  100  uses structure from motion, and relies on image tie points between each view to measure the depth. In one implementation, an IR pigment is applied to the subject  104  to help with contrast and depth capture solve for the two or more IR cameras. The pigment reduces or stops the IR light from scattering into the skin. 
     In one example of a system operation, the video system is used for volumetric capture of a person. Before setting up the cameras, operators decide on coverage of the subject needed for the final production, for example,  180  degrees. The cameras are arranged into a system of nodes, for instance 3 posts with 3 rows of 3 cameras. The posts are arranged around the subject for maximum coverage. The cameras are all shutter synchronized. Then the video system captures the images and data for the subject using the cameras. The system performs a depth solve of the resulting IR data and projects the corresponding color camera data onto the solved depth data. The system merges the resulting depth and projected color data into a final full capture. 
       FIG. 2  is a flow diagram of a method  200  for video capture of a subject in accordance with one implementation of the present disclosure. In the illustrated implementation of  FIG. 2 , depth solve data for the subject is generated, at block  210 , using data from a first IR camera and a second IR camera. Further, projected color data is generated, at block  220 , using data from a color camera to project color onto the depth solve data. At block  230 , final capture data is then generated by merging the depth solve data and the projected color data, wherein the color camera is positioned between the first IR camera and the second IR camera. 
     In one implementation, the subject is illuminated using at least one IR light source. In a further implementation, at least one more color camera and at least two more IR cameras are arranged in groups of three cameras to form a system of nodes, with each node having two infrared cameras and a color camera. The final capture data is then generated using data from all the cameras. In a further implementation, a laser pattern is projected on the subject. In one implementation, the laser pattern is projected from each camera node position. In one implementation, each of the first IR camera and the second IR camera includes a filter that removes visible light. In one implementation, the filter is a high-cut filter in the range of 700-715 nm. In a further implementation, a background panel configured with an IR blocking paint is provided. In a further implementation, an IR pigment is applied to the subject. 
       FIG. 3A  is a representation of a computer system  300  and a user  302  in accordance with an implementation of the present disclosure. The user  302  uses the computer system  300  to implement a video application  390  for implementing a technique for video capture of a subject with respect to the video system  100  of  FIG. 1  and the method  200  of FIG.  2 . 
     The computer system  300  stores and executes the video application  390  of  FIG. 3B . In addition, the computer system  300  may be in communication with a software program  304 . Software program  304  may include the software code for the video application  390 . Software program  304  may be loaded on an external medium such as a CD, DVD, or a storage drive, as will be explained further below. 
     Furthermore, computer system  300  may be connected to a network  380 . The network  380  can be connected in various different architectures, for example, client-server architecture, a Peer-to-Peer network architecture, or other type of architectures. For example, network  380  can be in communication with a server  385  that coordinates engines and data used within the video application  390 . Also, the network can be different types of networks. For example, the network  380  can be the Internet, a Local Area Network or any variations of Local Area Network, a Wide Area Network, a Metropolitan Area Network, an Intranet or Extranet, or a wireless network. 
       FIG. 3B  is a functional block diagram illustrating the computer system  300  hosting the video application  390  in accordance with an implementation of the present disclosure. A controller  310  is a programmable processor and controls the operation of the computer system  300  and its components. The controller  310  loads instructions (e.g., in the form of a computer program) from the memory  320  or an embedded controller memory (not shown) and executes these instructions to control the system. In its execution, the controller  310  provides the video application  390  with a software system, such as to enable video capture of a subject. Alternatively, this service can be implemented as separate hardware components in the controller  310  or the computer system  300 . 
     Memory  320  stores data temporarily for use by the other components of the computer system  300 . In one implementation, memory  320  is implemented as RAM. In one implementation, memory  320  also includes long-term or permanent memory, such as flash memory and/or ROM. 
     Storage  330  stores data either temporarily or for long periods of time for use by the other components of the computer system  300 . For example, storage  330  stores data used by the video application  390 . In one implementation, storage  330  is a hard disk drive. 
     The media device  340  receives removable media and reads and/or writes data to the inserted media. In one implementation, for example, the media device  340  is an optical disc drive. 
     The user interface  350  includes components for accepting user input from the user of the computer system  300  and presenting information to the user  302 . In one implementation, the user interface  350  includes a keyboard, a mouse, audio speakers, and a display. The controller  310  uses input from the user  302  to adjust the operation of the computer system  300 . 
     The I/O interface  360  includes one or more I/O ports to connect to corresponding I/O devices, such as external storage or supplemental devices (e.g., a printer or a PDA). In one implementation, the ports of the I/O interface  360  include ports such as: USB ports, PCMCIA ports, serial ports, and/or parallel ports. In another implementation, the I/O interface  360  includes a wireless interface for communication with external devices wirelessly. 
     The network interface  370  includes a wired and/or wireless network connection, such as an RJ-45 or “Wi-Fi” interface (including, but not limited to 802.11) supporting an Ethernet connection. 
     The computer system  300  includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system), though these components are not specifically shown in  FIG. 3B  for simplicity. In other implementations, different configurations of the computer system can be used (e.g., different bus or storage configurations or a multi-processor configuration). 
     The description herein of the disclosed implementations is provided to enable any person skilled in the art to make or use the present disclosure. Numerous modifications to these implementations would be readily apparent to those skilled in the art, and the principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principal and novel features disclosed herein. 
     Accordingly, additional variations and implementations are also possible. For example, in addition to video production for movies or television, implementations of the system and methods can be applied and adapted for other applications, such as virtual production (e.g., virtual reality environments) for movies, television, games, other volumetric capture systems and environments, or in other capture systems to replace a green screen operation. 
     All features of each of the above-discussed examples are not necessarily required in a particular implementation of the present disclosure. Further, it is to be understood that the description and drawings presented herein are representative of the subject matter which is broadly contemplated by the present disclosure. It is further understood that the scope of the present disclosure fully encompasses other implementations that may become obvious to those skilled in the art and that the scope of the present disclosure is accordingly limited by nothing other than the appended claims.