Patent Description:
Three dimensional (3D) rendering is the computer graphics process of automatically converting 3D wire frame models into two dimensional (2D) images with 3D photorealistic effects or non-photorealistic rendering on a computer. A wire frame model is a visual presentation of a 3D or physical object used in 3D computer graphics. A scene description language can be used to describe a scene that is to be rendered by a 3D renderer into a 2D image. In the process of rendering a 2D image of a scene out of a 3D scene description, each pixel for the 2D image may receive contributions (e.g., image fragments) from multiple objects in the scene. In order to produce a single color value for each pixel, these image fragments are generally combined together in a process that is usually referred to as "compositing". To correctly account for occlusion (e.g., hidden objects), foreground fragments should generally fully occlude the background fragments. If the foreground fragments are semi-transparent, the foreground fragments should be composited on top of the background fragments using a physically-based blending technique, such as an "over" operation. Depending on which order the fragments are composited in, a different result for the pixel color may be obtained. The 3D scene description can be inaccurate, and as the scene is animated, small errors in the surface positions may cause the fragment order to change abruptly. Such sudden changes may cause large discontinuous changes in the final color of the composited pixel, showing up as spatial aliasing in a 3D image or flickering in a video.

<NPL>), an image based rendering system that renders multiple frames per second on a PC. The method performs warping from an intermediate representation called a layered depth image (LDI). An LDI is a view of the scene from a single input camera view, but with multiple pixels along each line of sight. When n input images are preprocessed to create a single LDI, the size of the representation grows linearly only with the observed depth complexity in the n images, instead of linearly with n. Moreover, because the LDI data are represented in a single image coordinate system, McMillan's warp ordering algorithm can be successfully adapted. As a result, pixels are drawn in the output image in back to front order.

<CIT> proposes methods and apparatus for hidden surface removal with soft occlusion. Soft occlusion methods are described that treat surfaces as having a degree of uncertainty in depth. The soft occlusion methods may, for example, be used to remove artifacts from rendered images due to nearly coplanar surfaces or to render novel effects such as soft intersections between objects including consistent shadows and other global illumination effects. The soft occlusion methods may compute the 'expected' or average image given depth probability density functions. This has the effect of visually blending together surfaces that are close together in depth, leading to soft intersections. The computation of soft occlusion may be achieved analytically, for certain probability density functions, or stochastically. The stochastic soft occlusion methods extend the approach to a probability distribution of models, which allows for the effects of shadows and other global illumination effects to be included.

The following presents a simplified summary of various aspects of this disclosure in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements nor delineate the scope of such aspects. Its purpose is to present some concepts of this disclosure in a simplified form as a prelude to the more detailed description that is presented later.

In an aspect of the present disclosure, a method comprises creating an output image of a scene, as defined in claim <NUM>. The method may be carried out by a processing device.

In one implementation, the pre-defined thickness is defined in a disparity space. The pre-defined thickness may also be defined in a depth space. According to the invention, the determining of the color of the volume span comprises identifying the one or more of the 3D fragments that contribute color to the volume span, and determining the color contribution of the identified 3D fragments. Identifying the one or more of the plurality of 3D fragments that contribute color to the volume span may in some implementations comprise identifying one or more of the plurality of 3D fragments has a portion that overlaps the volume span. In one implementation, the determining of the color contribution comprises determining a length of the volume span, determining a total thickness of individual 3D fragments of the identified one or more 3D fragments, and dividing the length of the volume span by the total thickness. In an implementation, determining the color contribution of the identified one or more of the plurality of 3D fragments comprises determining the sum of the colors of the identified 3D fragments and multiplying the determined sum of the colors with the contribution of the identified 3D fragments. In one implementation, the creating of the volume spans for the pixel based on the 3D fragments comprises marking start events and end events for the 3D fragments, ordering the start events and end events sequentially, and defining start events and end events for the volume spans for the pixel based on a sequential ordering of the start events and end events for the 3D fragments.

There is also described herein an apparatus comprising means for creating an output image of a scene using 2D images of the scene. For a pixel in the output image, means for identifying, in the output image, 2D fragments corresponding to the pixel, means for converting the 2D fragments into 3D fragments, means for creating volume spans for the pixel based on the 3D fragments, means for determining a color of a volume span of the volume spans based on color contribution of respective one or more of the 3D fragments for the volume span, and means for determining a color of the pixel for the output image from determined colors of the volume spans. In one implementation, the 2D images are captured by multiple cameras. According to the invention, the means for converting the 2D fragments into the 3D fragments comprises means for adding a pre-defined thickness to individual 2D fragments.

In one implementation, the pre-defined thickness is defined in a disparity space. In one implementation, the means for determining the color of the volume span comprises means for identifying the one or more of the 3D fragments that contribute color to the volume span, and means for determining the color contribution of the identified 3D fragments. In one implementation, the means for determining of the color contribution comprises means for determining a length of the volume span, means for determining a total thickness of individual 3D fragments of the identified one or more 3D fragments, and means for dividing the length of the volume span by the total thickness. In one implementation, the means for creating of the volume spans for the pixel based on the 3D fragments comprises means for marking start events and end events for the 3D fragments, means for ordering the start events and end event for the 3D fragments sequentially, and means for defining start events and end events for the volume spans for the pixel based on a sequential ordering of the start events and end events for the 3D fragments.

In additional implementations, computing devices for performing the operations of the above described implementations are also implemented. Additionally, in implementations of the disclosure, a computer readable storage media may store instructions for performing the operations of the implementations described herein.

It will be appreciated that implementations can be combined such that features described in the context of one implementation can be combined with features of other implementations. In particular, whilst the above describes various implementations, it will be appreciated that the features described above may be combined into one or more combinations of features of the implementations to provide further implementations.

Aspects and implementations of the present disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various aspects and implementations of the disclosure, which, however, should not be taken to limit the disclosure to the specific aspects or implementations, but are for explanation and understanding only.

Aspects and implementations of the present disclosure are directed to continuous depth-ordered image compositing. Cameras can capture a sequence of images for a video. A video frame is one of the many still images which compose a moving picture (hereinafter referred to as "video"). Video flicker refers to a result from large discontinuous changes in the colors of pixels in the images. In the process of rendering a two dimensional (2D) image of a scene out of a three dimensional (3D) scene description, each pixel for the 2D image may receive contributions (e.g., image fragments) from multiple objects in the scene. In order to produce a single color value for each pixel, these image fragments are generally combined together in a process that is usually referred to as "compositing". Generally, compositing involves compositing fragments in an order. Depending on which order the fragments are composited in, a different result for the pixel color may be obtained. Conventional compositing techniques may inaccurately cluster fragments together, and as the scene is animated, the clustered fragments may cause the fragment order to change abruptly. Such sudden changes may cause large discontinuous changes in the final color of the composited pixel, and produce spatially inconsistent results in the 2D images and temporally inconsistent results in videos. Aspects of the present disclosure can remove and/or prevent sudden changes in a fragment order to provide spatially and temporally continuous image compositing. Continuous and/or continuity hereinafter refers to an infinitesimal change in depth should produce only an infinitesimal change in the composited result.

Aspects of the present disclosure collect fragments for each pixel during the rendering of a 2D output image. Unlike conventional compositing solutions, aspects of the present disclosure assign a finite thickness to each fragment to convert the fragments into 3D fragments. Aspects of the present disclosure use the finite thicknesses of the 3D fragments to determine how to group fragments together. The grouping of the fragments based on the finite thicknesses of the 3D fragments prevents sudden changes in a fragment order. Aspects of the present disclosure can determine a color of a pixel based on the color contribution of the fragment grouping. Aspects of the present disclosure produce more accurate pixel colors to provide spatially and temporally continuous image compositing, which helps prevent video flickering.

<FIG> illustrates an example of system architecture <NUM> for continuous image compositing, which helps prevent undesired artifacts in 2D images and temporal flickering in videos, in accordance with one implementation of the disclosure. The system architecture <NUM> includes one or more cameras (e.g., cameras 105A-105P), one or more servers (e.g., server <NUM>), one or more data stores (e.g., data store <NUM>), one or more user devices (e.g., user device <NUM>) and one or more platforms (e.g., platform <NUM>), coupled via one or more networks (e.g., network <NUM>).

In one implementation, network <NUM> may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an <NUM> network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and/or a combination thereof. In one implementation, the network <NUM> may be a cloud.

The cameras 105A-105P can be, for example, ODS (omni-directional stereo) cameras and/or depth aware cameras. An ODS camera is a camera with a <NUM>-degree field of view in the horizontal plane, or with a visual field that covers (approximately) the entire sphere. A depth aware camera can create depth data for one or more objects that are captured in range of the depth aware camera. In one implementation, system architecture <NUM> includes one depth-aware camera. In another implementation, system architecture <NUM> includes multiple cameras, such as ODS cameras and/or depth-aware cameras.

The cameras can be setup in a camera array <NUM>. The cameras in the camera array <NUM> can share settings and frame-level synchronization to have the cameras act as one camera. For example, system architecture <NUM> may include <NUM> ODS cameras setup in a camera array <NUM>. The camera array <NUM> can be circular. The circular array <NUM> can include a stereoscopic virtual reality (VR) rig to house the cameras. The camera array <NUM> can be a multi-viewpoint camera system, which can link the motion of the cameras together to capture images of a scene from different angles. A scene can include one or more objects.

A camera can use a RGB (Red, Green, Blue) color space. In one implementation, a camera can produce output data, such as color data. The color data can include a RGB vector for each pixel in an image captured by the camera. In another implementation, a camera can produce output data that can be converted into color data (e.g., RGB vector). A camera using a RGB color space is used as an example throughout this document.

The cameras 105A-P can capture a scene that is in range of the cameras 105A-P to create content, for example, for a video stream. The content can be a sequence of RGB images of the scene. Each of the cameras 105A-P can output a video stream, which is composed of a sequence of RGB images captured by a respective camera. The RGB images in a video stream are made of pixels. The RGB images can be encoded, for example, as binary char arrays, with <NUM> bytes per pixel.

The video streams and color data can be stored in a data store (e.g., data store <NUM>). The data store <NUM> can be a persistent storage that is capable of storing data. A persistent storage can be a local storage unit or a remote storage unit. Persistent storage can be a magnetic storage unit, optical storage unit, solid state storage unit, electronic storage units (main memory), or similar storage unit. Persistent storage can be a monolithic device or a distributed set of devices. A 'set', as used herein, refers to any positive whole number of items.

One or more servers (e.g., server <NUM>) can process the video streams, which are generated by the cameras 105A-P to produce output images (e.g., output image <NUM>) to generate an output video stream <NUM>. The servers can be hosted by one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to provide content to users.

In one implementation, the output images are of a panorama scene (or "panorama"), and the output video stream can be an immersive video. In other implementations, the output images are of non-panoramic views. A panorama is a wide-angle view or representation of a physical space. Immersive videos, also known as "<NUM> videos", "<NUM> degree videos", or "spherical videos", are video streams of a real-world panorama, where the view in every direction is recorded at the same time, and shot using, for example, omni-directional stereo cameras (e.g., cameras 105A-105P), or a collection of cameras (e.g., camera array <NUM>). During playback, a user can control the viewing direction, which is a form of virtual reality. Virtual reality (VR) is a computer technology that replicates an environment, real or imagined, and can simulate a user's physical presence and environment in a way that allows the user to interact with it. Virtual realities can artificially create sensory experience, which can include sight, touch, hearing, and/or smell.

The server <NUM> can use computer vision, and 3D alignment for aligning the RGB images produced by the cameras 105A-P and stitching the RGB images into a seamless photo-mosaic to create the output video stream <NUM>. Computer vision refers to methods for acquiring, processing, analyzing, and understanding images and high-dimensional data from the real world in order to produce numerical or symbolic information. The output images (e.g., output image <NUM>) and the output video stream <NUM> can be stored in the data store <NUM>.

The server <NUM> can include a compositor <NUM> to determine colors of pixels for the output images (e.g., output image <NUM>) for the output video stream <NUM>. The compositor <NUM> applies a splatting technique to the input images produced by the cameras 105A-P to produce splats for the output images (e.g., output image <NUM>) to produce a RGBA vector that includes four channels (e.g., Red channel, Green channel, Blue channel, Alpha channel) for each pixel. "Alpha" is hereinafter also represented by "α. " Opacity information is represented by the alpha channel (Alpha). Opacity is the condition of lacking transparency or translucence (opaqueness). The combination of RGB images produced by the cameras 105A-P, along with the alpha channel information for the pixel, is hereinafter referred to as RGBA images. The RGBA images can be 2D images. The output images (e.g., output image <NUM>), which are generated from the RGBA images <NUM>, can be 2D images. The compositor <NUM> can analyze the scenes in the 2D RGBA images <NUM> in 3D. The server <NUM> can adapt the stitching based on the analysis of the compositor <NUM>.

To analyze the scenes in the 2D RGBA images <NUM> in 3D modeling, the compositor <NUM> identifies a pixel in the output image <NUM>, and identifies 2D fragments in the output image <NUM> that are associated with the pixel. The compositor <NUM> converts the 2D fragments into 3D fragments for the pixel, and create volume spans for the pixel based on the 3D fragments. The compositor <NUM> will, according to the invention, determine the colors of the volume spans based on the color contribution of the 3D fragments and determine a color of the pixel for the output image based on the colors of the volume spans. The analyzing of the scenes in the 2D RGBA images <NUM> in 3D modeling to determine more accurate colors of the pixels is described in greater detail below in conjunction with <FIG>. The compositor <NUM> determines the color of each pixel in an output image <NUM>. The compositor <NUM> determines the colors of the pixels in the output images that are used to generate the output video stream <NUM>.

The output video stream <NUM> can be a single video stream that can be accessed by one or more platforms (e.g., content sharing platform). The one or more platforms can provide the output video stream <NUM> to one or more user devices (e.g., VR headset, smart phone, etc.). For example, the output video stream <NUM> can be played back in a VR headset or any type of system supporting <NUM> views. The providing of the output video stream <NUM> by one or more platforms to one or more user devices is described in greater detail below in <FIG>.

<FIG> depicts a flow diagram of aspects of a method <NUM> for determining a color of a pixel using three-dimensional fragments, in accordance with one implementation of the present disclosure. The method <NUM> is performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by a compositor <NUM> of <FIG>, while in some other implementations one or more blocks of <FIG> may be performed by another machine.

At block <NUM>, the processing device creates an output image (e.g., output image <NUM> in <FIG>) of a scene using one or more two-dimensional (2D) images of the scene. The one or more 2D images can be 2D RGBA images (e.g., RGBA images <NUM> in <FIG>) that are generated by one or more cameras (e.g., cameras 105A-P in <FIG>). The processing device can use computer vision techniques and 3D alignment techniques for aligning the 2D RGBA images produced by the cameras to create the output image.

At block <NUM>, for a pixel in the output image, the processing device identifies, in the output image, 2D fragments that correspond to the pixel. The processing device can identify one to four fragments for a pixel. There may be a significantly large number of points with high resolution textures in the 2D RGBA images. According to the invention, the processing device uses a surface splatting point rendering technique to tolerate inherent loss in geometric accuracy and texture fidelity when processing a large volume of points. According to the invention, the processing device uses surface splatting to generate a splat in an output image, for each pixel, in the 2D RGBA images. A splat is a point in a pixel. The processing device can use a forward splatting algorithm to generate splats. In one implementation, the processing device applies the forward splatting algorithm to a pixel in an input image (e.g., image produced by a camera) and produces one splat in the output image. The processing device applies the splatting technique to each pixel in the input image to convert each input image pixel into a splat, which lands in a location in an output image. There can be multiple splats that land into each output pixel (i.e., pixel in an output image). The processing device uses the splat for a pixel in an output image to generate fragments for the pixel of the output image.

The processing device can compute the color for each pixel by blending the contributions, for example, of fragments of images that are drawn nearby and weighting the contribution of each fragment in accordance with how much of the pixel is covered by the fragment. This technique is referred to as "anti-aliasing", because it reduces the "aliasing" effect of under-sampling which results in abrupt changes in the images. An example scheme for implementing anti-aliased image generation is the A-buffer scheme. The A-buffer can be used to produce an open-ended list of all the contributions of fragments to a pixel. A list of fragments is maintained for each drawn object that overlaps the pixel. The processing device can apply an A-buffer algorithm to the splat to create fragments for the pixel.

<FIG> depicts an example of identifying 2D fragments for a pixel, in accordance with one implementation of the present disclosure. An output image <NUM> can include a set of pixels. The pixels can be square. The processing device can generate one or more splats that land in each pixel in the output image. For example, a splat <NUM> is in pixel <NUM>.

The processing device can apply an A-buffer to the set of pixels in image <NUM> to produce a set of fragments (e.g., fragments <NUM>-<NUM>). A fragment is a polygon clipped to a pixel boundary. For example, pixel <NUM> has four fragments <NUM>-<NUM>, which are each a polygon. The A-buffer can process polygons in scan-line order by clipping them to each square pixel they cover to output a list of fragments corresponding to each square pixel. The A-buffer can slide a grid <NUM> across the pixels in the image <NUM> to define the fragments (e.g., fragments <NUM>-<NUM>) for a pixel (e.g., pixel <NUM>) in reference to a splat (e.g. splat <NUM>) for the pixel. The processing device can generate and store color data (e.g., RGBA vector) for each fragment.

Referring back to <FIG>, at block <NUM>, the processing device converts the 2D fragments into three-dimensional (3D) fragments. A 2D fragment has an x-y footprint using an x-axis and a y-axis. For each 2D fragment, the processing device adds a pre-defined thickness to a front side and a back side of the 2D fragment along an axis of a third dimension. The third dimension can be in a disparity ("d") space or a depth ("z") space. Disparity d is the flow vector length in place of depth, and is inversely proportional to depth z, such that disparity d = <NUM>/z. Disparity d is used as an example of a third dimension throughout this document.

<FIG> depicts an example of converting a 2D fragment into a 3D fragment, in accordance with one implementation of the present disclosure. The processing device identifies one or more 2D fragments for the pixel. For simplicity of explanation, one 2D fragment <NUM> for the pixel is depicted in <FIG>. The processing device adds the pre-defined thickness ("n") <NUM> to a front side of the 2D fragment <NUM> and to the back side of the back side of the 2D fragment <NUM> along a d-axis (e.g., disparity axis) to convert the 2D fragment <NUM> into a 3D fragment <NUM>. In another implementation, the pre-defined thickness can be defined in a depth space along a z-axis (e.g., depth axis). The pre-defined thickness n <NUM> can be configurable and/or user-defined.

Referring to <FIG>, at block <NUM>, the processing device creates volume spans for the pixel based on the 3D fragments. The processing device can order the 3D fragments along a third dimensional axis (e.g., d-axis). The processing device can position the 3D fragments along the d-axis according to the disparity of the 3D fragments.

<FIG> depicts an example of creating volume spans for a pixel, in accordance with one implementation of the present disclosure. In the example in <FIG>, the d-axis 500A-C represents disparity relative to a camera origin <NUM>. There are four 2D fragments (e.g., fragments <NUM>,<NUM>,<NUM>,<NUM>) for the pixel. The d-axis 500A represents a third dimension in a disparity space. Disparity is inversely proportional to depth, and can be relative to a camera's original <NUM>. The d-axes 500A-500C represent disparity d = <NUM>/z and can have a reference point representing the camera origin <NUM>. The d-axis 500A illustrates a cross-sectional view of four 2D fragments <NUM>,<NUM>,<NUM>,<NUM>, each having an x-y along an x-axis (e.g., x-axis in <FIG>) and a y-axis (e.g., y-axis in <FIG>). For simplicity of explanation, the x-axis and y-axis are not depicted in <FIG>.

The d-axis 500B illustrates a cross-sectional view of four 3D fragments <NUM>,<NUM>,<NUM>,<NUM>. The processing device can add a pre-defined finite thickness n to both sides of each of the 2D fragments <NUM>,<NUM>,<NUM>,<NUM> in d-axis 500A to convert the 2D fragments <NUM>,<NUM>,<NUM>,<NUM> into the 3D fragments <NUM>,<NUM>,<NUM>,<NUM> shown in d-axis 500B.

The processing device can identify and mark a start event and an end event for each of the 3D fragments <NUM>,<NUM>,<NUM>,<NUM> in d-axis 500B. A start event is a start of a 3D fragment and can be represented by the metric (e.g., disparity) of the d-axis 500B. An end event is an end of the 3D fragment and can be represented by the metric (e.g., disparity) of the d-axis 500B. For example, the processing device marks a start event d<NUM> and an end event d<NUM> for 3D fragment <NUM>, a start event d<NUM> and an end event d<NUM> for 3D fragment <NUM>, a start event d<NUM> and an end event d<NUM> for 3D fragment <NUM>, and a start event d<NUM> and an end event d<NUM> for 3D fragment <NUM>. The event markings can be stored in a data store (e.g., data store <NUM> in <FIG>).

The processing device can create volume spans for the pixel based on the start events and end events of the 3D fragments <NUM>,<NUM>,<NUM>,<NUM>. The processing device can define volume spans by, for example, ordering the event markings sequentially based on the values for disparity d for the events. The processing device can use the order of event markings to define start events and end events for volume spans for the pixel. A volume span is a 3D fragment for which one of the event markings (e.g., dv) is selected as its start event, and the next event marking (e.g., dv+<NUM>), in the sequential order of event markings is selected as its end event. For example, the processing device can create a volume span <NUM> by defining the volume span <NUM> as having a start event of d<NUM> and an end event as d<NUM>. In another example, the processing device can create another volume span <NUM> by defining the volume span <NUM> as having a start event of d<NUM> and an end event as d<NUM>. In other examples, the processing device can use the event markings d<NUM>, d<NUM>, d<NUM>, d<NUM>, d<NUM>, d<NUM>, d<NUM>, and d<NUM> to define and create volume spans <NUM>-<NUM> for the pixel.

Referring to <FIG>, at block <NUM>, the processing device determines the colors of the volume spans (e.g., volume spans <NUM>-<NUM> in <FIG>) based on color contribution of respective 3D fragments that make up the volume span. To determine the color contribution, the processing device identifies which 3D fragments have a portion that overlaps the volume span, and thus contribute color to the volume span. For example, in <FIG>, for volume span <NUM>, 3D fragment <NUM> has a portion <NUM> that overlaps volume span <NUM>. The color of portion <NUM> contributes to the color of the volume span <NUM>. In another example, for volume span <NUM>, the processing device may determine that 3D fragment <NUM> has a portion <NUM> that overlaps volume span <NUM>, and that 3D fragment <NUM> has a portion <NUM> that overlaps volume span <NUM>. The color of portion <NUM> and the color of portion <NUM> contribute to the color of the volume span <NUM>.

The processing device can determine a color contribution for the identified 3D fragments. The color contribution of the 3D fragments for a particular volume span can be determined as the length of the volume span divided by the total thickness k of the 3D fragments. The total thickness k is the same for the individual 3D fragments. The processing device can determine the color contribution of the identified 3D fragments as: <MAT>.

In Equation <NUM>, parameter i refers to the volume span. The length of the volume span is (di+<NUM> - di). Parameter k refers to the total thickness of the 3D fragment. The color contribution can be presented as a value or a percentage. For example, for volume span <NUM>, a single 3D fragment (e.g., 3D fragment <NUM>) has at least a portion that overlaps the volume span <NUM>. In this case, the entirety of 3D fragment <NUM> overlaps the volume span <NUM>. The length <NUM> of the volume span <NUM> divided by the total thickness <NUM> of 3D fragment <NUM> may be <NUM> or <NUM>%. The contribution of the 3D fragment <NUM> to the color of the volume span <NUM> is <NUM>%.

In another example, for volume span <NUM>, a single 3D fragment (e.g., 3D fragment <NUM>) has at least a portion that overlaps the volume span <NUM>. In this case, the length <NUM> of the volume span <NUM> divided by the total thickness <NUM> of 3D fragment <NUM> may be <NUM> or <NUM>%. The contribution of the 3D fragment <NUM> to color of the volume span <NUM> is <NUM>%.

In another example, for volume span <NUM>, two 3D fragments (e.g., 3D fragment <NUM>, 3D fragment <NUM>) have at least a portion that overlaps the volume span <NUM>. In this case, the length <NUM> of the volume span <NUM> divided by the total thickness (e.g., total thickness <NUM> or total thickness <NUM>, which are the same) of the 3D fragments (e.g., 3D fragment <NUM>, 3D fragment <NUM>) may be <NUM> or <NUM>%. The contribution of the 3D fragments (e.g., 3D fragment <NUM>, 3D fragment <NUM>) to the color of the volume span <NUM> is <NUM>%.

The processing device can determine the color of the volume span from the color contributions for the identified 3D fragments that have portions that overlap the volume span. The processing device can determine the colors of the identified 3D fragments, and determine the sum of the colors. The processing device can multiply the sum of the colors of the identified 3D fragments with the contribution of the identified 3D fragments. For example, the contribution of the 3D fragments (e.g., 3D fragment <NUM>, 3D fragment <NUM>) to the color of the volume span <NUM> is <NUM>%. The processing device can multiply the sum of the colors of the identified 3D fragments by <NUM>%, as described in detail below in conjunction with Equation <NUM>.

The processing device can use the contribution (e.g., [(di+<NUM> - di) / k]) to determine the color (c') of a volume span (e.g., volume span <NUM>) as: <MAT>.

In Equation <NUM>, parameter i refers to the volume span (e.g., volume span <NUM>). Parameter j refers to a 3D fragment (e.g., 3D fragment <NUM>, 3D fragment <NUM>) that includes a portion (e.g., portion <NUM>, portion <NUM>) that overlaps volume span i (e.g., volume span <NUM>). Parameter k refers to the total thickness (e.g., total thickness <NUM>, total thickness <NUM>) of 3D fragment j (e.g., 3D fragment <NUM>, 3D fragment <NUM>). θi refers to the group of portions (e.g., portion <NUM>, portion <NUM>) that overlap with volume span i.

In Equation <NUM>, fj refers to the RGBA color of the 3D fragment j. A color can be represented by an RGBA color value. An RGBA color value can be specified as three color channels, such as, RGB(Red, Green, Blue), along with opacity information, which is represented by an alpha channel (Alpha). Opacity is the condition of lacking transparency or translucence (opaqueness). The alpha channel specifies the opacity of the object being represented by the 3D fragment j.

An RGBA color value can be specified as a vector, such as RGBA(red, green, blue, alpha). Each color parameter (e.g., red, green, and blue) in the color channels of the vector defines the intensity of the color and can be an integer between <NUM> and <NUM> or a percentage value (from <NUM>% to <NUM>%). For example, for a RGB color value, the RGB(<NUM>,<NUM>,<NUM>) value or RGB(<NUM>%,<NUM>%,<NUM>%) can be rendered as blue, because the blue parameter is set to its highest value (<NUM> or <NUM>%), and the other parameters values are set to <NUM> or <NUM>%.

The alpha parameter, represented as A or α, in the alpha channel in the vector can be a number, for example, between <NUM> (fully transparent) and <NUM> (fully opaque). In another example, the alpha parameter, represented as A or α, can be represented using an <NUM>-bit color scheme where <NUM> is fully opaque and <NUM> is fully transparent. The processing device can determine the RGBA color value of the 3D fragment j from output data (e.g., color data) that is produced by a camera (e.g., cameras 105A-P in <FIG>). The processing device can access the color data from a data store (e.g., data store <NUM> in <FIG>).

The quadruple (R,G,B,A) for a pixel can express that the pixel is covered by a full colored object by a percentage. For example, the RGBA vector (<NUM>,<NUM>,<NUM>,. <NUM>) indicates that a full red object half covers the pixel. The quadruple (R,G,B,A) indicates that the pixel is A covered by the color (R/A,G/A,B/A).

In Equation <NUM> above, M is a weight multiplier that can be configurable and/or user-defined. In one implementation, M is set to <NUM>. Mfj refers to the weighted RGBA color of the 3D fragment j. In one implementation, M is applied only to the A-channel in a RGBA vector to determine the alpha value.

The processing device can use Equation <NUM> to determine a color (c') of a volume span. The color (c') of the volume span can be represented by a RGBA vector. In one implementation, if the resulting alpha value for the volume span in the RGBA vector exceeds the value of <NUM>, or <NUM> in an <NUM>-bit color scheme, the processing device can normalize the RGBA vector of the volume span.

The processing device can examine the value of the alpha α in the A-channel in the RGBA color c'i vector for the volume span and determine whether the α value exceeds <NUM>, or <NUM> in an <NUM>-bit color scheme. If the α value exceeds the threshold value, then the processing device can normalize the α value by dividing all of the channels in the RGBA vector for the volume span by the α value.

Referring to <FIG>, at block <NUM>, the processing device determines a color of the pixel for the output image from the colors of the volume spans. The processing device can apply Equation <NUM> to each volume span. For example, in <FIG>, the processing device can determine the color (c<NUM>) for volume span <NUM>, the color (c<NUM>) for volume span <NUM>, the color (c2) for volume span <NUM>, the color (c<NUM>) for volume span <NUM>, the color (c<NUM>) for volume span <NUM>, and the color (c<NUM>) for volume span <NUM>.

The processing device can determine a composite color for the pixel by performing an over operation on the colors of the volume spans as follows: <MAT>.

An over operation is defined as follows: <MAT>.

Where a and b are rgba-vectors, and a_alpha is the fourth component of a, i.e. the alpha value. The processing device can store the color of the pixel for the output image (e.g., output image <NUM> in <FIG>) in a data store (e.g., data store <NUM> in <FIG>). The processing device can repeat blocks <NUM> through <NUM> for other pixels in the output image (e.g., output image <NUM> in <FIG>). The pixel colors determined by the processing device are more accurate than traditional compositing solutions, and help prevent sudden pixel color changes in the images that are used to create an output video stream (e.g., output video stream <NUM> in <FIG>). The pixel colors determined by the processing device provide spatially and temporally continuous compositing to reduce spatial artifacts in 2D images and video flickering in the output video stream.

<FIG>illustrates an example of system architecture <NUM> for improved pixel color for spatially and temporally continuous image compositing, in accordance with one implementation of the disclosure. The system architecture <NUM> includes user devices 610A through 610Z, one or more networks <NUM>, one or more data stores <NUM>, one or more servers <NUM>, and one or more platforms (e.g., content sharing platform <NUM>, recommendation platform <NUM>, advertisement platform <NUM>, mobile platform <NUM>, social network platform <NUM>, search platform <NUM>, content provider platform <NUM>, and collaboration platform <NUM>). The user devices 610A through 610Z can be client devices.

The one or more networks <NUM> can include one or more public networks (e.g., the Internet), one or more private networks (e.g., a local area network (LAN) or one or more wide area networks (WAN)), one or more wired networks (e.g., Ethernet network), one or more wireless networks (e.g., an <NUM> network or a Wi-Fi network), one or more cellular networks (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and/or a combination thereof. In one implementation, some components of architecture <NUM> are not directly connected to each other. In one implementation, architecture <NUM> includes separate networks <NUM>.

The one or more data stores <NUM> can be memory (e.g., random access memory), cache, drives (e.g., hard drive), flash drives, database systems, or another type of component or device capable of storing data. The one or more data stores <NUM> can include multiple storage components (e.g., multiple drives or multiple databases) that may also span multiple computing devices (e.g., multiple server computers). The data stores <NUM> can be persistent storage that are capable of storing data. A persistent storage can be a local storage unit or a remote storage unit. Persistent storage can be a magnetic storage unit, optical storage unit, solid state storage unit, electronic storage units (main memory), or similar storage unit. Persistent storage can be a monolithic device or a distributed set of devices. A 'set', as used herein, refers to any positive whole number of items.

Content items <NUM> can be stored in one or more data stores <NUM>. The data stores <NUM> can be part of one or more platforms. Examples of a content item <NUM> can include, and are not limited to, output video streams (e.g., output video stream <NUM> in <FIG>), digital video, digital movies, animated images, digital photos, digital music, digital audio, website content, social media updates, electronic books (ebooks), electronic magazines, digital newspapers, digital audio books, electronic journals, web blogs, real simple syndication (RSS) feeds, electronic comic books, software applications, etc. Content item <NUM> is also referred to as a media item. For brevity and simplicity, an output video stream (also hereinafter referred to as a video) (e.g., output video stream <NUM> in <FIG>) is used as an example of a content item <NUM> throughout this document.

The content items <NUM> can be provided by server <NUM> for storage in one or more data stores <NUM>. The contents items <NUM> can be immersive videos, also known as "<NUM> videos", "<NUM> degree videos", or "spherical videos", are video streams of a real-world panorama, where the view in every direction is recorded at the same time. During playback, a user can control the viewing direction. The server <NUM> can include a compositor <NUM> for providing output video streams (e.g., immersive videos) to users via communication applications <NUM>. The server <NUM> can be one or more computing devices (e.g., a rackmount server, a server computer, etc.). In one implementation, the server <NUM> is included in one or more of the platforms. In another implementation, the server <NUM> is separate from the platforms, but may communicate (e.g., exchange data) with the one or more platforms.

The content items <NUM> can be provided by content providers for storage in one or more data stores <NUM>. In one implementation, a content provider provides RGBA images (e.g., RGBA image <NUM> in <FIG>) to a server <NUM> (e.g., server <NUM> in <FIG>), and the server <NUM> processes the RGBA images to produce output images (e.g., output image <NUM>) and uses the output images to to create the content item <NUM> (e.g., output video stream <NUM> in <FIG>). A content provider can be a user, a company, an organization, etc..

A service provider (e.g., content sharing platform <NUM>, recommendation platform <NUM>, advertisement platform <NUM>, mobile platform <NUM>, social network platform <NUM>, search platform <NUM>, content provider platform <NUM>, or collaboration platform <NUM>) can provide the immersive videos on user devices 610A-610Z to be viewed by users.

The user devices 610A-610Z can include devices, such as virtual reality headsets. smart phones, cellular telephones, personal digital assistants (PDAs), portable media players, netbooks, laptop computers, electronic book readers, tablet computers, desktop computers, set-top boxes, gaming consoles, televisions, or any type of device supporting <NUM> view.

The individual user devices 610A-610Z can include a communication application <NUM>. A content item <NUM> can be consumed via a communication application <NUM>, the Internet, etc. As used herein, "media," "media item," "online media item," "digital media," "digital media item," "content," and "content item" can include an electronic file that can be executed or loaded using software, firmware or hardware configured to present a content item. In one implementation, the communication applications <NUM> may be applications that allow users to compose, send, and receive content items <NUM> (e.g., immersive videos) over a platform (e.g., content sharing platform <NUM>, recommendation platform <NUM>, advertisement platform <NUM>, mobile platform <NUM>, social network platform <NUM>, search platform <NUM>, collaboration platform <NUM>, and content provider platform <NUM>) and/or a combination of platforms and/or networks.

For example, the communication application <NUM> may be a social networking application, video sharing application, photo sharing application, chat application, mobile application of a content provider or any combination of such applications. The communication application <NUM> in a user device can render, display, and/or present one or more content items <NUM> (e.g., immersive videos) to one or more users. For example, the communication application <NUM> can provide one or more user interfaces (e.g., graphical user interfaces) to be rendered in a display of a user device for sending, receiving and/or playing immersive videos.

In one implementation, the individual user devices 610A-610Z includes a content viewer <NUM> (e.g., media player) to render, display, and/or present content items <NUM> (e.g., immersive videos) to one or more users. In one implementation, a content viewer <NUM> is embedded in an application (e.g., communication application <NUM>). For example, for mobile devices, the communication application <NUM> can be a mobile application that can be downloaded from a platform (e.g., content sharing platform <NUM>, social network platform <NUM>, content provider platform <NUM>, etc.) and can include a content viewer <NUM> (e.g., media player). In another example, the communication application <NUM> can be a desktop application, such as a web browser that can access, retrieve, present, and/or navigate content (e.g., web pages such as Hyper Text Markup Language (HTML) pages, digital media items, etc.) served by a web server of a platform. The content viewer <NUM> can be a web browser plugin or a separate application. In one implementation, the content viewer <NUM> is embedded in a web page. For example, the content viewer <NUM> may be an embedded media player (e.g., a Flash® player or an HTML5 player) that is embedded in a document (e.g., a web page).

The content provider platform <NUM> can provide a service and the content provider can be the service provider. For example, a content provider may be a streaming service provider that provides a media streaming service via a communication application <NUM> for users to play TV shows, clips, and movies, on user devices 610A-210Z via the content provider platform <NUM>. The content provider platform <NUM> can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to provide content to users.

The social network platform <NUM> can provide an online social networking service. The social network platform <NUM> can provide a communication application <NUM> for users to create profiles and perform activity with their profile. Activity can include updating a profiling, exchanging messages with other users, posting status updates, photos, videos, etc. to share with other users, evaluating (e.g., like, comment, share, recommend) status updates, photos, videos, etc., and receiving notifications of other users activity. The social network platform <NUM> can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to provide communication between users.

The mobile platform <NUM> can be and/or include one or more computing devices (e.g., servers), data stores, networks (e.g., phone network, cellular network, local area network, the Internet, and/or a combination of networks), software components, and/or hardware components that can be used to allow users to connect to, share information, and/or interact with each other using one or more mobile devices (e.g., phones, tablet computers, laptop computers, wearable computing devices, etc.) and/or any other suitable device. For example, the mobile platform <NUM> may enable telephony communication, Short Message Service (SMS) messaging, Multimedia Message Service (MMS) messaging, text chat, and/or any other communication between users. The mobile platform <NUM> can support user communications via video messaging, video chat, and/or videoconferences.

The collaboration platform <NUM> can enable collaboration services, such as video chat, video messaging, and audio and/or videoconferences (e.g., among the users of devices 610A-610Z) using, for example, streaming video or voice over IP (VoIP) technologies, cellular technologies, LAN and/or WAN technologies, and may be used for personal, entertainment, business, educational or academically oriented interactions. The collaboration platform <NUM> can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to provide communication between users.

The recommendation platform <NUM> can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to generate and provide content recommendations (e.g., articles, videos, posts, news, games, etc.).

The search platform <NUM> can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to allow users to query the one or more data stores <NUM> and/or one or more platforms and receive query results.

The advertisement platform <NUM> can provide video advertisements. The advertisement platform <NUM> can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to provide the video advertisements. The content items <NUM> (e.g., immersive videos) can be used as video advertisements.

The content sharing platform <NUM> can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that can be used to provide one or more users with access to content items <NUM> and/or provide the content items <NUM> to one or more users. For example, the content sharing platform <NUM> may allow users to consume, upload, download, and/or search for content items <NUM>. In another example, the content sharing platform <NUM> may allow users to evaluate content items <NUM>, such as approve of ("like"), dislike, recommend, share, rate, and/or comment on content items <NUM>. In another example, the content sharing platform <NUM> may allow users to edit content items <NUM>. The content sharing platform <NUM> can also include a website (e.g., one or more webpages) and/or one or more applications (e.g., communication applications <NUM>) that may be used to provide one or more users with access to the content items <NUM>, for example, via user devices 610A-610Z. Content sharing platform <NUM> can include any type of content delivery network providing access to content items <NUM>. The content sharing platform <NUM> can include a content feed component <NUM> to present a content feed <NUM> that lists feed items, such as content items <NUM> in the user interface of the communication application <NUM>.

The content sharing platform <NUM> can include multiple channels (e.g., Channel A <NUM>). A channel can be data content available from a common source or data content having a common topic or theme. The data content can be digital content chosen by a user, digital content made available by a user, digital content uploaded by a user, digital content chosen by a content provider, digital content chosen by a broadcaster, etc. For example, Channel A <NUM> may include immersive videos Y and Z. A channel can be associated with an owner, who is a user that can perform actions on the channel. The data content can be one or more content items <NUM>.

Different activities can be associated with the channel based on the channel owner's actions, such as the channel owner making digital content available on the channel, the channel owner selecting (e.g., liking) digital content associated with another channel, the channel owner commenting on digital content associated with another channel, etc. Users, other than the owner of the channel, can subscribe to one or more channels in which they are interested. Once a user subscribes to a channel, the user can be presented with information from the channel's activity feed. Although channels are described as one implementation of a content sharing platform, implementations of the disclosure are not limited to content sharing platforms that provide content items <NUM> via a channel model.

For simplicity of explanation, the methods of this disclosure are depicted and described as a series of acts. However, acts in accordance with this disclosure can occur in various orders and/or concurrently, and with other acts not presented and described herein. Furthermore, not all illustrated acts may be required to implement the methods in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the methods could alternatively be represented as a series of interrelated states via a state diagram or events. Additionally, it should be appreciated that the methods disclosed in this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methods to computing devices. The term "article of manufacture," as used herein, is intended to encompass a computer program accessible from any computer-readable device or storage media.

<FIG> illustrates a diagram of a machine in an example form of a computer system <NUM> within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, can be executed, in accordance with one implementation of the present disclosure. The computer system <NUM> can host server <NUM> in <FIG>. In alternative implementations, the machine can be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. The machine can operate in the capacity of a server or a client machine in client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system <NUM> includes a processing device (processor) <NUM>, a main memory <NUM> (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), double data rate (DDR SDRAM), or DRAM (RDRAM), etc.), a static memory <NUM> (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device <NUM>, which communicate with each other via a bus <NUM>.

Processor (processing device) <NUM> represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processor <NUM> can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processor <NUM> can also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processor <NUM> is configured to execute instructions <NUM> for performing the operations and steps discussed herein.

The computer system <NUM> can further include a network interface device <NUM>. The computer system <NUM> also can include a video display unit <NUM> (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device <NUM> (e.g., a keyboard, and alphanumeric keyboard, a motion sensing input device, touch screen), a cursor control device <NUM> (e.g., a mouse), and a signal generation device <NUM> (e.g., a speaker). The computer system <NUM> also can include a camera <NUM> to record images that can be stored directly, transmitted to another location, or both. These images can be still photographs or moving images such as videos or movies. The camera <NUM> can be a depth aware camera that can capture RGB images along with per-pixel depth information.

The data storage device <NUM> can include a non-transitory computer-readable storage medium <NUM> on which is stored one or more sets of instructions <NUM> (e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions <NUM> can also reside, completely or at least partially, within the main memory <NUM> and/or within the processor <NUM> during execution thereof by the computer system <NUM>, the main memory <NUM> and the processor <NUM> also constituting computer-readable storage media. The instructions <NUM> can further be transmitted or received over a network <NUM> via the network interface device <NUM>.

In one implementation, the instructions <NUM> include instructions for a compositor (e.g., compositor <NUM> in <FIG>) and/or a software library containing methods that call the compositor. While the computer-readable storage medium <NUM> (machine-readable storage medium) is shown in an exemplary implementation to be a single medium, the term "computer-readable storage medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "computer-readable storage medium" shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term "computer-readable storage medium" shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

In the foregoing description, numerous details are set forth. It will be apparent, however, to one of ordinary skill in the art having the benefit of this disclosure, that the present disclosure can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present disclosure.

Some portions of the detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as "receiving", "rendering", "determining", "selecting", or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

For simplicity of explanation, the methods are depicted and described herein as a series of acts. However, acts in accordance with this disclosure can occur in various orders and/or concurrently, and with other acts not presented and described herein. Furthermore, not all illustrated acts can be required to implement the methods in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the methods could alternatively be represented as a series of interrelated states via a state diagram or events. Additionally, it should be appreciated that the methods disclosed in this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methods to computing devices. The term article of manufacture, as used herein, is intended to encompass a computer program accessible from any computer-readable device or storage media.

Certain implementations of the present disclosure also relate to an apparatus for performing the operations herein. This apparatus can be constructed for the intended purposes, or it can comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions.

Reference throughout this specification to "one implementation" or "an implementation" means that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation. Thus, the appearances of the phrase "in one implementation" or "in an implementation" in various places throughout this specification are not necessarily all referring to the same implementation. " It will, however, be appreciated that features described in the context of one implementation can be combined with features described in the context of other implementations. Moreover, the words "example" or "exemplary" are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words "example" or "exemplary" is intended to present concepts in a concrete fashion.

It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

In situations in which the systems discussed here collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), or to control whether and/or how to receive content from the content server that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and used by a content server.

Claim 1:
A method comprising:
creating (<NUM>) an output image (<NUM>) of a scene for an output video stream, by aligning a plurality of two-dimensional (2D) images of the scene, the plurality of two-dimensional (2D) images captured by a plurality of cameras (105A, 105B, ..., 105P), and using surface splatting to generate a splat in the output image, for each pixel in the plurality of two-dimensional (2D) images;
for a pixel (<NUM>) in the output image (<NUM>), identifying (<NUM>), in the output image (<NUM>) and using the splat for the pixel in the output image, a plurality of 2D fragments (<NUM>-<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) corresponding to the pixel (<NUM>), wherein each 2D fragment (<NUM>-<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) contributes a color to a color for the pixel of the output image for the output video stream and is associated with a respective location on an axis of a third dimension (d);
converting (<NUM>), by a processing device, the identified plurality of 2D fragments (<NUM>-<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) into a plurality of three dimensional (3D) fragments (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), comprising adding a pre-defined thickness (n, n) along the axis (d) of the third dimension to each 2D fragment (<NUM>-<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) at the respective location on the axis (d) of the third dimension;
creating (<NUM>), by the processing device, a plurality of volume spans (<NUM>, <NUM> ... <NUM>) for the pixel (<NUM>), based on start events (do, d<NUM>, d<NUM>, d<NUM>) and end events (d<NUM>, d<NUM>, d<NUM>, d<NUM>) of the plurality of 3D fragments (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) for the pixel (<NUM>), the start events (do, d<NUM>, d<NUM>, d<NUM>) and the end events (d<NUM>, d<NUM>, d<NUM>, d<NUM>) on the axis (d) of the third dimension;
determining (<NUM>) a color of each volume span (<NUM>, <NUM> ... <NUM>) of the plurality of volume spans (<NUM>, <NUM> ... <NUM>) based on a color contribution of respective one or more of the plurality of 3D fragments (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) that make up the respective volume span (<NUM>, <NUM> ... <NUM>); and
determining (<NUM>) a color of the pixel (<NUM>) for the output image (<NUM>) from determined colors of the plurality of volume spans (<NUM>, <NUM> ... <NUM>) of the pixel (<NUM>).