Patent Description:
Three hundred sixty degree (<NUM>°) video is emerging as a new way of experiencing immersive video due to the ready availability of powerful handheld devices such as smartphones. <NUM>° video enables immersive "real life," "being there" experience for consumers by capturing the <NUM>° view of the world. Users can interactively change their viewpoint and dynamically view any part of the captured scene or object they desire. Display and navigation sensors track head movement in real-time to determine the region of the <NUM>° video that the user wants to view. <NUM>° video provides a three Degrees of Freedom (3DoF) immersive experience. (6DoF) is the next level of immersive experience where in the user can turn his head as well as move around in a virtual/augmented environment. Multimedia data that is <NUM>-dimmentional in nature, such as point clouds, is needed to provide 6DoF experience.

Point clouds and meshes are a set of three-dimensional (<NUM>-D) points that represent a model of a surface of an object or a scene. Point clouds are common in a variety of applications such as gaming, <NUM>-D maps, visualizations, medical applications, augmented reality (AR), virtual reality (VR), autonomous driving, multi-view replay, 6DoF immersive media, to name a few. Point clouds, if uncompressed, generally require a large amount of bandwidth for transmission. Hence, the bitrate requirements are higher, necessitating the need for compression prior to transmission of a point cloud. Compression hardware and processes of point clouds are different than traditional compression hardware and processes for traditional two-dimensional (<NUM>-D) multimedia.

In prior art document "<NPL>, a point cloud is compressed using projection onto patches that are used to create occupancy maps, geometry frames ("height maps") and attribute frames.

In <CIT>, a point cloud is compressed by applying a chain coding scheme on cross sections perpendicular to one axis.

This disclosure provides point cloud and mesh compression using image/video codecs.

In a first embodiment, a decoding device for point cloud decoding is provided, as defined by the appended claims.

In another embodiment an encoding device for point cloud encoding is provided, as defined by the appended claims.

The phrase "computer readable medium" includes any type of medium capable of being accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.

Various embodiments of the present disclosure provide a image processing scheme that is more effective.

Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably-arranged system or device.

Virtual reality (VR) is a rendered version of a visual and audio scene. The rendering is designed to mimic the visual and audio sensory stimuli of the real world as naturally as possible to an observer or user as they move within the limits defined by the application. For example, VR places a user into immersive worlds that interact with their head movements. At the video level, VR is achieved by providing a video experience that covers as much of the field of view (FOV) as possible together with the synchronization of the viewing angle of the rendered video with the head movements. Although many different types of devices are able to provide such an experience, head-mounted displays are the most popular. Typically, head-mounted displays rely either on dedicated screens integrated into the device and running with external computers (tethered) or on a smartphone inserted into the HMD (untethered). The first approach utilizes lightweight screens and benefiting from a high computing capacity. In contrast, the smartphone-based systems utilizes higher mobility and can be less expensive to produce. In both instances, the video experience is generated the same.

A point cloud is a <NUM>-D representation of an object that is similar to VR. Similarly, a point mesh is a <NUM>-D representation of an object that is similar to VR. Generally, a point cloud is a collection of data points defined by a coordinate system. For example, in a <NUM>-D Cartesian coordinate system, each point of a point cloud is identified by three coordinates, that of X, Y, and Z. When each point is identified by the three coordinates, a precise location in <NUM>-D space is identified, relative to an origin point where the X, Y, and Z axes intersect. The points of a point cloud often represent the external surface of the object. Each point of a point cloud is defined by three coordinates and some attributes such as color, texture coordinates, intensity, normal, reflectance, and the like.

Similarly, a <NUM>-D mesh is a <NUM>-D representation of an object that is similar to a point cloud as well as VR. A <NUM>-D mesh illustrates the external structure of an object that is built out of polygons. For example, a <NUM>-D mesh is a collection of verities, edges, and faces that define the shape of an object. For another example, a mesh (or a point cloud) can be rendered on spherical coordinate system and where each point is displayed throughout a sphere. In certain embodiments, each point can be located in the X, Y, Z coordinates within the sphere and texture coordinates U and V indicate a location of texture of the image. When the point cloud is rendered, the vertices of the mesh, the corresponding texture coordinate, and the texture image are inputted into a graphical processing unit which maps the mesh onto the <NUM>-D geometry. The user can be placed at the center of the virtual sphere and sees a portion of the <NUM>° scene corresponding to the viewport. In certain embodiments, alternative shapes can be used instead of a sphere such as a cube, an icosahedron, an octahedron, and the like. Point clouds and <NUM>-D meshes are illustrated and discussed in greater detail below with reference to <FIG>.

Point clouds and meshes are commonly used in a variety of applications, including gaming, <NUM>-D mapping, visualization, medicine, augmented reality, VR, autonomous driving, multiview replay, <NUM> degrees of freedom immersive media, to name a few. As used hereinafter, the term 'point cloud' also refers to a '<NUM>-D point cloud,' and a '<NUM>-D mesh.

Transmitting a point cloud, from one electronic device to another, often requires significant bandwidth due to the size and complexity of the data associated with a single point cloud. The transmission of a point cloud often requires specific compression techniques to reduce the size of the data prior to transmission. For example, compressing a point cloud can require dedicated hardware or specific compression algorithms or a combination thereof. Compression algorithms for a point cloud are different than compression algorithms of other multimedia forms, such as images and video, VR, and the like.

According to embodiments of the present disclosure, architecture for carrying out a point cloud compression using a video codec is provided. According to embodiments of the present disclosure, architecture for carrying out a point cloud compression using an image codec is provided. According to embodiments of the present disclosure, a point cloud is deconstructed, and multiple <NUM>-D frames are generated that represent the geometry of each point of the point cloud, as well as various attributes of the point cloud. For example, the point cloud can be deconstructed and mapped onto a <NUM>-D frame. The <NUM>-D frame can be compressed using various video or image or both compression.

<FIG> illustrates an example computing system <NUM> according to this disclosure. The embodiment of the system <NUM> shown in <FIG> is for illustration only. Other embodiments of the system <NUM> can be used without departing from the scope of this disclosure.

The system <NUM> includes network <NUM> that facilitates communication between various components in the system <NUM>. For example, network <NUM> can communicate Internet Protocol (IP) packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, or other information between network addresses. The network <NUM> includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems at one or more locations.

The network <NUM> facilitates communications between a server <NUM> and various client devices <NUM>-<NUM>. The client devices <NUM>-<NUM> may be, for example, a smartphone, a tablet computer, a laptop, a personal computer, a wearable device, or a head-mounted display (HMD). The server <NUM> can represent one or more servers. Each server <NUM> includes any suitable computing or processing device that can provide computing services for one or more client devices. Each server <NUM> could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network <NUM>. As described in more detail below, the server <NUM> transmits a point cloud to one or more users.

Each client device <NUM>-<NUM> represents any suitable computing or processing device that interacts with at least one server or other computing device(s) over the network <NUM>. In this example, the client devices <NUM>-<NUM> include a desktop computer <NUM>, a mobile telephone or mobile device <NUM> (such as a smartphone), a personal digital assistant (PDA) <NUM>, a laptop computer <NUM>, a tablet computer <NUM>, and a HMD <NUM>. However, any other or additional client devices could be used in the system <NUM>.

In this example, some client devices <NUM>-<NUM> communicate indirectly with the network <NUM>. For example, the client devices <NUM> and <NUM> (mobile devices <NUM> and PDA <NUM>, respectively) communicate via one or more base stations <NUM>, such as cellular base stations or eNodeBs (eNBs). Mobile device <NUM> includes smartphones. Also, the client devices <NUM>, <NUM>, and <NUM> (laptop computer, tablet computer, and HMD, respectively) communicate via one or more wireless access points <NUM>, such as IEEE <NUM> wireless access points. As described in more detail below the HMD <NUM> can display a <NUM>° view of a point cloud. Note that these are for illustration only and that each client device <NUM>-<NUM> could communicate directly with the network <NUM> or indirectly with the network <NUM> via any suitable intermediate device(s) or network(s). In certain embodiments, server <NUM> or any client device <NUM>-<NUM> can be used to compress a point cloud and transmit the data to another client device such as any client device <NUM>-<NUM>.

In certain embodiments, the mobile device <NUM> (or any other client device <NUM>-<NUM>) can transmit information securely and efficiently to another device, such as, for example, the server <NUM>. The mobile device <NUM> (or any other client device <NUM>-<NUM>) can function as a VR display when attached to a headset via brackets, and function similar to HMD <NUM>. The mobile device <NUM> (or any other client device <NUM>-<NUM>) can trigger the information transmission between itself and server <NUM>.

Although <FIG> illustrates one example of a system <NUM>, various changes can be made to <FIG>. For example, the system <NUM> could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. While <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

The processes and systems provided in this disclosure allow for a client device <NUM>-<NUM> or the server <NUM> to compress, transmit, receive, render a point cloud, or a combination thereof. For example, the server <NUM> can then compress and transmit the point cloud data to client devices <NUM>-<NUM>. For another example, any client device <NUM>-<NUM> can compress and transmit point cloud data to any client devices <NUM>-<NUM>.

<FIG> and <FIG> illustrate example devices in a computing system in accordance with an embodiment of this disclosure. In particular, <FIG> illustrates an example server <NUM>, and <FIG> illustrates an example electronic device <NUM>. The server <NUM> could represent the server <NUM> of <FIG>, and the electronic device <NUM> could represent one or more of the client devices <NUM>-<NUM> of <FIG>.

Server <NUM> can represent one or more local servers, one or more compression servers, or one or more encoding servers. As shown in <FIG>, the server <NUM> includes a bus system <NUM> that supports communication between at least one processor(s) <NUM>, at least one storage device(s) <NUM>, at least one communications interface <NUM>, and at least one input/output (I/O) unit <NUM>.

The processor <NUM> executes instructions that can be stored in a memory <NUM>. The instructions stored in memory <NUM> can include instructions for decomposing a point cloud, compressing a point cloud. The instructions stored in memory <NUM> can also include instructions for encoding a point cloud in order to generate a bitstream. The instructions stored in memory <NUM> can also include instructions for rendering the point cloud on an omnidirectional <NUM>° scene, as viewed through a VR headset, such as HMD <NUM> of <FIG>. The processor <NUM> can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processor(s) <NUM> include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The memory <NUM> and a persistent storage <NUM> are examples of storage devices <NUM> that represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, or other suitable information on a temporary or permanent basis). The memory <NUM> can represent a random access memory or any other suitable volatile or nonvolatile storage device(s). The persistent storage <NUM> can contain one or more components or devices supporting longer-term storage of data, such as a ready-only memory, hard drive, Flash memory, or optical disc.

The communications interface <NUM> supports communications with other systems or devices. For example, the communications interface <NUM> could include a network interface card or a wireless transceiver facilitating communications over the network <NUM> of <FIG>. The communications interface <NUM> can support communications through any suitable physical or wireless communication link(s).

The I/O unit <NUM> allows for input and output of data. For example, the I/O unit <NUM> can provide a connection for user input through a keyboard, mouse, keypad, touchscreen, motion sensors, or any other suitable input device. The I/O unit <NUM> can also send output to a display, printer, or any other suitable output device.

In certain embodiments, server <NUM> implements the compression of a point cloud, as will be discussed in greater detail below. In certain embodiments, server <NUM> generates multiple <NUM>-D frames that correspond to the three dimensions of the point cloud. In certain embodiments, server <NUM> maps the three dimensions of a point cloud into <NUM>-D. In certain embodiments, server <NUM> generates a compressed bitstream by encoding the compressed two-dimensional frames that represent the point cloud.

Note that while <FIG> is described as representing the server <NUM> of <FIG>, the same or similar structure could be used in one or more of the various client devices <NUM>-<NUM>. For example, a desktop computer <NUM> or a laptop computer <NUM> could have the same or similar structure as that shown in <FIG>.

<FIG> illustrates an electronic device <NUM> in accordance with an embodiment of this disclosure. The embodiment of the electronic device <NUM> shown in <FIG> is for illustration only, and other embodiments could be used without departing from the scope of this disclosure. The electronic device <NUM> can come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular implementation of an electronic device. In certain embodiments, one or more of the client devices <NUM>-<NUM> of <FIG> can include the same or similar configuration as electronic device <NUM>. In certain embodiments, electronic device <NUM> can be an encoder and a decoder.

In certain embodiments, electronic device <NUM> is usable with data transfer, image or video compression, image or video decompression, encoding, decoding, and media rendering applications. The electronic device <NUM> can be a mobile communication device, such as, for example, a wireless terminal, a desktop computer (similar to desktop computer <NUM> of <FIG>), a mobile device (similar to mobile device <NUM> of <FIG>), a PDA (similar to PDA <NUM> of <FIG>), a laptop (similar to laptop computer <NUM> of <FIG>), a tablet (similar to tablet computer <NUM> of <FIG>), a head-mounted display (similar to HMD <NUM> of <FIG>), and the like.

As shown in <FIG>, the electronic device <NUM> includes an antenna <NUM>, a radiofrequency (RF) transceiver <NUM>, a transmit (TX) processing circuitry <NUM>, a microphone <NUM>, and a receive (RX) processing circuitry <NUM>. The electronic device <NUM> also includes a speaker <NUM>, a one or more processors <NUM>, an input/output (I/O) interface (IF) <NUM>, an input <NUM>, a display <NUM>, and a memory <NUM>.

The RF transceiver <NUM> receives, from the antenna <NUM>, an incoming RF signal transmitted by another component on a system. For example, the RF transceiver <NUM> receives RF signal transmitted by a BLUETOOTH or WI-FI signal from an access point (such as a base station, Wi-Fi router, Bluetooth device) of the network <NUM> (such as a WI-FI, BLUETOOTH, cellular, <NUM>, LTE, LTE-A, WiMAX, or any other type of wireless network). The RF transceiver <NUM> can down-convert the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuitry <NUM> that generates a processed baseband signal by filtering, decoding, or digitizing the baseband or intermediate frequency signal, or a combination thereof.

The TX processing circuitry <NUM> receives analog or digital voice data from the microphone <NUM> or other outgoing baseband data from the processor <NUM>. The outgoing baseband data can include web data, e-mail, or interactive video game data. The TX processing circuitry <NUM> encodes, multiplexes, digitizes, or a combination thereof, the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The RF transceiver <NUM> receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuitry <NUM> and up-converts the baseband or intermediate frequency signal to an RF signal that is transmitted via the antenna <NUM>.

The processor <NUM> can include one or more processors or other processing devices and execute the OS <NUM> stored in the memory <NUM> in order to control the overall operation of the electronic device <NUM>. The processor <NUM> is also capable of executing other applications <NUM> resident in the memory <NUM>, such as decompressing and generating a received point cloud.

The processor <NUM> can execute instructions that are stored in a memory <NUM>. The processor <NUM> can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in some embodiments, the processor <NUM> includes at least one microprocessor or microcontroller. Example types of processor <NUM> include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The processor <NUM> is also capable of executing other processes and programs resident in the memory <NUM>, such as operations that receive, store, and timely instruct by providing image capturing and processing. The processor <NUM> can move data into or out of the memory <NUM> as required by an executing process. In some embodiments, the processor <NUM> is configured to execute the plurality of applications <NUM> based on the OS <NUM> or in response to signals received from eNBs or an operator. The processor <NUM> is also coupled to the I/O interface <NUM> that provides the electronic device <NUM> with the ability to connect to other devices, such as client devices <NUM>-<NUM>. The I/O interface <NUM> is the communication path between these accessories and the processor <NUM>.

The processor <NUM> is also coupled to the input <NUM>. The operator of the electronic device <NUM> can use the input <NUM> to enter data or inputs into the electronic device <NUM>. Input <NUM> can be a keyboard, touch screen, mouse, track-ball, voice input, or any other device capable of acting as a user interface to allow a user in interact with electronic device <NUM>. For example, the input <NUM> can include voice recognition processing thereby allowing a user to input a voice command via microphone <NUM>. For another example, the input <NUM> can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel can recognize, for example, a touch input in at least one scheme among a capacitive scheme, a pressure sensitive scheme, an infrared scheme, or an ultrasonic scheme. For example, in the capacitive scheme, the input <NUM> can recognize touch or proximity. The input <NUM> can also include a control circuit. Input <NUM> can be associated with sensor(s) <NUM> and/or a camera by providing additional input to processor <NUM>. As discussed in greater detail below, sensor <NUM> includes inertial sensors (such as accelerometers, gyroscope, and magnetometer), optical sensors, motion sensors, cameras, pressure sensors, heart rate sensors, altimeter, and the like. For example, input <NUM> can utilize motion as detected by a motion sensor, associated with sensor <NUM>, as an input.

The processor <NUM> is also coupled to the display <NUM>. The display <NUM> can be a liquid crystal display (LCD), light-emitting diode (LED) display, organic LED (OLED), active matrix OLED (AMOLED), or other display capable of rendering text and/or graphics, such as from websites, videos, games, images, and the like. Display <NUM> can be sized to fit within a HMD. Display <NUM> can be a singular display screen or multiple display screens capable of creating a stereoscopic display. In certain embodiments, display <NUM> is a heads-up display (HUD).

The memory <NUM> can include persistent storage (not shown) that represents any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory <NUM> can contain one or more components or devices supporting longer-term storage of data, such as a ready only memory, hard drive, Flash memory, or optical disc.

Electronic device <NUM> can further include one or more sensors <NUM> that meter a physical quantity or detect an activation state of the electronic device <NUM> and convert metered or detected information into an electrical signal. For example, sensor(s) <NUM> may include one or more buttons for touch input (located on the headset or the electronic device <NUM>), one or more cameras, a gesture sensor, an eye tracking sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a Red Green Blue (RGB) sensor), a bio-physical sensor, a temperature/humidity sensor, an illumination sensor, an Ultraviolet (UV) sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG) sensor, an Electrocardiogram (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, and the like. The sensor(s) <NUM> can further include a control circuit for controlling at least one of the sensors included therein. As will be discussed in greater detail below, one or more of these sensor(s) <NUM> may be used to control a user interface (UI), detect UI inputs, determine the orientation and facing the direction of the user for three-dimensional content display identification, etc. Any of these sensor(s) <NUM> may be located within the electronic device <NUM>, within a secondary device operably connected to the electronic device <NUM>, within a headset configured to hold the electronic device <NUM>, or in a singular device where the electronic device <NUM> includes a headset.

As will be discussed in greater detail below, in this illustrative embodiment, electronic device <NUM> receives an encoded and compressed bitstream. The electronic device <NUM> decodes the compressed bitstream into multiple <NUM>-D frames. In certain embodiments, the decoded bitstream also includes an occupancy map. The electronic device <NUM> decompresses the multiple <NUM>-D frames. The multiple <NUM>-D frames can include a frame that indicates coordinates of each point of a point cloud. A frame can include the location of each geometric point of the point cloud. For example, the frame can include a pictorial depiction of each geometric point of the point cloud as represented in <NUM>-D. Another frame can include an attribute of each point such as color. The electronic device <NUM> can then generate the point cloud in three dimensions.

As will be discussed in greater detail below, in this illustrative embodiment, electronic device <NUM> can be similar to server <NUM> and encode a point cloud. The electronic device <NUM> can generate multiple <NUM>-D frames that represent the geometry and texture or color or both of the point cloud. The point cloud can be mapped to the <NUM>-D frame. For example, one frame can include the geometric points. In another example, another frame can include the texture or color or both of the point cloud. The electronic device <NUM> can compress the <NUM>-D frames. The electronic device <NUM> can generate an occupancy map to indicate the location of valid pixels within each frame. The electronic device <NUM> can encode the frames to generate a compressed bitstream.

Although <FIG> and <FIG> illustrate examples of devices in a computing system, various changes can be made to <FIG> and <FIG>. For example, various components in <FIG> and <FIG> could be combined, further subdivided, or omitted and additional components could be added according to particular needs. In addition, as with computing and communication networks, electronic devices and servers can come in a wide variety of configurations, and <FIG> and <FIG> do not limit this disclosure to any particular electronic device or server.

<FIG> illustrates a point cloud and an example mesh in accordance with an embodiment of this disclosure. Point cloud <NUM> depicts an illustration of a point cloud. A point cloud is digitized data that visually defines an object in <NUM>-D space. As depicted the point cloud <NUM> each point represents an external coordinate of the object, similar to a topographical map. The coordinate of each point has a geographical location as well as an attribute. The attribute can be color, intensity, texture, and the like.

Similarly, mesh <NUM> depicts an illustration of a <NUM>-D mesh. The mesh is a digitized data that visually defines an object in <NUM>-D space. The object is defined by many polygons. Each polygon can portray various information, such as topological, geometrical, attribute (such as color, texture, and the like), and the like. For example, topological data provide connectivity information among vertices such as adjacency of vertices, edges, and faces. Geometrical information provides the geometric location of each vertex in <NUM>-D space. Attribute information provides the normal, color, and application dependent information for each individual vertex. The vertices of each polygon are similar to the points in the point cloud <NUM>. Each polygon of the mesh <NUM> represents the external surface of the object.

Point clouds and meshes, similar to point cloud <NUM> and mesh <NUM>, require substantial bandwidth to transmit from one computing device to another. Compression is necessary to reduce storage and bandwidth requirements. For example, lossy compression can compress a Point cloud and mesh while maintaining the distortion within a tolerable level while reducing the size of the data.

<FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG> illustrate an example block diagram of an encoder or decoder in accordance with embodiments of this disclosure. Specifically, <FIG>, <FIG>, <FIG>, and <FIG> illustrate example encoders while <FIG>, <FIG>, and <FIG> illustrate example decoders. The encoders of <FIG>, <FIG>, <FIG>, and <FIG> can be similar to the server of <FIG>, any of the client devices <NUM>-<NUM> of <FIG>, the server <NUM> of <FIG>, and the electronic device <NUM> of <FIG>. The decoders of <FIG>, <FIG>, and <FIG> can be similar to any of the client devices <NUM>-<NUM> of <FIG> and the electronic device <NUM> of <FIG>. The encoders of <FIG>, <FIG>, <FIG>, and <FIG> can communicate via network <NUM> to any decoders of <FIG>, <FIG>, and <FIG>. The embodiment of the encoders as shown in <FIG>, <FIG>, <FIG>, and <FIG> and the decoders as shown in <FIG>, <FIG>, and <FIG> are for illustration only. Other embodiments could be used without departing from the scope of the present disclosure.

The encoders of <FIG>, <FIG>, <FIG>, and <FIG> can compress, encode, and transmit a point cloud or a mesh, or both. In certain embodiments, the encoders of <FIG>, <FIG>, <FIG>, and <FIG> generate multiple <NUM>-D frames in which a point cloud or a <NUM>-D mesh is mapped or projected onto. For example, the point cloud is unwrapped and mapped onto the <NUM>-D frame. For example, the point cloud can be unwrapped along one axis (such as the Y-axis), and the image is mapped along the remaining axis (such as X and Z axis). In certain embodiments, the encoders of <FIG>, <FIG>, <FIG>, and <FIG> generate an occupancy map that indicates where each pixel of the point cloud is located when the point cloud is mapped onto the <NUM>-D frame. In certain embodiments, encoders of <FIG>, <FIG>, <FIG>, and <FIG> are a web server, a server computer such as a management server, or any other electronic computing system capable of, mapping the three dimensions of a point cloud into two dimensions, compressing encoders of <FIG>, <FIG>, <FIG>, and <FIG> the <NUM>-D images, and encoding the images for transmission. In certain embodiments, each encoder of <FIG>, <FIG>, <FIG>, and <FIG> are a "cloud" of computers interconnected by one or more networks, where each is a computing system utilizing clustered computers and components to act as a single pool of seamless resources when accessed through network <NUM> of <FIG>.

The decoders of <FIG>, <FIG>, and <FIG> can decode, decompress, and generate a received point cloud or a mesh, or both. In certain embodiments, the decoders of <FIG>, <FIG>, and <FIG> generate multiple point clouds from a received bitstream that includes <NUM>-D frames. For example, each of the pixels of the point cloud can be mapped based on the information received in the <NUM>-D frames and a received occupancy map.

<FIG> illustrates encoder <NUM>. Encoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that encodes and compresses a point cloud for transmission. In certain embodiments, encoder <NUM> packages a point cloud for transmission by a bitstream to one or more decoders. Encoder <NUM> includes a decompose point cloud <NUM>, a compression engine <NUM>, includes an occupancy map generator <NUM>, an auxiliary information generator <NUM>, and a multiplexer <NUM>. The decompose point cloud <NUM> generates a geometry frame <NUM> and an attribute frame <NUM>.

Encoder <NUM> receives a point cloud <NUM> via an input. The input of the point cloud <NUM> can be accessed via an information repository or receives via a transmission from another electronic device.

The input of the point cloud <NUM> enters the encoder <NUM> and is mapped by the decompose point cloud <NUM>. In certain embodiments, the decompose point cloud is similar to a demultiplexer as it separates various features of the received input. For example, the decompose point cloud <NUM> can separate the geometry of the point cloud, and other attributes of the point cloud. The attibutes can include color, texture, intensity, normal, reflection, and the like. The decompose point cloud <NUM> is a mapping function that maps the point cloud <NUM> onto one or more <NUM>-D frames. For example, the decompose point cloud <NUM> generates two frames a geometry frame <NUM> and an attribute frame <NUM>. In certain embodiments, additional frames (not shown) are generated by the decompose point cloud <NUM>. In certain embodiments, the decompose point cloud <NUM> maps the point cloud into a frame of a video. For example, in an embodiment not forming part of the claimed invention, each pixel of the X coordinates of the point cloud can be mapped onto the R component of the video frame, where the R component of the video frame is the red color portion of a video. Similarly, each pixel of the Y coordinates of the point cloud can be mapped onto the G component of the video frame, where the G component of the video frame is the green color portion of a video. Similarly, each pixel of the Z coordinates of the point cloud can be mapped onto the B component of the video frame, where the B component of the video frame is the blue color portion of a video. That is, the R, G, and B video frames include the geometry of the point cloud mapped onto the <NUM>-D of each video frame. Similarly, additional video frames can include various attributes of each pixel such as color, intensity, texture, and the like. For example, the point cloud attributes can be mapped on different regions of the same video picture. In another example, the point cloud attributes can be mapped on separate video frames.

The geometry frame <NUM> is the geometric location of each pixel of the point cloud. For example, the geometry frame <NUM> is a mapping of the3-D locations of the point-cloud points onto the two dimensions of a video frame. In certain embodiments, not forming part of the claimed invention, the geometry frame <NUM> indicates all three dimensions on a two-dimensional projection. In certain embodiments, not all three XYZ coordinates of the points are stored in the geometry frame <NUM>. For example, the geometry frame <NUM> stores two of the three XYZ coordinates in such a way that the missing coordinate(s) can be determined. For instance, the Y coordinate is not stored on the geometry frame <NUM> but the X and Z coordinates are stored such that the Y coordinate can be determined from the row number of the pixel storing the X and Z coordinates within the 2D frame. The attribute frame <NUM> stores an attibute such as the RGB color of each pixel of the point cloud. In certain embodiments, the attribute frame <NUM> stores an attribute such as the color or the texture of each pixel of the point cloud. In certain embodiments, the attributes frame <NUM> can include multiple frames. For example, the each individual attribute frame can indicate a color, normal, texture coordinates, material properties, intensity, and the like.

The geometry frame <NUM> and the attribute frame <NUM> are compressed by the compression engine <NUM>. The compression engine <NUM> can be a video compression codec or an image compression codec. In certain embodiments, the compression engine <NUM> compresses the received frames by a codec such as HEVC, SHVC, AVC, SVC, VP9, VP8 JVET, AVC, JPEG, and the like. In certain embodiments, the compression engine <NUM> compresses each frame (geometry frame <NUM> and the attribute frame <NUM>) individually and send each compressed frame to the multiplexer <NUM>. In certain embodiments, the compression engine <NUM> compresses all the received frames together (such as the geometry frame <NUM> and the attribute frame <NUM>) and send the frames together to the multiplexer <NUM>.

The occupancy map generator <NUM> generates a binary mask. The occupancy map generator <NUM> analyzes the received point cloud <NUM>, the geometry frame <NUM> and the attribute frame <NUM> and identifies valid and invalid points of each frame. The generated binary mask indicates whether each pixel on each frame (the geometry frame <NUM> and the attribute frame <NUM>) is a valid pixel or blank. For example, the binary mask is used to determine the region of the video frame that contains valid point cloud attribute information. The region and therefore the generated binary mask to which the region corresponds to can be the same for all projected attributes, or it could be different for a portion of the attributes. For example, the occupancy map generator <NUM> can generate a single binary mask that indicates valid pixels for all attributes. In another example, the occupancy map generator <NUM> can generate multiple binary masks that indicate valid pixels for each attribute. In certain embodiments, the generated binary mask is also inputted into the compression engine <NUM>. For example, the generated binary mask can be compressed as another <NUM>-D frame similar to the geometry frame <NUM> or the attribute frame <NUM>. In certain embodiments, the generated binary mask can be multiplexed with the compressed <NUM>-D frames and transmitted as metadata with the compressed <NUM>-D frames, as a single bitstream. In certain embodiments, the generated binary mask can be transmitted as metadata as a separate bitstream. The generated binary mask is described in further detail below with reference to <FIG>.

The auxiliary information generator <NUM> generates metadata that is transmitted with the compressed geometry frame <NUM> and the compressed attribute frame <NUM>. The auxiliary information that is generated by the auxiliary information generator <NUM> is generated metadata that can be used by a decoder, to generate and render the three dimensions of the point cloud from the <NUM>-D frames. For example, the auxiliary information generator <NUM> generates data that relates the information of the geometry frame <NUM> and the information of the attribute frame <NUM> of each point in the point cloud. In certain embodiments, the auxiliary information generator <NUM> generates a look-up table. In certain embodiments, the auxiliary information generated by the auxiliary information generator <NUM> can be added to the encoded bitstream by the video encoders as metadata. For example, the metadata can be added to the encoded bitstream using a SEI message. In certain embodiments, the auxiliary information generated by the auxiliary information generator <NUM> can be encoded by non-video-based encoding methods, such as octree. In certain embodiments, the auxiliary information generated by the auxiliary information generator <NUM> can be added by the multiplexer <NUM>) to the encoded bitstream <NUM> without any encoding.

The multiplexer <NUM> combines the inputs from the compression engine <NUM>, the occupancy map generator <NUM>, and the auxiliary information generator <NUM> and creates a single output, such as bitstream <NUM>.

In certain embodiments, the geometries of the geometry frame <NUM> and the attribute frame <NUM> can be tuned to a particular video frame in order to improve the efficiency of the compression engine <NUM>. For example, the geometries of the geometry frame <NUM> and the attribute frame <NUM> can be transmitted with the video or every frame of the video. For instance, the transmitted bitstream <NUM> can include a mesh box scheme of spherical video V2 RFC. In another instance, the transmitted bitstream <NUM> can include an extensible polygon-based projection format (EPPF). Both the spherical video V2 RFC and the EPPF store the vertex and texture coordinates in a list in combination with relevant video and audio data.

<FIG> illustrates decoder <NUM>. Decoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that decodes and generates the point cloud. In certain embodiments, the decoder extracts the compressed geometry and attributes from the bitstream. In certain embodiments, the decoder <NUM> maps the geometry and attribute to generate the point cloud. The decoder <NUM> includes a demultiplexer <NUM>, a decompression engine <NUM>, and a reconstruction engine <NUM>.

The demultiplexer <NUM> is a component that takes the single input such as bitstream <NUM> and routes it to one of several output lines. Specifically, the demultiplexer <NUM> receives the bitstream <NUM> and extracts the frames (such as the geometry frame <NUM> and the attribute frame <NUM>), the occupancy map <NUM>, and the auxiliary information <NUM>.

The decompression engine <NUM> decompresses the geometry frame <NUM> and the attribute frame <NUM>. In certain embodiments, the decompression engine <NUM> produces an exact replication of the original pre-compressed frames. In certain embodiments, the decompression engine <NUM> produces a similar replication of the original pre-compressed frames. That is, the decompression engine <NUM> reverses the effects of the compression engine <NUM> of the encoder.

The occupancy map <NUM> is the generated binary map. The occupancy map <NUM> identifies valid and invalid pixels of each frame. The binary mask indicates which pixels of each frame contain information regarding the generation of the three dimensions of the point cloud from the <NUM>-D frames. Additional details regarding the binary map are described in further detail below with reference to <FIG>.

The auxiliary information <NUM> is metadata that is used by the reconstruction engine <NUM> to generate and render the point cloud. For example, the auxiliary information includes data that relates the geometry frame <NUM> and the attribute frame <NUM> to each point in the point cloud. In certain embodiments, the auxiliary information <NUM> is a look-up table.

The reconstruction engine <NUM> reconstructs the three-dimensional shape of the point cloud based on the decompressed <NUM>-D frames, the occupancy map <NUM>, and the auxiliary information <NUM>. The reconstruction engine <NUM> generates the three-dimensional point cloud. The generated point cloud can then be transmitted to a display (similar to display <NUM> of <FIG>) via an output.

<FIG> illustrates encoder <NUM>. Encoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that encodes and compresses a point cloud for transmission. Encoder <NUM> is similar to encoder <NUM> of <FIG>. Encoder <NUM> includes mapping <NUM>, encoding <NUM> and <NUM>, and a multiplexer <NUM>. The point cloud <NUM> is the input into the encoder <NUM>, and bitstream <NUM> is the output. Similarly <FIG> illustrates decoder <NUM>. Decoder <NUM> is similar to decoder <NUM> of <FIG>. Decoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that decodes the compressed point cloud of the encoder <NUM>. Decoder <NUM> includes a demultiplexer <NUM>, decoding <NUM> and <NUM>, frame unpacking <NUM> and renderer <NUM>. The input is the same or similar bitstream <NUM> from encoder <NUM>, and the output is the generated point cloud <NUM>.

Mapping <NUM> maps the point cloud <NUM> into two dimensions. In certain embodiments, the mapping <NUM> separates the geometry values and attribute values of the point cloud <NUM>. For example, the X, Y, and Z geometry components which correspond to the <NUM>-D coordinates of each pixel are stored in a separate frame than the R, G, and B color components which correspond to the color attribute of each respective pixel. The frame containing the geometry information is sent to the frame packing <NUM> while the frame containing the color information is sent to the encoding <NUM>.

In certain embodiments, mapping <NUM> maps the geometry and the color components of the point cloud to images that are then encoded using an image or video codec via encoding <NUM> or <NUM>. For example, mapping <NUM> can map the vertex and the texture coordinates to an image. For instance, the latitude of the points can correspond to the row number while the longitude of points can correspond to the column number in the image. Converting the Cartesian coordinates of X, Y, and Z to the spherical coordinates of r, θ, and ϕ can be performed using the following equations.

The above equations depict θ as the latitude or row number and ϕ is the longitude or column number in the image. The above equations can yield two separate images I and T. Two separate images I and T are used to store the (X, Y, Z) and (U, V) coordinates respectively. In certain embodiments, X, Y, and Z are vertex coordinates depicting the vertices of an object in three dimensions, while U, and V denote the texture coordinates when the <NUM>-D texture is mapped onto <NUM>-D. For example:
<MAT> <MAT> <MAT> <MAT> <MAT>.

Mapping <NUM> initially maps the point cloud from the <NUM>-D space to <NUM>-D planes. In certain embodiments, mapping <NUM> processes the geometry and attribute components identically. This allows the decoder <NUM> of <FIG> to identify the corresponding values to each point without additional information. In certain embodiments, the mapping <NUM> processes the geometry and attribute components differently. If the mapping <NUM> processes the geometry and attribute values differently, additional data such as metadata or a look-up table is generated to identify the related geometry and attribute values which are required to be encoded via encoding <NUM> and <NUM> respectively. Mapping the geometry and attribute of the point cloud <NUM> is described in further detail with respect to in <FIG> below.

Frame packing <NUM> packs the XYZ geometry values of the point cloud <NUM> to support chroma-subsampling schemes such as <NUM>:<NUM>:<NUM> rather than <NUM>:<NUM>:<NUM> as assumed in the previous descriptions. In certain embodiments, the geometry values X, Y, and Z indicate positions of the points in a <NUM>-D space that are packed to support a predetermined chroma subsampling. For example, the points in the <NUM>-D space are packed to support a chroma subsampling of <NUM>:<NUM>:<NUM> when the projected geometry values are encoded by encoding <NUM>. Example frame packing are discussed with reference to <FIG> below.

In certain embodiments, frame packing <NUM> is omitted from the encoder <NUM> when the mapping <NUM> stores the geometry frame in <NUM>:<NUM>:<NUM> format. That is, the geometry values X, Y, and Z are sent to encoding <NUM> from mapping <NUM>. Then the geometry values are compressed using a video encoder. For instance, the geometry values can be compressed via HEVC encoding.

The encoder <NUM> sends the R, G, and B color components to the encoding <NUM>. In certain embodiments, the RGB color values are compressed by encoding <NUM> using a video encoder. In certain embodiments, other encoders can be used. In certain embodiments, a frame packing step similar to the frame packing <NUM> can be used with the geometry values of R, G, and B.

The multiplexer <NUM> combines the X, Y, and Z geometry components (coordinates) and the R, G, and B color components into a single bitstream <NUM>. In certain embodiments, the XYZ and the RGB components of the point cloud mapped to 2D are multiplexed with additional audio and video tracks.

Encoder <NUM> can indicate locations within the <NUM>-D mapped geometry and attribute frames where the point cloud data exists. Other locations within the <NUM>-D frames that do not contain valid point cloud data are filled with zero or another default value.

In certain embodiments, encoder <NUM> can indicate the absence of the point cloud data in the <NUM>-D mapped frames. For instance, the three components of a geometry video are <NUM>, <NUM>, and <NUM>, at any location with no mapping data. In certain embodiments, encoder <NUM> can generate a syntax to indicate the absence of the point cloud data in the <NUM>-D mapped frames. In certain embodiments, the syntax is similar to the auxiliary information <NUM> of <FIG>. The syntax can be transmitted with the bitstream <NUM>. The syntax can include a pixel value that is used to signal the absence of point cloud data at any pixel location in the <NUM>-D frame.

In certain embodiments, by generating metadata by the mapping <NUM> the encoder <NUM> can indicate the type of video track in the bitstream, whether it is normal <NUM>-D video track, a geometry video track of a point cloud, or an attribute video track of a point cloud. For example, the type of video track can be identified by the syntax: unsigned char pc_track_type. The integer <NUM> can be used to indicate that the information is a normal <NUM>-D video track and not a point cloud track. The integer <NUM> can be used to indicate a point cloud geometry video track. The integer <NUM> can be used to indicate a point cloud attribute video track such as color. The integer <NUM> can be used to indicate another point cloud attribute video track such as normal. Additional integers can be used to indicate additional attributes. In certain embodiments, of the above syntax, an integer that indicates the geometry video track is lower than any integer that indicates an attribute video track. For example, an integer that indicates a geometry track is a lower number than an integer that indicates a color attribute track. Therefore, in this example when the bitstream <NUM> is received by the decoder <NUM>, any frame that is received with a syntax integer other than <NUM> and an integer that indicates geometry, is identified as an attribute type.

For example, <FIG> illustrates the <NUM>-D mapping of a point cloud using multiple video tracks in accordance with an embodiment of this disclosure. Environment 600C of <FIG> illustrates a MPEG video with multiple tracks. In particular, the generated metadata by the mapping <NUM> the encoder <NUM> can indicate the type of video track in the bitstream. For example, track <NUM> illustrates an audio track, and tracks <NUM>, <NUM>, and <NUM> illustrates video tracks. For instance, track <NUM> is a <NUM>° video, track <NUM> is the XYZ geometry of the point cloud, and track <NUM> illustrates the color attribute of the point cloud. Additional tracks can be added for each additional attribute. For example, track <NUM> can include the integer <NUM> to indicate that the information is a normal <NUM>-D video track and not a point cloud track. Similarly, track <NUM> can include the integer <NUM> to indicate a point cloud geometry video track. Similarly, the track <NUM> can include the integer <NUM> to indicate a point cloud attribute video track such as color.

In certain embodiments, by generating metadata by the mapping <NUM> the encoder <NUM> can indicate a composite of different point clouds or a composite of point cloud(s) and a <NUM>-D video. A <NUM>-D video could be a normal <NUM>-D video of the front view or a video of <NUM>-degree view. For example, a scene is a composite of one or more multiple foreground point cloud(s) with a background point cloud. For example, the syntax of unsigned short int pc_pos_offset_X can be used to indicate the X position offset of the point cloud object within the background point cloud or <NUM>-D video. In another example, the syntax of unsigned short int pc_pos_offset_Y can be used to indicate the Y position offset of the point cloud object within the background video. In another example, the syntax of unsigned short int pc_pos_offset_Z can be used to indicate the Z position offset of the point cloud object within the background video. In another example, the syntax of unsigned short int pc_rot_X can be used to indicate the initial pitch angle of a foreground point cloud. In another example, the syntax of unsigned short int pc_rot_Y can be used to indicate the initial yaw angle of a foreground point cloud. In another example, the syntax of unsigned short int pc_rot_Z can be used to indicate the initial roll angle of a foreground point cloud. In another example, the syntax of unsigned short int pc_scale can be used to indicate the scale of a foreground point cloud. In another example, the syntax of unsigned char pc_alpha can be used to indicate the transparency value of a foreground point cloud. In another example, the syntax of unsigned int pc_time_offset can be used to indicate the time offset of a point cloud with respect to the background video. In another example, the frame rate of each point cloud and that of the background video can be different. For instance, a point cloud object can have a frame rate of <NUM> Frames Per Second (FPS) and the background video can have a frame rate of <NUM> FPS. In certain embodiments, if the background video does not contain depth information, the point cloud tracks are composited on top the background video.

Decoder <NUM> extracts the geometry and attribute bitstreams from the bitstream <NUM> by demultiplexer <NUM>. The geometry bitstream are decoded by decoding <NUM>. Similarly, the attribute bitstream are decoded by decoding <NUM>. In certain embodiments, the decoding <NUM> is a <NUM>+-bit video decoder. In certain embodiments, the decoding <NUM> is an <NUM>-bit video decoder.

In certain embodiments, decoding <NUM> can be lossless or lossy. Similarly, in certain embodiments, decoding <NUM> can be lossless or lossy. For example, lossless or lossy decoding is dependent on a <NUM>-D mapping scheme and the quantization parameters of the engaged <NUM>-D video encoders such as encoding <NUM> and <NUM> from the encoder <NUM>. If encoding <NUM> or <NUM> is a lossy encoder, then the number and locations of the points in the two generated videos may not be matched. In one embodiment, if the encoder is lossy, the encoder generates a binary mask (also known as alpha mask or occupancy map) and add it to the output bitstream <NUM> either uncompressed or using lossless compression.

Frame unpacking <NUM> unpacks the geometry values as packed by frame packing <NUM> of the encoder <NUM>. Renderer <NUM> renders the point cloud <NUM> by mapping the attribute values onto the <NUM>-D space based on the geometry values. For example, renderer <NUM> extracts the geometry values X, Y, and Z as well as the attribute values, R, G, and B to renders the point cloud <NUM>. If a binary mask is available in bitstream <NUM>, renderer <NUM> uses it to find the location of valid points in the decoded geometry and attribute frames. In one embodiment, if encoder <NUM> encodes the geometry or one of attribute videos using lossless compression, and if the geometry and attributes are all stored at identical locations in the <NUM>-D videos, then encoder <NUM> may not store a binary mask in bitstream <NUM>. In such a case, the render <NUM> can use the location of valid pixels in one of the lossless <NUM>-D videos to indicate the valid pixels in the lossy <NUM>-D videos.

<FIG> illustrates encoder <NUM>. Encoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that encodes and compresses a point cloud for transmission. Encoder <NUM> is similar to encoder <NUM> of <FIG> and encoder <NUM> of <FIG>. Encoder <NUM> includes a demultiplexer <NUM>, a number of frames with a respective video encoder and a multiplexer <NUM>. The frames include a geometry frame <NUM>, a color frame <NUM>, a normal texture frame <NUM>, and a texture coordinate frame. Each frame is associated with an individual video encoder <NUM>-<NUM>. The point cloud <NUM> is the input into the encoder <NUM> and bitstream <NUM> is the output. Similarly <FIG> illustrates decoder <NUM>. Decoder <NUM> is similar to decoder <NUM> of <FIG> and decoder <NUM> of <FIG>. Decoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that decodes the compressed point cloud of the encoder <NUM>. Decoder <NUM> includes a demultiplexer <NUM>, video decoders 754A-D and a multiplexer. The input is the same or similar bitstream <NUM> from encoder <NUM> and the output is the generated point cloud <NUM>.

In certain embodiments, point cloud <NUM>, the point cloud <NUM> of <FIG> and the point cloud <NUM> of <FIG> are the same. In certain embodiments, each point of the point cloud <NUM> can be represented by a single equation that indicates each attribute and the geometric location of the point within the point cloud. For example, the following equation can be used:
Equation: <MAT>.

In the above equation, n is an integer starting at <NUM>. In the above equation (Xn, Yn, Zn) are the X, Y, and Z coordinates of a single point n in <NUM>-D space. In the above equation (Rn, Bn, Gn) are the color of the point n in <NUM>-D space. In the above equation (NXn, NYn, NZn) is the normal for the point n in <NUM>-D space. In the above equation (Un, Vn) are the texture coordinates of the point n. In certain embodiments, of the above equation, a point n can have a subset of attributes. In certain embodiments, of the above equation, a point n can have additional attributes such as material properties, intensity, quality, flags, and the like.

Demultiplexer <NUM> is similar to the mapping <NUM> of <FIG> and the decompose point cloud <NUM> of <FIG>. In certain embodiments, demultiplexer <NUM> the mapping <NUM> of <FIG> and the decompose point cloud <NUM> of <FIG> are the same. Demultiplexer <NUM> analyzes the point cloud <NUM> and decomposes the point cloud into various components. Based on the above equation the demultiplexer <NUM> can separate out the various variables for each point n in the point cloud <NUM>. For example, the demultiplexer <NUM> extracts the geometry attributes (Xn, Yn, Zn) and the geometry attributes are mapped into geometry frame <NUM>. Similarly, the demultiplexer <NUM> extracts the color attributes (Rn, Bn, Gn) and the color attributes are mapped into color frame <NUM>. Similarly, the demultiplexer <NUM> extracts the normal attributes (NXn, NYn, NZn) and the normal attributes are mapped into normal texture frame <NUM>. Similarly, the demultiplexer <NUM> extracts the texture coordinate attributes (Un, Vn) and the texture coordinate attributes are mapped into texture coordinate frame <NUM>.

Each frame <NUM>-<NUM> maps a different attribute of the point cloud <NUM> into a video frame. For example, frames <NUM>-<NUM> illustrate four mappings that correspond to storing eleven different attribute components in the four video frames that of the geometry frame <NUM>, the color frame <NUM>, the normal texture frame <NUM>, and the texture coordinate frame <NUM>. In certain embodiments, each frame <NUM>-<NUM> can be the same size, each frame <NUM>-<NUM> can be different sizes, or a portion of the frames <NUM>-<NUM> can be the same size while other frames are different sizes. For example, the geometry frame <NUM> can have the width of W and a height of H.

The geometry frame <NUM> represents a <NUM>-D video picture containing the geometric attributes of the point cloud <NUM>. The video encoder <NUM> receives the <NUM>-D video picture containing the geometric attributes and compresses the video of the geometric attributes. Similarly, the color frame <NUM> represents a <NUM>-D video picture containing the color attributes of the point cloud <NUM>. The video encoder <NUM> receives the <NUM>-D video picture containing the color attributes and compresses the video of the color attributes. Similarly, the normal texture frame <NUM> represents a <NUM>-D video picture containing the normal attributes of the point cloud <NUM>. The video encoder <NUM> receives the <NUM>-D video picture containing the normal attributes and compresses the video of the normal attributes. Similarly, the texture coordinate frame <NUM> represents a <NUM>-D video picture containing the texture coordinates of the point cloud <NUM>. The video encoder <NUM> receives the <NUM>-D video picture containing the texture coordinates and compresses the video of the texture coordinates attributes. The multiplexer <NUM> combines the individually compressed frames to generate a bitstream <NUM>.

The decoder <NUM> extracts the geometry and attribute values from the bitstream <NUM> by demultiplexer <NUM>. The demultiplexer <NUM> splits the bitstream <NUM> into the four compressed frames. The compressed video of the geometry attributes are decoded by video decoder 754A. The compressed video of the color attributes are decoded by video decoder 754B. The compressed video of the normal attributes are decoded by video decoder 754C. The compressed video of the texture coordinate attributes are decoded by video decoder 754D.

The multiplexer <NUM> combines the decoded video pictures from video decoder 754A-D. The video decoder 754A sends the video picture of the geometry attributes (Xn, Yn, Zn) to the multiplexer <NUM>. The video decoder 754B sends the video picture of the color attributes (Rn, Bn, Gn) to the multiplexer <NUM>. The video decoder 754C sends the video picture of the normal attributes (NXn, NYn, NZn) to the multiplexer <NUM>. The video decoder 754D sends the video picture of the texture coordinate attributes (Un, Vn) to the multiplexer <NUM>. Thereafter, the multiplexer <NUM> generates the point cloud based on Pn = Xn, Yn, Zn, Rn, Bn, Gn, NXn, NYn, NZn, Un, Vn.

<FIG> illustrates encoder <NUM>. Encoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that encodes and compresses a point cloud for transmission. Encoder <NUM> is similar to encoder <NUM> of <FIG>, encoder <NUM> of <FIG>, and encoder <NUM> of <FIG>. Encoder <NUM> includes mapping <NUM>, video encoding, <NUM> and <NUM>, a binary mask compression, <NUM> and a multiplexer <NUM>. The point cloud <NUM> is the input into the encoder <NUM> and bitstream <NUM> is the output.

Similarly <FIG> illustrates decoder <NUM>. Decoder <NUM> is similar to decoder <NUM> of <FIG>, decoder <NUM> of <FIG>, and decoder <NUM> of <FIG>. Decoder <NUM> illustrates a high-level overview of an embodiment of the present disclosure of an electronic device that decodes the compressed point cloud of the encoder <NUM>. Decoder <NUM> includes a demultiplexer <NUM>, video decoders <NUM> and <NUM>, a valid point selector <NUM> and <NUM>, and a binary mask reconstruction <NUM>. The input is the same or similar bitstream <NUM> from encoder <NUM> and the output is the generated point cloud <NUM>.

In certain embodiments, point cloud <NUM>, the point cloud <NUM> of <FIG>, the point cloud <NUM> of <FIG>, the point cloud <NUM> of <FIG> are the same. Mapping <NUM> maps the point cloud <NUM> into multiple frames. Each frame is then individually encoded. For example, video encoder <NUM> encodes the geometry attributes, such as X, Y, and Z. Video encoder <NUM> represents multiple video encoders, that encode the various attributes such as color, texture, normal and the like. In certain embodiments, the point cloud <NUM> is subsampled prior to mapping <NUM> to reduce the data size. In certain embodiments, the frames generated by the mapping <NUM> can be subsampled to reduce the data size of each frame.

In certain embodiments, if the point cloud <NUM> is uniformly sparse then the geometry values are downscaled by a predetermined factor in order to move each point closer to each other. By downscaling the density of points is increased. The predetermined downscaling factor can be added to the bitstream as metadata.

In certain embodiments, if the point cloud <NUM> is non-uniformly sparse, such that there are portions of the point cloud <NUM> that are denser than other parts, then the geometry values are downscaled regionally by different factors. By regionally downscaling the point cloud <NUM> by different factors to make the point cloud evenly dense in all regions. The downscaling factors can be added to the bitstream as metadata.

In certain embodiments, the various attributes of the point cloud <NUM> can be stored in a video frame by using padding or copying the edge pixels of the region used to store the point cloud attributes.

In certain embodiments, the size of each video frame is adaptively modified based on the size and shape of the unfilled space in the video frame. For example, each video frame can have a present initial size. Once the point cloud is projected onto the video frame, the frames can be expanded or reduced to fit the projection. In certain embodiments, the size of each frame is in multiples of eight.

The binary mask compression <NUM> is similar to the occupancy map generator <NUM> of <FIG>. The binary mask compression <NUM> compresses a binary image of the frame. In certain embodiments, the mask information generated by the binary mask compression <NUM> represents the start pixel location number and the end pixel location number for each line of the video frame.

Each encoded video frame and the binary mask is combined by multiplexer <NUM>. The multiplexer <NUM> generates the compressed bitstream <NUM>.

The decoder <NUM> extracts the geometry and attribute values from the bitstream <NUM> by demultiplexer <NUM>. The demultiplexer <NUM> splits the bitstream <NUM> into the various compressed frames. The geometry bitstream is decoded by video decoding <NUM>. The various attributes are decoded by video decoding <NUM>. In certain embodiments, video decoding <NUM> represents multiple video decoders each to decode a particular attribute such as color, texture, normal, and the like.

Demultiplexer <NUM> also splits the bitstream <NUM> into the binary mask reconstruction <NUM>. The binary mask reconstruction is the binary mask that indicates valid and invalid points on each frame. The valid point selector <NUM> analyzes the binary mask and selects valid points of the geometric frame. Similarly, valid point selector <NUM> analyzes the binary mask and selects valid points of each attribute frame. The decoder <NUM> then generates a reconstructed point cloud <NUM>.

In certain embodiments, not forming part of the claimed invention, additional mappings and frame generation techniques can be used. For example, in examples not forming part of the claimed invention, the encoder, such as encoder <NUM>, <NUM>, <NUM>, or <NUM>, uses a projection to fill a single video frame continuously with a single point cloud attribute. The encoder then generates and sends metadata along with the bitstream that indicates the type of the attribute stored in the particular video frame. If the size of the frame is larger than the number of points in the point cloud, then the encoder partially fills the frame with the valid attribute data and leaves the unfilled pixels are empty. The empty pixels can be filled with zeros or left as a default value. The decoder, such as decoder <NUM>, <NUM>, <NUM>, or <NUM>, receives the bitstream from the respective encoder. The decoder can use the generated metadata to determine the type of attribute stored in the decoded frame to output the attribute appropriately. Example syntax can include "unsigned char attr_type;" to indicate the attribute type. The example syntax of "unsigned short int attr_num_pts;" can be used to indicate the number of points of the particular attribute. The example syntax of "unsigned short int unfil_val;" can be used to indicate the default value of unfilled pixels within a particular frame.

In another example, not forming part of the claimed invention, the encoder, such as encoder <NUM>, <NUM>, <NUM>, or <NUM>, uses a projection to fill a video frame with point cloud data and sends metadata indicating the type of attribute stored in the particular frame along with the number of valid points. The decoder, such as decoder <NUM>, <NUM>, <NUM>, or <NUM>, receives the bitstream from the respective encoder. The decoder uses the metadata to determine the type of number of attributes stored in the decoded frame in order to output attribute appropriately.

In certain embodiments, it is possible for the number of points in a particular point cloud to exceed the maximum number of pixels that can be stored in a single frame. In this situation, the point cloud attributes can be spread over multiple frames, and metadata can be generated to indicate the attribute or portion of an attribute that is stored in each frame.

In certain embodiments, it is possible for the number of points in a particular point cloud to be less than the total number of pixels in a generated frame. For example, in this situation, the pixels in the unfilled frame that are left empty can be filled with zeros or another default value. In another example, in this situation, the frame can be filled with multiple attributes. <FIG>, below illustrates multiple attributes stored in a single video frame. Metadata can be generated to indicate the default values of empty pixels or indicate the multiple attributes that are stored in the frame.

<FIG> illustrates a binary mask in accordance with an embodiment of this disclosure. The embodiment of <FIG> is for illustration only and other embodiments could be used without departing from the scope of this disclosure. <FIG> illustrates frame <NUM>, frame <NUM>, and frame <NUM>.

Frame <NUM> illustrates a video frame of the geometric coordinates Xn of a point cloud. Frame <NUM> represents a binary mask which contains mask 904A that indicates the portion of frame <NUM> that contains valid geometric coordinates. The invalid area 904B indicates pixels that do not contain any valid point cloud data. Frame <NUM> indicates frame <NUM> overlaid with frame <NUM>. The region that contains the point cloud coordinates are the location of valid geometric coordinates of the point cloud. The mask 904A can be applied to all attributes in which the valid data are stored at the same location in the <NUM>-D frames.

In certain embodiments, the binary mask is compressed and transmitted with the bitstream. For example, the binary mask can be compressed to an array of <NUM>-bit integers, where each number is the length of each row.

<FIG> illustrates an example raster scan mapping in accordance with an embodiment of this disclosure not forming part of the claimed invention. The embodiment of the raster scan according to <FIG> is for illustration only and other embodiments could be used without departing from the scope of this disclosure. <FIG> illustrates a point cloud <NUM>, a geometry (XYZ) frame <NUM> and a color (RGB) frame <NUM>. The geometry frame <NUM> is the mapping of each X, Y, and Z coordinate of the input point cloud <NUM> from the <NUM>-D space to a <NUM>-D plane using a raster scan. The color frame <NUM> is the mapping of each R, G, and B color coordinate of the input point cloud <NUM> from the <NUM>-D space to a <NUM>-D plane using a raster scan. In the raster scan method, points are read from the point cloud <NUM> in the order they are stored in the point cloud <NUM>. Then each attribute of the input point is mapped to the corresponding <NUM>-D frame line by line. It is noted that point cloud <NUM> is similar to the point cloud <NUM> of <FIG>, <NUM> of <FIG>, <NUM> of <FIG>, and <NUM> of <FIG>.

For instance, the geometry frame <NUM> can be designated as f<NUM>, the color frame <NUM> can be designated as f<NUM>, the normal texture frame <NUM> can be designated as f<NUM>, and the texture coordinate frame <NUM> can be designated as f<NUM>. The four frames can then be identified by f<NUM>(i, j, k), f<NUM>(i, j, k), f<NUM>(i, j, k), and f<NUM>(i, j, k). Each mapped frame can have one, two, or three components in each respective video encoder <NUM>-<NUM>. For example, variable k can represent the different components of f(i, j, k,). For instance, if the there are three components the frame is f(i, j, <NUM>), f(i, j, <NUM>), and f(i, j, <NUM>). If there are two components the frame is f(i, j, <NUM>) and f(i, j, <NUM>). If there is one component the frame is f(i, j, <NUM>), or simply f(i, j). For example, if the four mappings are indicated by M<NUM>(), M<NUM>(), M<NUM>(), and M<NUM>() the following equation can define the mapping:
Equation: <MAT>.

Based on the above equation, a raster scan can yield the following equations that define each frame:
Equations: <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> <MAT> <MAT>.

Based on the above equations, for the n-th point of a point cloud, the geometry components (Xn, Yn, Zn) and an attribute component, such as color (Rn, Gn, Bn) can be mapped onto individual frames. For example, geometry components can be mapped onto a G frame and the color components can be mapped onto a C frame. The projection of n-th point of the geometry frame is illustrated in frame <NUM> as (i) G(n/w, n%w, <NUM>) = Xn; (ii) G(n/w, n%w, <NUM>) = Yn; and (iii) G(n/w, n%w, <NUM>) = Zn. Similarly, the projection of n-th point of the color frame is illustrated in frame <NUM> as (i) C(n/w, n%w, <NUM>) = Rn; (ii) C(n/w, n%w, <NUM>) = Gn; and (iii) C(n/w, n%w, <NUM>) = Bn.

<FIG>, and <FIG> illustrate an example of row-wise unwrapping in accordance with an embodiment of this disclosure. <FIG> illustrates a point cloud <NUM> located on a Cartesian coordinate system. <FIG> illustrates cross sections of a point cloud that is mapped in two dimensions. In particular, <FIG> illustrates a cross section of a point cloud in the XZ plane for a particular Y value. <FIG> illustrates the mapping of a point cloud onto individual frames. The embodiment of the row-wise unwrapping according to <FIG> are for illustration only and other embodiments could be used without departing from the scope of this disclosure.

<FIG> illustrates a point cloud <NUM> located on a Cartesian coordinate system of the X-axis <NUM>, the Y-axis <NUM> and the Z-axis <NUM>. The point cloud <NUM> is similar to the point cloud <NUM> of <FIG>, <NUM> of <FIG>, <NUM> of <FIG>, <NUM> of <FIG>, and <NUM> of <FIG>. Plane <NUM> scans the point cloud <NUM> along with a single axis. As illustrated plane <NUM> scans the point cloud <NUM> along the Y-axis <NUM>. In certain embodiments, plane <NUM> can scan the point cloud <NUM> along the X-axis <NUM> or the Z axis <NUM>. For example, the point cloud <NUM> is scanned along the Y axis <NUM> to find the X and Z points along the X-axis <NUM> and the Z axis <NUM>, respectively. In certain embodiments, the point cloud <NUM> is scanned along the longest axis. In certain embodiments, the encoder <NUM>, <NUM>, <NUM>, or <NUM> can determine the longest axis and scan along that axis. In certain embodiments, the encoder <NUM>, <NUM>, <NUM>, or <NUM> scans the point cloud <NUM> along with a predetermined axis regardless of the size or shape of the point cloud <NUM>.

<FIG> illustrates a graph <NUM> that depicts a cross-section of the point cloud <NUM> in the XZ plane for a particular Y plane. Then, the X and Z values of the cross-section points are stored in the y-th rows of the first and second components of one frame of a video (or one image). Similarly, the R, G, and B color components of the cross-section points are written in the y-th rows of the first, second, and third components of one frame of another video. Every other or additional point cloud attribute can also be stored either in one frame of a separated video or in a different row of the same video used for other attributes.

In certain embodiments, the cross section points can be sorted and reordered. For example, sorting techniques such as a raster scan (as discussed with reference to <FIG>) a zig-zag sorting, a depth-first search (DFS), a breath-first search (BFS), and the like can be utilized to sort and reorder the cross section points. Sorting the cross section points such as <NUM> or <NUM> can increase efficiency when the mapped point cloud is transmitted. In certain embodiments, sorting is applied to different connected components of the cross section image separately.

<FIG> illustrates the mapping of a point cloud <NUM> onto individual frames such as the geometry frame <NUM>, the color frame <NUM>, and a binary mask frame <NUM>. The geometry frame <NUM> illustrates the entirety of the point cloud <NUM> that is scanned along the Y axis <NUM>. The geometry frame <NUM> illustrates the entirety of the cross section points such as <NUM> and <NUM> along the entirety of the Y axis <NUM>. For example, as the plane <NUM> moves along the Y axis <NUM>, multiple graphs similar to graph <NUM> are generated and mapped to video frames line by line to create geometry frame <NUM>. The geometry frame <NUM> illustrates the X and Z points of the point cloud <NUM> mapped along the Y axis <NUM>. Similarly, the color frame <NUM> represents the colors R, G, B that correspond to each point of the geometry frame <NUM>. Binary mask frame <NUM> illustrates the valid and invalid pixels of the point cloud. The binary mask frame <NUM> is similar to the binary mask of <FIG>. The binary mask frame <NUM> indicates each invalid pixel to allow a decoder to correctly reconstruct and generate the point cloud. In certain embodiments, additional frames are generated to indicate different attributes such as texture, normal, texture coordinates, and the like.

An example syntax is provided below to illustrate the mapping of the point cloud <NUM> into the graph <NUM> for a single cross section location and the geometry frame <NUM>, the color frame <NUM> and the binary mask frame <NUM> along the entirety of a single axis. The example syntax below is described with respect to the longest dimension of the point cloud <NUM> which is the Y axis <NUM>.

In the above example, the syntax illustrates mapping a point cloud along the longest dimension. The Geo = memset(width,height) assigns memory for the geometry frame. Similarly, the Col = memset(width,height) assigns memory for the geometry frame. The [X,Y,Z,R,G,B,. ] = read_PC(file_name) reads a particular point cloud, such as point cloud <NUM>, from a particular file. The syntax for ( i = <NUM> to max(Y) ) provides instructions to scan the point cloud along the axis from a starting at a point and ending at a maximum point. The [X0,Z0,R0,G0,B0,. ] = get_Cross_Sec_Im(i) provides instructions to the get cross-section points in the XZ plane for Y = i. The [X1,Z1,R1,G1,B1,. ] = sort(X0,Z0,R0,G0,B0,. ); is an optional step that applies a sorting algorithm to each row of the scan. The Geo(i,:,<NUM>) = X1; Geo(i,:,<NUM>) = Z1 provides instructions to write data into one row of the geometry image. The Col(i,:,<NUM>) = R1; Col(i,:,<NUM>) = G1; Col(i,:,<NUM>) = B1 provides instructions to write data into i-th row of color image. Similar syntax is applied for each additional attribute.

In the above syntax the geometry component only stores two components, the X and the Z values while the Y value of each point is implicitly encoded as the row index. This approach of mapping each cross section to one video line may result in projected video frames with large widths for some point clouds. In one embodiment, a smaller width can be utilized for a projected video by splitting the long rows into multiple rows. When long rows are split into multiple rows, the Y values can be written in one component of the projected video.

<FIG> illustrates example packing operations in accordance with an embodiment of this disclosure. In particular <FIG> illustrates multiple attributes stored in individual frame components. <FIG> illustrate an example of packing XYZ geometry components of a point cloud frame. <FIG> illustrate example frames with attributes stored in irregular regions. The embodiment of the various frame packing according to <FIG> are for illustration only and other embodiments could be used without departing from the scope of this disclosure.

<FIG> illustrates multiple attribute components stored in individual frame components. For instance, the first video component <NUM> (i.e. luma Y) of a single video frame stores Xn and Yn geometry components. Similarly, the second component <NUM> (i.e. chroma Cr) stores Zn and Rn components. Similarly, the third video component <NUM> (i.e. chroma Cb) stores Gn and Bn components. In certain embodiments, various components of the geometry and other point cloud attributes can be stored in three component of the same video frame.

<FIG> illustrate an example of packing all XYZ geometry components of a point cloud frame into the luma component of a <NUM>-D video frame <NUM>, and <NUM> and fill the two chroma component frames <NUM>, <NUM>, <NUM>, and <NUM> with <NUM>, another default value, or another attribute's mapping data to support <NUM>:<NUM>:<NUM>, <NUM>:<NUM>:<NUM> or other chroma subsampling formats in which the chroma components are subsampled. For example, chroma components frames <NUM>, <NUM>, or chroma components frames <NUM>, <NUM> can include an attribute or a default value such as zero. In <FIG> the XYZ components are packed horizontally, whereas in <FIG> the XYZ components are packed vertically.

<FIG> illustrate the attributes stored in irregular regions which could be the same for all video frames or different for some of the video frames. <FIG> illustrates individual geometry components mapped onto individual <NUM>-D frame components such as frames <NUM>, <NUM>, and <NUM>. In certain embodiments, frames <NUM>, <NUM>, and <NUM> illustrate multiple video frames or multiple components of the same video frame. <FIG> illustrates an individual geometry attributes mapped onto a single <NUM>-D frame component such as <NUM>. In certain embodiments, a binary mask is used to indicate locations of valid points. A binary mask can indicate that the gray pixels of frame <NUM> are valid pixels whereas white pixels of frame <NUM> are invalid pixels.

In certain embodiments, the RGB color attribute can be converted from RGB to YCbCr, YCoCg, or other video formats prior to encoding. When the decoder, such as decoder <NUM>, <NUM>, <NUM>, or <NUM>, decodes the frame containing the color attribute, the decoder can convert the various color formats back into the RGB format.

As described above, <FIG>, and <FIG> illustrate how geometry and attributes can be stored in three components of a single video frame. For example, geometry frame <NUM> illustrates two component's, that of X and Y that are stored in a single frame. Color frame <NUM> illustrates three component's, that of R, G, and B that are stored in a single frame. Similarly, multiple attributes can be stored in a single frame, such as the frames of <FIG>, and <FIG>. For example, a particular frame can be designated into multiple regions that are separated by a bounding box. A bounding box is a box that is defined by the top left pixel and width and height of the bounding box. The top left pixel indicates where in the frame the second region starts and is measured by the height and width of the bounding box. The example syntax below can be used to indicate a bounding box when multiple attributes are positioned in a single frame. The syntax of unsigned short int attr_box_left;" can be used to indicate the top index of the top left portion of a bounding box. The syntax of "unsigned short int attr_box_top;" can be used to indicate the top index of the top-left point of the bounding box. The syntax of "unsigned short int attr_box_width;" can be used to indicate the width of attribute's bounding box. The syntax of "unsigned short int attr_box_height;" can be used to indicate the height of attribute's bounding box.

<FIG> illustrates an example flowchart for encoding a point cloud in accordance with an embodiment of this disclosure. <FIG> depicts flowchart <NUM>, for point cloud encoding. For example, the process depicted in <FIG> is described as implemented by the server <NUM> of <FIG>, any one of client devices <NUM>-<NUM> of <FIG>, the server <NUM><NUM> of <FIG>, the electronic device <NUM> of <FIG>, encoder <NUM> of <FIG>, encoder <NUM> of <FIG>, encoder <NUM> of <FIG> or encoder <NUM> of <FIG>.

The process begins with an encoder, such encoder <NUM>, <NUM>, <NUM>, or <NUM> generating multiple <NUM>-D frames from a point cloud (<NUM>). The two-dimensional frames include at least a first frame representing a geometry of points in the three-dimensional point cloud. The two-dimensional frames also include a second frame representing a texture of points in the three-dimensional point cloud such as color, texture, normal, reflectace, and the like. In certain embodiments, the location of each point in the three-dimensional point cloud is mapped onto the first frame to indicate the geometry of points in the three-dimensional point cloud. In certain embodiments, the texture attribute of each point in the three-dimensional point cloud is mapped onto the second frame to indicate the texture of points in the three-dimensional point cloud.

In certain embodiments, the processor is further configured to analyze the three-dimensional point cloud at a plurality of positions along a first axis in order to store each point in the three-dimensional point cloud along a second and third axis that correspond to each position along the first axis. Each point is stored in row of the first frame that corresponds the position along the first axis. In certain embodiments, the first axis is the longest dimension of the three-dimensional point cloud.

The process generates an occupancy map that indicates locations of pixels on the <NUM>-D frames (<NUM>). In certain embodiments, the occupancy map is similar to the binary mask that indicates valid points on each frame. For example, the generated occupancy map can overlay each frame to indicate whether each pixel is part of the point cloud.

The process also encodes the <NUM>-D frames and the occupancy map to generate a compressed bitstream (<NUM>). The process combines each frame and the occupancy map to compress each object in order to generate the bitstream. In certain embodiments, the encoded frames are compressed using a video or image codec. The process then transmits the compressed bitstream (<NUM>).

<FIG> illustrates an example flowchart for decoding a point cloud in accordance with an embodiment of this disclosure. For example, the process depicted in <FIG> is described as implemented by any one of the client devices <NUM>-<NUM> of <FIG>, the electronic device <NUM> of <FIG>, decoder <NUM> of <FIG>, decoder <NUM> of <FIG>, decoder <NUM> of <FIG> or decoder <NUM> of <FIG>.

The process begins with a decoder, such decoder <NUM>, <NUM>, <NUM>, or <NUM> receiving a compressed bitstream (<NUM>). The received bitstream can include an encoded point cloud that was mapped into two-dimensional frames that was compressed for transmission.

The process decodes the compressed bitstream into <NUM>-D frames and a occupancy map (<NUM>). In certain embodiments the occupancy map is not included in the bitstream. The two-dimensional frames include at least a first frame representing a geometry of points in a three-dimensional point cloud. The two-dimensional frames also include a second frame representing a texture of points in the three-dimensional point cloud. In certain embodiments, a first series of frames, each representing the geometry of points, is decompressed to generate the first frame that includes the geometry of points. Similarly, a second series of frames, each representing the texture of points of the three-dimensional point cloud is decompressed to generate the second frame that includes the texture of points.

The process then identifies or decodes an occupancy map that is included in the decoded bit stream (<NUM>). The occupancy map indicates locations of pixels in the two-dimensional frames that represent each point in the three-dimensional point cloud. In certain embodiments, the occupancy map is similar to the binary mask that indicates valid points on each frame. For example, the occupancy map can overlay each frame to indicate whether each pixel is part of the point cloud.

The process then generates the three-dimensional point cloud using the occupancy map bsed on the multiple frames (<NUM>). In certain embodiments, the location of each point of the first frame is mapped in order to reconstruct the geometry of points in a three-dimensional point cloud. Similarly, the texture attribute of each point in the second frame is mapped in order to reconstruct the texture of points in the three-dimensional point cloud. In certain embodiments, the texture attribute of each point is mapped based on the mapped location of each point that indicates the geometry of points in three-dimensional point cloud.

In certain embodiments, the process analyzes each row of the first frame to identify a first axis. Each point that is identified is positioned in each row that corresponds a point of the three-dimensional point cloud at a position along the first axis. The three-dimensional point cloud is generated by positioning each point at a second axis and a third axis that corresponds to each row of the first frame. In certain embodiments, the process identifies the longest dimension as the first axis.

Although the figures illustrate different examples of user equipment, various changes may be made to the figures. For example, the user equipment can include any number of each component in any suitable arrangement. In general, the figures do not limit the scope of this disclosure to any particular configuration(s). Moreover, while figures illustrate operational environments in which various user equipment features disclosed in this patent document can be used, these features can be used in any other suitable system.

Claim 1:
A decoding device (<NUM>; <NUM>; <NUM>; <NUM>; <NUM>) for point cloud decoding, the decoding device comprising:
a communication interface configured to receive a compressed bitstream (<NUM>); and
a processor (<NUM>; <NUM>) coupled with the communication interface and configured to:
decode (<NUM>) the compressed bitstream into two-dimensional frames (<NUM>, <NUM>; <NUM>, <NUM>, <NUM>, <NUM>) and an occupancy map (<NUM>), wherein the two-dimensional frames include at least a first frame (<NUM>; <NUM>) representing a geometry of points in a three-dimensional point cloud and a second frame (<NUM>; <NUM>, <NUM>, <NUM>) representing an attribute of points in the three-dimensional point cloud;
identify (<NUM>) the occupancy map indicating locations of valid and invalid pixels in each of the two-dimensional frames that represent each point in the three-dimensional point cloud, wherein the locations of valid and invalid pixels are indicated based on overlaying the occupancy map with each of the two-dimensional frames; and
generate (<NUM>) from the two-dimensional frames the three-dimensional point cloud (<NUM>; <NUM>; <NUM>; <NUM>) based on the locations of the valid pixels in each two-dimensional frame as indicated by the occupancy map,
characterised in that each row of the first frame corresponds to a cross section of the three-dimensional point cloud for a first axis and includes geometry values of cross-section points, and
each row of the second frame corresponds to the cross section and includes attribute values of the cross-section points.