Patent ID: 12249019

DETAILED DESCRIPTION

FIG.1Adepicts an example method100for rendering a 3D image, in accordance with some embodiments of this disclosure.

At step102, rendering application of a rendering system receives image data. For example, image data may comprise a frame of a video (e.g., 3D video) or other media content (e.g., 3D video game, 3D video conference, a 3D virtual or digital representation of a real-world space, etc.). In some embodiments, image data may include a texture matrix120and depth matrices122for the image data. For example, the texture matrix may be a table comprising luminance and/or color (e.g., RGB) matrix, where each cell in the matrix represents brightness (or luminance) and color of a single pixel. The depth matrix may be a matrix (of the same size as the texture matrix) that defines depth for each of the pixels in the texture matrix. The term matrix may refer to any data that is formed as an array of values of any arbitrary size.

In some embodiments, at step104, the rendering application forms a focal control matrix124of the image data. In some embodiments, the control matrix is based on a certain number of MSB of the depth matrix122. For example, if 2 MSBs are used, bits other than 2 MSBs may be changed to zeros. In one example, depth bit value “1111 0101” would become “1100 0000,” depth bit value “0101 1101” would become “0100 0000” (e.g., bits other than leading two bits would be set to zero). While the matrices with 8 bits of data in each cell are provided here as an example, one skilled in the art would appreciate that any size of data may be processed in a similar way. In one example, the matrices may have 16 bits of data in each cell. In another example, the matrices may have 32 bits of data in each cell. In yet another example, the matrices may have 2nbits of data in each cell where n is any integer. In the exemplary description of the procedure shown inFIG.1A, the convention for representing a depth matrix is that small values correspond to long distances. In some embodiments, the exemplary algorithm ofFIG.1Amay be primed with an opposite convention (i.e., that a small distance shows up as a small value).

In some embodiments, at step106, the rendering application forms two weight matrices based on a certain number of LSBs (e.g., 6 or 5) of the depth matrix122. In one approach, the number of LSBs is residual to the number of MSBs used for the focal control matrices124. That is, for 8-bit matrices, if 2 MSBs were used for a control matrix, 6 LSBs are used for the weight matrix; if 3 MSBs were used for a control matrix, 5 LSBs are used for the weight matrix, etc.

In some embodiments, the first weight matrix126is based on a certain number of LSB of the depth matrix122. For example, if 6 LSBs are used, bits other than 6 LSBs may be changed to zeros. In one example, depth bit value “0111 0010” would become “0011 0010”, depth bit value “1100 0110” would become “0000 0110” (i.e. the leading two bits would be set to zeros).

In some embodiments, the second weight matrix128is a complement of the first weight matrix126. In some embodiments, the term “complement” to matrix may herein refer to a counterpart matrix to the initial matrix, such that the sum of the matrix and a complement matrix results in a matrix with a value of “1” in each position. In some embodiments, the term complement matrix may refer to one's complement matrix. In some embodiments, the term complement matrix may refer to 255's complement matrix (e.g., for 8-bit value matrix). In some embodiments, the term complement matrix may refer to (2{circumflex over ( )}n)−1 complement matrix where n is any integer (e.g., for n-bit value matrix). For example, to generate the second weight matrix128, the rendering application may subtract each value of the first weight matrix126from bit value “1111 1111” (255 in decimal). For example, value depth bit value “0111 0010” (114 in decimal) in the first weight matrix126would become 1111 1111-0111 0010=0100 01101 (141 in decimal). Similar inversion may be performed for data larger or small than 8 bits (e.g., subtraction from 1111 1111 1111 1111 may be used for data bit-length 16).

As mentioned above, any split between MSB and LSB of an 8-bit data filed can be used. The use of different splits results in different effective focal planes, as shown in Table 1 below.

TABLE 1Resulting No. ofMSBsLSBsFocal Planes (MFP)08117226535944175333626571129801

One skilled in the art would appreciate that similar splits into LSBs and MSBs can be done for any bit sizes of data fields in the depth matrix122.

In some embodiments, at step108, the rendering application applies the weights matrices126,128to the texture matrix120. For example, luminance values of the texture matrix120may be (e.g., multiplied) to corresponding values of the weight's matrix126to create focal plane matrix136. In parallel, luminance values of the texture matrix120may be scaled (e.g., multiplied) to corresponding values of the weight's matrix128to create focal plane matrix138. In some embodiments, if the texture matrix120includes both luminance value and color values, only the luminance values are weighted, while color values are retained. For example, the luminance values may be extracted, weighted using the weight matrices126,128, and recombined with the color values. The scaling may also involve initial division by 255 (or other largest value for given data bit size) before multiplication by the weight value (e.g., to keep the final value in 0-255 range).

In some embodiments, at step110, the rendering application stores the formed matrices136and138as two focal plane matrices.

In some embodiments, at step112, the rendering application optically projects the two folded focal plane matrices136,138to generate a 3D image for the input data120,122. For example, light based on the two folded focal plane matrices (e.g., tiled together) may be projected into SLM device that is configured using the control matrix124(e.g., as further discussed inFIG.4). In some embodiments, the rendering application may use systems and methods described in the '793 application to generate a 3D images using the two folded focal planes matrices136,138. In another example, two focal plane matrices may be used to generated pixels on respective displays of a display device with two stacked displays.

FIG.1Bdepicts an example method for reducing a number of focal planes for display of a three-dimensional image, in accordance with some embodiments of this disclosure. In particular,FIG.1Bdepicts an example method for reducing a number of focal planes for display of a three-dimensional image. At step151, three or more multiple focal planes (MFPs) for representing a three-dimensional image are received. The focal planes may be implemented using a stack of displays, with each display corresponding to one of the planes. The MFPs may be generated from data representing a three-dimensional image (e.g., a 3D frame of 3D media content) using a depth-blending algorithm. In some embodiments, the MFPs are received from an external computing device.

MFPs150comprise a graphical representation of a cross-section of a three-dimensional image displayed across a plurality of focal planes. Curve152depicts the intended appearance of focal depths of the cross-section of the three-dimensional. Each section of curve152is represented by pixel values on one or more focal planes. For instance, the first section of curve152is generated from pixel luminance values on the third and fourth focal planes from the front (bottom) with the location of a pixel of the final image being dependent on a magnitude of the luminance values of that pixel in the third focal plane and the fourth focal plane. In some instances, a pixel luminance value may only exist on one focal plane, such as at the locations where the line crosses focal planes. In some instance, a pixel luminance value may exist across more than two focal planes, such as when a depth blending algorithm that is used to generate the MFPs blends a pixel value across more than two focal planes.

At step153, the first plurality of focal planes is mapped to a second plurality of folded focal planes. The second plurality of folded focal planes to which the first plurality of focal planes comprises at least one less focal plane than the first plurality of focal planes. In some embodiments, the first plurality of focal planes is mapped to a total of two folded focal planes, such as when pixel luminance values for any given pixel location only exist on a maximum of two focal planes.

FIG.1Bdepicts an implementation where five focal planes (e.g., as received at step151) are mapped down to two folded focal planes (e.g., matrices136,138). Each portion of the curve152is represented in the five focal planes150as pixel luminance values on a maximum of two folded focal planes. For each pixel location with pixel luminance values in two focal planes, the pixel luminance value of the backmost folded focal plane is mapped to the first of mapped focal planes154and the pixel luminance value of the frontmost folded focal plane is mapped to the second of the mapped focal planes.

At step153, each pixel location with a pixel luminance value in only one of the focal planes, the pixel luminance value is mapped to one of the backmost folded focal plane or frontmost folded focal plane according to some embodiments. In some approaches, the backmost folded focal plane or frontmost folded focal planes154are generated as described with respect toFIG.1Abased on focal planes data150.

At step155, focal map data, i.e., pixel location data identifying focal distances of mapped pixels is stored. For example, the rendering application may store, for each mapped pixel, data identifying a focal plane from which the pixel was mapped. Graph156(a pixel row of a control matrix124) depicts a graph that of pixel location values that identify a focal plane from which the pixels were mapped, starting at focal plane 0 and ending at focal plane 4. While pixel location values are depicted as identifying the focal plane, in other embodiments the pixel location values may comprise the focal distance of the mapped focal plane. Alternatively, the rendering application may store data identifying, for each focal plane, the focal distance of the focal plane. WhileFIG.1Cdepicts the pixel location values as a graph, in some embodiments the pixel locations values are stored in a table and/or integrated into a phase function equation for rendering a three-dimensional image.

In addition to all shifted depth blended segments shown inFIG.1B, focal distances (focal/rendering map) of the original MFP stack are stored to be used by the rendering application. Exemplary focal control data (focal map) is illustrated in chart156. This data may be used by rendering application when re-rendering the two FFPs ofFIG.1B, e.g., as shown by arrows in charts150and154. Accordingly, a focal map may refer to an image that contains pixel-by-pixel information of the focal plane that each FFP pixel originates from.

FIG.1Cdepicts an example of depth blending techniques, in accordance with some embodiments of this disclosure.

For example, in a case where there are only two planes (L1and L2) that generate blended voxels, the depth blending160between the two focal planes may be determined according to the following equation:
L0=L1(z)+L2(z)=w1(z)L0+w2(z)L0
where w1and w2are depth-weighted fusing functions. The perceived depth z{circumflex over ( )} of the fused voxel may be considered as a function of the depth-weighted fusing functions: z{circumflex over ( )}=f(w1, w2), where a simple approximation function may be the weighted sum of the depths of the two focal planes: z{circumflex over ( )}=w1(z)z1+w2(z)z2. The meaning of symbols if the equation above are demonstrated byFIG.1C. In particular, A refers to aperture of the eye, L1and L2refer to planes for generating blended voxels at distances z1and z2, L0refers to appearance point of the blended voxel at distance z. The symbol Ω refers to view angle from the eye through distances z1and z2.

FIG.2depicts an example method200for generating weights for texture matrix, in accordance with some embodiments of this disclosure. In some embodiments, weights generated as shown inFIG.2apply to initial input120,122ofFIG.1A. For example, matrix204may be the same depth matrix as matrix122, where the depth matrix provides depth data for respective pixels of texture data120.

Diagram205shows a histogram of depth values from matrix204. For example, depth value may vary from 0-255 and, the histogram shows how often each such depth value occurs. The sawtooth line shows a weight for respective pixel of texture data (e.g., texture matrix120) based on the depth value. As shown, four periods are provided where the weight increase in a linear fashion, before restarting. However, in other embodiments, any number of such periods may be used (e.g., more periods to achieve more aggressive folding and less periods of less aggressive folding). The weights shown in diagram205can be applied to the texture matrix to form first focal plane matrix.

Diagram206also shows a histogram of depth values from matrix204. For example, depth value may vary from 0-255 and, the histogram shows how often each such depth value occurs. The sawtooth line (which is an inverse of sawtooth line in diagram205) shows a weight for respective pixel of texture data (e.g., texture matrix120) based on the depth value. As shown, four periods are provided where the weight increase in a linear fashion, before restarting. However, in other embodiments, any number of such periods may be used (e.g., more periods to achieve more aggressive folding and less periods of less aggressive folding). The weights shown in diagram205can be applied to the texture matrix to form second focal plane matrix.

While the sawtooth function weight values can be applied directly to texture matrix, an alternative approach is shown in diagram208. By removing first 2 MSBs (in little endian notation), the depth values become folded in range that is one quarter of the initial range. The weight values (shown by two lines) then become linear and are complements of each other. However, application of such weights (shown by two lines) to texture data would result in similar adjustment as application of saw-toothed function weights shown in diagrams205and206. As the result of the respective application of the weights (shown by two polylines on diagrams205and206) to texture data, two focal planes matrices are generated which can be used to project a 3D image e.g., as shown in element140ofFIG.1Aand inFIG.4discussed below.

FIG.3depicts pseudocode300for generation of two focal plane matrices, in accordance with some embodiments of this disclosure. For example, the pseudocode may be used by rendering application to generate (when executed by a processor) two focal planes matrices that are used to project a 3D image e.g., as shown in element140ofFIG.1Aand inFIG.4below.

At line1101, the rendering application receives as input Texture Matrix (e.g., Texture_Matrix) and Depth Matrix (e.g., Depth_Matrix). In some embodiments, Depth Matrix may be converted to gray values.

At line1105, the rendering application initializes first weight matrix (e.g., weight_matrix_1) with the same size as Depth Matrix. At line1106, the rendering application initializes second weight matrix (e.g., weight_matrix_1) with the same size as Depth Matrix.

At lines1108-1111, the rendering application iterates through each value of the Depth Matrix (by iterating via length and width) to set respective values of the first Weight Matrix to a predetermined number (e.g., 6) of last (when the binary numbers are stored in little endian notation) LSBs of the data in depth matrix.

At line1111, the rendering application sets each of the values of the second weight matrix to complements of the respective value in the first weight matrix (e.g., by subtraction from 255 for 8-bit data).

At line1115, the rendering application initializes control matrix (e.g., Control_matrix) with the same size as Depth Matrix.

At lines1117-1119, the rendering application iterates through each value of the Depth Matrix (by iterating via length and width) to set respective values of the control matrix to a predetermined number (e.g., 2) of first (when the binary numbers are stored in little endian notation) MSBs of the data in depth matrix.

At line1123-1124, the rendering application initializes first and second focal matrices (e.g., focal_matrix_1 and focal_matrix_2) with the same size as Texture Matrix.

At line1126, the rendering application sets weights of the first focal matrix (e.g., focal_matrix_1) to the values in texture matrix, weighted by respective values in the first weight matrix.

In some embodiments, if the texture matrix was received in RGB format, the texture matrix may be converted to e.g. YCbCr format. Then luminance values of the YCbCr may be extracted and weighted by the respective values in the first weight matrix (e.g., by dividing each respective values by 255 and multiplying it by the weight to keep the result in 0-255 range). Then the luminance values may be re-combined with Cb (blue difference) and Cr (red difference) values, and the resulting weights YCbCr matrix may be converted back to RGB as the first Focal Matrix. Any other format that allows extraction of luminance values may be used instead of YCbCr.

At line1127, the rendering application sets weights of the second focal matrix (e.g., focal_matrix_2) to the values in texture matrix (e.g., Control_matrix), weighted by respective values in the second weight matrix.

In some embodiments, if the texture matrix was received in RGB format, the texture matrix may be converted to e.g. YCbCr format. Then luminance values of the YCbCr may be extracted and weighted by the respective values in the second weight matrix (e.g., by dividing each respective values by 255 and multiplying it by the weight to keep the result in 0-255 range). Then the luminance values may be re-combined with Cb (blue difference) and Cr (red difference) values, and the resulting weights YCbCr matrix may be converted back to RGB as the second Focal Matrix. Any other format that allows extraction of luminance values may be used instead of YCbCr.

At line1130, the rendering application outputs the first and second focal matrices and the control matrix. These matrices may be used to generate a 3D image as described inFIG.4below.

FIG.4depicts an example system for rendering an image using two folded focal plane matrices (e.g., two folded focal plane matrices136,138ofFIG.1or two focal matrices generated as output in line1130ofFIG.3), in accordance with some embodiments of this disclosure. In particularFIG.4depicts an example system400that can be used to render an image using pixel values. Display screen402comprises a display configured to display the mapped pixel values. As depicted inFIG.4, display screen402comprises two displays, one for each tiled mapped focal plane (e.g., folded focal planes136,138ofFIG.1). In implementations where the pixel values are mapped to more than two focal planes, display screen402may be configured to display the higher number of focal planes. Light emanating from display screen402is collimated by lens404. A spatial multiplexing unit (SMU, for example one or more SLMs)406functions as a multifocal off-axis Fresnel lens that adds quadratic and linear phase terms to an incident wavefront. The quadratic phase terms can axially shift pixels to the designated depths, while the linear phase terms can laterally shift the centers of pixels to the optical axis. As a result, the pixels of the sub-panel images can be mapped to different axial locations and laterally aligned at the output end. Using SMU technology for shifting portions of the image, the system can use the mapping data to determine where to shift each of the pixels and modify the SMU to shift the pixels to their original locations. The shift functions may be adjusted based on control matrix (e.g., matrix124or control matrix generated at line1130ofFIG.3).

The light that passes through SMU406is collimated by lens408to create optically mapped pixels410at the different depths. The final output image is collected by eyepiece412and displayed to a user's eyeball414. In some embodiments, a reflective SLM device may also be used to achieve the effect shownFIG.4.

FIGS.5-6depict illustrative devices, systems, servers, and related hardware for image encoding/decoding.FIG.5shows generalized embodiments of illustrative user equipment devices which may perform the functions described herein. User equipment device500may be a smartphone device, a tablet, a virtual reality or augmented reality device, or any other suitable device capable of processing video data. In another example, user equipment device501may be a user television equipment system or device. User television equipment device501may include set-top box515. Set-top box515may be communicatively connected to microphone516, audio output equipment (e.g., speaker or headphones514), and display512. In some embodiments, display512may be a television display, computer display or a head mounted display comprising a stack of displays. In some embodiments, display512may contain an SLM device as shown inFIGS.4A and4B. In some embodiments, set-top box515may be communicatively connected to user input interface510. In some embodiments, user input interface510may be a remote-control device. Set-top box515may include one or more circuit boards. In some embodiments, the circuit boards may include control circuitry, processing circuitry, and storage (e.g., RAM, ROM, hard disk, removable disk, etc.). In some embodiments, the circuit boards may include an input/output path.

Each one of user equipment device500and user equipment device501may receive content and data via input/output (I/O) path (e.g., circuitry)502. I/O path502may provide content (e.g., broadcast programming, on-demand programming, internet content, content available over a local area network (LAN) or wide area network (WAN), and/or other content) and data to control circuitry504, which may comprise processing circuitry506and storage508. Control circuitry504may be used to send and receive commands, requests, and other suitable data using I/O path502, which may comprise I/O circuitry. I/O path502may connect control circuitry504(and specifically processing circuitry506) to one or more communications paths (described below). I/O functions may be provided by one or more of these communications paths, but are shown as a single path inFIG.5to avoid overcomplicating the drawing. While set-top box515is shown inFIG.6for illustration, any suitable computing device having processing circuitry, control circuitry, and storage may be used in accordance with the present disclosure. For example, set-top box515may be replaced by, or complemented by, a personal computer (e.g., a notebook, a laptop, a desktop), a smartphone (e.g., device600), a tablet, a network-based server hosting a user-accessible client device, a non-user-owned device, any other suitable device, or any combination thereof.

Control circuitry504may be based on any suitable control circuitry such as processing circuitry506. As referred to herein, control circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer. In some embodiments, control circuitry may be distributed across multiple separate processors or processing units, for example, multiple of the same type of processing units (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). In some embodiments, control circuitry504executes instructions for the rendering application stored in memory (e.g., storage508). Specifically, control circuitry504may be instructed by the rendering application to perform the functions discussed above and below. In some implementations, processing or actions performed by control circuitry504may be based on instructions received from the rendering application.

In client/server-based embodiments, control circuitry504may include communications circuitry suitable for communicating with a server or other networks or servers. The rendering application may be a stand-alone application implemented on a device or a server. The rendering application may be implemented as software or a set of executable instructions. The instructions for performing any of the embodiments discussed herein of the rendering application may be encoded on non-transitory computer-readable media (e.g., a hard drive, random-access memory on a DRAM integrated circuit, read-only memory on a BLU-RAY disk, etc.). For example, inFIG.5, the instructions may be stored in storage508, and executed by control circuitry504of a device500.

In some embodiments, the rendering application may be a client/server application where only the client application resides on device500(e.g., device104), and a server application resides on an external server (e.g., server604and/or server616). For example, the rendering application may be implemented partially as a client application on control circuitry504of device500and partially on server604as a server application running on control circuitry611. Server604may be a part of a local area network with one or more of devices500or may be part of a cloud computing environment accessed via the internet. In a cloud computing environment, various types of computing services for performing searches on the internet or informational databases, providing encoding/decoding capabilities, providing storage (e.g., for a database) or parsing data (e.g., using machine learning algorithms described above and below) are provided by a collection of network-accessible computing and storage resources (e.g., server604and/or edge computing device616), referred to as “the cloud.” Device600may be a cloud client that relies on the cloud computing capabilities from server604to determine whether processing (e.g., at least a portion of virtual background processing and/or at least a portion of other processing tasks) should be offloaded from the mobile device, and facilitate such offloading. When executed by control circuitry of server604or616, the rendering application may instruct control611or618circuitry to perform processing tasks for the client device and facilitate the encoding/decoding.

Control circuitry504may include communications circuitry suitable for communicating with a server, edge computing systems and devices, a table or database server, or other networks or servers. The instructions for carrying out the above-mentioned functionality may be stored on a server (which is described in more detail in connection withFIG.6). Communications circuitry may include a cable modem, an integrated services digital network (ISDN) modem, a digital subscriber line (DSL) modem, a telephone modem, Ethernet card, or a wireless modem for communications with other equipment, or any other suitable communications circuitry. Such communications may involve the Internet or any other suitable communication networks or paths (which is described in more detail in connection withFIG.6). In addition, communications circuitry may include circuitry that enables peer-to-peer communication of user equipment devices, or communication of user equipment devices in locations remote from each other (described in more detail below).

Memory may be an electronic storage device provided as storage508that is part of control circuitry504. As referred to herein, the phrase “electronic storage device” or “storage device” should be understood to mean any device for storing electronic data, computer software, or firmware, such as random-access memory, read-only memory, hard drives, optical drives, digital video disc (DVD) recorders, compact disc (CD) recorders, BLU-RAY disc (BD) recorders, BLU-RAY 3D disc recorders, digital video recorders (DVR, sometimes called a personal video recorder, or PVR), solid state devices, quantum storage devices, gaming consoles, gaming media, or any other suitable fixed or removable storage devices, and/or any combination of the same. Storage508may be used to store various types of content described herein as well as rendering application data described above. Nonvolatile memory may also be used (e.g., to launch a boot-up routine and other instructions). Cloud-based storage, described in relation toFIG.5, may be used to supplement storage508or instead of storage508.

Control circuitry504may include video generating circuitry and tuning circuitry, such as one or more analog tuners, one or more MPEG-2 decoders or other digital decoding circuitry, high-definition tuners, or any other suitable tuning or video circuits or combinations of such circuits. Encoding circuitry (e.g., for converting over-the-air, analog, or digital signals to MPEG signals for storage) may also be provided. Control circuitry504may also include scaler circuitry for upconverting and downconverting content into the preferred output format of user equipment500. Control circuitry504may also include digital-to-analog converter circuitry and analog-to-digital converter circuitry for converting between digital and analog signals. The tuning and encoding circuitry may be used by user equipment device500,501to receive and to display, to play, or to record content. The tuning and encoding circuitry may also be used to receive video encoding/decoding data. The circuitry described herein, including for example, the tuning, video generating, encoding, decoding, encrypting, decrypting, scaler, and analog/digital circuitry, may be implemented using software running on one or more general purpose or specialized processors. Multiple tuners may be provided to handle simultaneous tuning functions (e.g., watch and record functions, picture-in-picture (PIP) functions, multiple-tuner recording, etc.). If storage508is provided as a separate device from user equipment device500, the tuning and encoding circuitry (including multiple tuners) may be associated with storage508.

Control circuitry504may receive instruction from a user by way of user input interface510. User input interface510may be any suitable user interface, such as a remote control, mouse, trackball, keypad, keyboard, touch screen, touchpad, stylus input, joystick, voice recognition interface, or other user input interfaces. Display512may be provided as a stand-alone device or integrated with other elements of each one of user equipment device500and user equipment device501. For example, display512may be a touchscreen or touch-sensitive display. In such circumstances, user input interface510may be integrated with or combined with display512. In some embodiments, user input interface510includes a remote-control device having one or more microphones, buttons, keypads, any other components configured to receive user input or combinations thereof. For example, user input interface510may include a handheld remote-control device having an alphanumeric keypad and option buttons. In a further example, user input interface510may include a handheld remote-control device having a microphone and control circuitry configured to receive and identify voice commands and transmit information to set-top box515.

Audio output equipment514may be integrated with or combined with display512. Display512may be one or more of a monitor, a television, a liquid crystal display (LCD) for a mobile device, amorphous silicon display, low-temperature polysilicon display, electronic ink display, electrophoretic display, active matrix display, electro-wetting display, electro-fluidic display, cathode ray tube display, light-emitting diode display, electroluminescent display, plasma display panel, high-performance addressing display, thin-film transistor display, organic light-emitting diode display, surface-conduction electron-emitter display (SED), laser television, carbon nanotubes, quantum dot display, interferometric modulator display, SLM head mounted display, or any other suitable equipment for displaying visual images. A video card or graphics card may generate the output to the display512. Audio output equipment514may be provided as integrated with other elements of each one of device500and equipment501or may be stand-alone units. An audio component of videos and other content displayed on display512may be played through speakers (or headphones) of audio output equipment514. In some embodiments, audio may be distributed to a receiver (not shown), which processes and outputs the audio via speakers of audio output equipment514. In some embodiments, for example, control circuitry504is configured to provide audio cues to a user, or other audio feedback to a user, using speakers of audio output equipment514. There may be a separate microphone516or audio output equipment514may include a microphone configured to receive audio input such as voice commands or speech. For example, a user may speak letters or words that are received by the microphone and converted to text by control circuitry504. In a further example, a user may voice commands that are received by a microphone and recognized by control circuitry504. Camera518may be any suitable video camera integrated with the equipment or externally connected. Camera518may be a digital camera comprising a charge-coupled device (CCD) and/or a complementary metal-oxide semiconductor (CMOS) image sensor. Camera518may be an analog camera that converts to digital images via a video card.

The rendering application may be implemented using any suitable architecture. For example, it may be a stand-alone application wholly-implemented on each one of user equipment device500and user equipment device501. In such an approach, instructions of the application may be stored locally (e.g., in storage508), and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry504may retrieve instructions of the application from storage508and process the instructions to provide encoding/decoding functionality and preform any of the actions discussed herein. Based on the processed instructions, control circuitry504may determine what action to perform when input is received from user input interface510. For example, movement of a cursor on a display up/down may be indicated by the processed instructions when user input interface510indicates that an up/down button was selected. An application and/or any instructions for performing any of the embodiments discussed herein may be encoded on computer-readable media. Computer-readable media includes any media capable of storing data. The computer-readable media may be non-transitory including, but not limited to, volatile and non-volatile computer memory or storage devices such as a hard disk, floppy disk, USB drive, DVD, CD, media card, register memory, processor cache, Random Access Memory (RAM), etc.

In some embodiments, the rendering application is a client/server-based application. Data for use by a thick or thin client implemented on each one of user equipment device500and user equipment device501may be retrieved on-demand by issuing requests to a server remote to each one of user equipment device500and user equipment device501. For example, the remote server may store the instructions for the application in a storage device. The remote server may process the stored instructions using circuitry (e.g., control circuitry504) and generate the displays discussed above and below. The client device may receive the displays generated by the remote server and may display the content of the displays locally on device500. This way, the processing of the instructions is performed remotely by the server while the resulting displays (e.g., that may include text, a keyboard, or other visuals) are provided locally on device500. Device500may receive inputs from the user via input interface510and transmit those inputs to the remote server for processing and generating the corresponding displays. For example, device500may transmit a communication to the remote server indicating that an up/down button was selected via input interface510. The remote server may process instructions in accordance with that input and generate a display of the application corresponding to the input (e.g., a display that moves a cursor up/down). The generated display is then transmitted to device500for presentation to the user.

In some embodiments, the rendering application may be downloaded and interpreted or otherwise run by an interpreter or virtual machine (run by control circuitry504). In some embodiments, the rendering application may be encoded in the ETV Binary Interchange Format (EBIF), received by control circuitry504as part of a suitable feed, and interpreted by a user agent running on control circuitry504. For example, the rendering application may be an EBIF application. In some embodiments, the rendering application may be defined by a series of JAVA-based files that are received and run by a local virtual machine or other suitable middleware executed by control circuitry504. In some of such embodiments (e.g., those employing MPEG-2 or other digital media encoding schemes), rendering application may be, for example, encoded and transmitted in an MPEG-2 object carousel with the MPEG audio and video packets of a program.

FIG.6is a diagram of an illustrative system600for encoding/decoding and rendering, in accordance with some embodiments of this disclosure. User equipment devices607,608,610(e.g., which may correspond to one or more of computing device500or501may be coupled to communication network606). Communication network606may be one or more networks including the Internet, a mobile phone network, mobile voice or data network (e.g., a 5G, 4G, or LTE network), cable network, public switched telephone network, or other types of communication network or combinations of communication networks. Paths (e.g., depicted as arrows connecting the respective devices to the communication network606) may separately or together include one or more communications paths, such as a satellite path, a fiber-optic path, a cable path, a path that supports Internet communications (e.g., IPTV), free-space connections (e.g., for broadcast or other wireless signals), or any other suitable wired or wireless communications path or combination of such paths. Communications with the client devices may be provided by one or more of these communications paths but are shown as a single path inFIG.6to avoid overcomplicating the drawing.

Although communications paths are not drawn between user equipment devices, these devices may communicate directly with each other via communications paths as well as other short-range, point-to-point communications paths, such as USB cables, IEEE 1394 cables, wireless paths (e.g., Bluetooth, infrared, IEEE 702-11x, etc.), or other short-range communication via wired or wireless paths. The user equipment devices may also communicate with each other directly through an indirect path via communication network606.

System600may comprise media content source602, one or more servers604, and one or more edge computing devices616(e.g., included as part of an edge computing system, such as, for example, managed by mobile operator206). In some embodiments, the rendering application may be executed at one or more of control circuitry611of server604(and/or control circuitry of user equipment devices607,608,610and/or control circuitry618of edge computing device616). In some embodiments, data structure300ofFIG.3, may be stored at database605maintained at or otherwise associated with server604, and/or at storage622and/or at storage of one or more of user equipment devices607,608,610.

In some embodiments, server604may include control circuitry611and storage614(e.g., RAM, ROM, Hard Disk, Removable Disk, etc.). Storage614may store one or more databases. Server604may also include an input/output path612. I/O path612may provide encoding/decoding data, device information, or other data, over a local area network (LAN) or wide area network (WAN), and/or other content and data to control circuitry611, which may include processing circuitry, and storage614. Control circuitry611may be used to send and receive commands, requests, and other suitable data using I/O path612, which may comprise I/O circuitry. I/O path612may connect control circuitry611(and specifically control circuitry) to one or more communications paths.

Control circuitry611may be based on any suitable control circuitry such as one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer. In some embodiments, control circuitry611may be distributed across multiple separate processors or processing units, for example, multiple of the same type of processing units (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). In some embodiments, control circuitry611executes instructions for an emulation system application stored in memory (e.g., the storage614). Memory may be an electronic storage device provided as storage614that is part of control circuitry611.

Edge computing device616may comprise control circuitry618, I/O path620and storage622, which may be implemented in a similar manner as control circuitry611, I/O path612and storage624, respectively of server604. Edge computing device616may be configured to be in communication with one or more of user equipment devices607,608,610and video server604over communication network606, and may be configured to perform processing tasks (e.g., encoding/decoding) in connection with ongoing processing of video data. In some embodiments, a plurality of edge computing devices616may be strategically located at various geographic locations, and may be mobile edge computing devices configured to provide processing support for mobile devices at various geographical regions.

FIG.7is a flowchart of an illustrative process700for rendering a 3D image. In various embodiments, the individual steps of process700may be implemented by one or more components of the devices and systems ofFIGS.5-6. Although the present disclosure may describe certain steps of the process (and of other processes described herein) as being implemented by certain components of the devices and systems ofFIGS.5-6, this is for purposes of illustration only, and it should be understood that other components of the devices and systems ofFIGS.5-6may implement those steps instead.

At step702, input/output circuitry of a computing device receives image data defining a texture of an image and depth data for the image (e.g., data120,122) for display on a spatial light modulator (SLM) device or stack display device. The input/output circuitry may receive the data from an external device and/or from internal storage.

At step704, the control circuitry of computing device, generates a first weight matrix based on a number of least significant bits of each depth value of the depth matrix. For example, a first 2 MSBs (in little endian notation) of the depth matrix may be set to zero. At step706, the control circuitry, generate a second weight matrix that is complement of the first weight matrix. At step708, the control circuitry generates a control matrix based on a number of most significant bits of each depth value of the depth matrix. For example, the last 6 MSBs (in little endian notation) of the depth matrix may be set to zero.

At step710, the control circuitry, generates a first folded focal plane matrix by applying weights of the first weight matrix to the texture matrix. For example, this may be done as describe with respect to line1126of pseudocode ofFIG.3.

At step712, the control circuitry generates a second folded focal plane matrix by applying weights of the second weight matrix to the texture matrix. For example, this may be done as describe with respect to line1127of pseudocode ofFIG.3.

At step714, the control circuitry of computing device generates a phase function for a spatial light modulator of the SLM device (e.g., display512), based at least in part on control matrix generated at step708, and configures the spatial light modulator of the SLM device in accordance with the generated phase function. At step716, the control circuitry provides the first and the second focal planes matrices as input to a spatial display device to generate for display a 3D representation of the received texture matrix. For example, light may be generated on two tiled displays according to the first and the second folded focal planes matrices, e.g., as shown in element402ofFIG.4. The light may be processed as shown inFIG.4to result in 3D image being projected to an eye of a user.

The systems and processes discussed above are intended to be illustrative and not limiting. One skilled in the art would appreciate that the actions of the processes discussed herein may be omitted, modified, combined, and/or rearranged, and any additional actions may be performed without departing from the scope of the invention. More generally, the above disclosure is meant to be exemplary and not limiting. Only the claims that follow are meant to set bounds as to what the present disclosure includes. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and flowcharts or examples relating to one embodiment may be combined with any other embodiment in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and/or methods described above may be applied to, or used in accordance with, other systems and/or methods.