Patent Description:
According to the invention, there is provided a system according to claim <NUM> and a method according to claim <NUM>.

In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

Systems, apparatuses, and methods for game engines rendering directly to video encoders are disclosed herein. In one embodiment, a system includes a game engine and a video encoder in a server coupled via a network to a client with a decoder. The game engine includes an embedded rendering unit configured to render images in a first color space for display. The rendering unit is also configured to render images in a second color space directly to a video encoder. In one embodiment, the first color space is RGB and the second color space is YUV. The game engine also generates a plurality of attributes associated with each rendered image and conveys the plurality of attributes to the video encoder. The plurality of attributes includes various types of data that were used by the rendering unit to render the image. The video encoder encodes each rendered image into an encoded bitstream based on the plurality of attributes associated with the rendered image.

In one embodiment, the rendering unit is located on a server in the cloud, and rendered content is being conveyed to a client via a network. For example, a cloud gaming application can be executed, with the gaming application frames being rendered in the cloud. In this scenario, the rendering unit will convey rendered frames to an encoder. The rendering unit can be implemented using any of various types of processing units. In one embodiment, the rendering unit is implemented on a graphics processor or graphics subsystem. In another embodiment, the rendering unit is implemented on a general purpose central processing unit (CPU). In other embodiments, the rendering unit can be implemented on other types of processing units (e.g., application specific integrated circuit (ASIC), field programmable gate array (FPGA), digital signal processor (DSP)).

Referring now to <FIG>, a block diagram of one embodiment of a system <NUM> for encoding rendered images into a video bitstream which is sent over a network is shown. System <NUM> includes server <NUM>, network <NUM>, client <NUM>, and display <NUM>. In other embodiments, system <NUM> can include multiple clients connected to server <NUM> via network <NUM>, with the multiple clients receiving the same bitstream or different bitstreams generated by server <NUM>. System <NUM> can also include more than one server <NUM> for generating multiple bitstreams for multiple clients. In one embodiment, system <NUM> is configured to implement real-time rendering and encoding of game content as part of a cloud gaming application. Latency, quality, bitrate, power, and performance challenges typically arise while delivering such a workload in real-time. In other embodiments, system <NUM> is configured to execute other types of applications.

In one embodiment, server <NUM> is configured to render video or image frames, encode the frames into a bitstream, and then convey the encoded bitstream to client <NUM> via network <NUM>. Client <NUM> is configured to decode the encoded bitstream and generate video frames or images to drive to display <NUM> or to a display compositor. In one embodiment, server <NUM> includes a game engine for rendering images to be displayed to a user. As used herein, the term "game engine" is defined as a real-time rendering application for rendering images. A game engine can include various shaders (e.g., vertex shader, geometry shader) for rendering images. The game engine is typically utilized to generate rendered images to be immediately displayed on a display connected to server <NUM>. However, some applications can run using a client-server model where the rendered content will be displayed at a remote location. For these applications, the rendered images are encoded by a video encoder into a video bitstream. The video bitstream is then sent over network <NUM> to client <NUM> to be viewed on display <NUM>. In various embodiments, the video bitstream <NUM> is conveyed to the network <NUM> via a network interface (not shown) according to any of a variety of suitable communication protocols (e.g., TCP/IP, etc.).

Network <NUM> is representative of any type of network or combination of networks, including wireless connection, direct local area network (LAN), metropolitan area network (MAN), wide area network (WAN), an Intranet, the Internet, a cable network, a packet-switched network, a fiber-optic network, a router, storage area network, or other type of network. Examples of LANs include Ethernet networks, Fiber Distributed Data Interface (FDDI) networks, and token ring networks. Network <NUM> can further include remote direct memory access (RDMA) hardware and/or software, transmission control protocol/internet protocol (TCP/IP) hardware and/or software, router, repeaters, switches, grids, and/or other components.

Server <NUM> includes any combination of software and/or hardware for rendering video/image frames and encoding the frames into a bitstream. In one embodiment, server <NUM> includes one or more software applications executing on one or more processors of one or more servers. Server <NUM> also includes network communication capabilities, one or more input/output devices, and/or other components. The processor(s) of server <NUM> can include any number and type (e.g., graphics processing units (GPUs), CPUs, DSPs, FPGAs, ASICs) of processors. The processor(s) can be coupled to one or more memory devices storing program instructions executable by the processor(s). Similarly, client <NUM> includes any combination of software and/or hardware for decoding a bitstream and driving frames to display <NUM>. In one embodiment, client <NUM> includes one or more software applications executing on one or more processors of one or more computing devices. Client <NUM> can be a computing device, game console, mobile device, streaming media player, or other type of device.

Turning now to <FIG>, a block diagram of one embodiment of the software components of a server <NUM> is shown. In one embodiment, server <NUM> includes an application <NUM> configured to execute on the processing units of server <NUM>. In one embodiment, application <NUM> is a video game application. For example, application <NUM> can be a cloud gaming application configured to convey and/or receive video of gameplay. In another embodiment, application <NUM> is a gaming spectatorship application configured to view video of gameplay. In various embodiments, server <NUM> includes one or more processors, one or more memory devices, and additional components which are indicative of a server or other computing device. The various software components shown in <FIG> are configured to execute on the processing units of server <NUM>. In one embodiment, server <NUM> is part of a cloud computing architecture.

In one embodiment, application <NUM> is configured to receive inputs (e.g., game commands) which specify user movements/actions that were captured on a client device where the user displays and interacts with the video stream generated by application <NUM>. Application <NUM> includes graphics information which is provided to game engine <NUM> to render frames for display. Game engine <NUM> is configured to render image frames based on a game state of application <NUM>. The rendered image frames are then intended to be provided to a display engine and then driven to a display. However, in one embodiment, the rendered image frames will be displayed on the display of a remote client device. In this embodiment, the rendered image frames will be encoded into a video bitstream, with the video bitstream sent to the remote client device.

In one embodiment, game engine <NUM> is configured to provide a plurality of attributes <NUM> associated with each rendered image <NUM> to encoder <NUM> for use in the encoding of the video bitstream from the rendered images. The plurality of attributes <NUM> include a variety of data which rendering unit <NUM> used to build the rendered image. Accordingly, these attributes <NUM> already exist and rendering unit <NUM> can provide these attributes <NUM> to encoder <NUM> to enhance the encoding process. By receiving these attributes <NUM>, encoder <NUM> is able to forgo various processing and analysis tasks which would normally be performed for each rendered image <NUM>. In various embodiments, the plurality of attributes <NUM> can include, but are not limited to, the following attributes: camera-view velocity buffer, per-object motion information, texture data hints, regions of interest (ROI), depth information (e.g., stereoscopic video encode), temporal statistics, rendering frame rate, scene change, areas (e.g., rectangles) that require extra bits, skip or static frame indications, and post processing data. In other embodiments, other information can be collected and/or one or more of these attributes can be omitted from the attributes <NUM> that are provided to video encoder <NUM>.

In one embodiment, the camera-view velocity information can be utilized to assist a motion estimator. In one embodiment, the ROI information can include dirty rectangles data, content dependent objects of importance, game/app statistics, and so on. The game/app statistics can be shown in an opaque, transparent, or translucent overlay. The scene change data can assist the encoder refresh logic. Areas that can require extra bits include overlaid game statistics or menus.

Depending on the embodiment, various rendered image attributes can be utilized to control or adjust different encoder settings. For example, if two consecutive images (ImageN-<NUM> and ImageN) are similar (e.g., ImageN - ImageN-<NUM> < Threshold), then rendering unit <NUM> can signal the encoder <NUM> to skip a frame. Also, if marked areas are unchanged when comparing two consecutive images (ImageN compared to ImageN-<NUM>), encoder <NUM> can build a direct reference for those regions in ImageN to ImageN-<NUM>. Also, rendering unit <NUM> can alternate the marked ROI quantization parameter values (to achieve more or less compression). Furthermore, motion information from velocity buffers (ImageN compared to ImageN-<NUM>) can be utilized. For example, the motion data can be used to modify the per-macroblock quantization parameter which is used in the encoding process. This means allocating more encoding bits towards specific content based on its velocity. Still further, depth information can be used to encode stereoscopic images.

In one embodiment, game engine <NUM> is configured to render images in the Red/Green/Blue (RGB) color space. However, in one embodiment, encoder <NUM> is configured to encode images in the Luminance/Chrominance (YUV) color space into a video stream. Accordingly, in such a case the rendered RGB images are not compatible with the encoder <NUM> that is configured to render images in the YUV color space. In order to address this discrepancy, in various embodiments the rendering unit <NUM> is configured to render images in either or both of the RGB and YUV color spaces. It is noted that the YUV color space can also be referred to as YCbCr or Y'CbCr color space.

In one embodiment, rendering unit <NUM> of game engine <NUM> writes a rendered image <NUM> into a first portion of buffer <NUM> and rendering unit <NUM> of game engine <NUM> writes attributes <NUM> associated with rendered image <NUM> into a second portion of buffer <NUM>. In one embodiment, video encoder <NUM> is configured to retrieve rendered image <NUM> and the associated attributes <NUM> from buffer <NUM>. Video encoder <NUM> is configured to utilize attributes <NUM> when encoding rendered image <NUM> into encoded video bitstream <NUM>. Video encoder <NUM> is configured to retrieve (or otherwise receive) subsequently rendered images and associated attributes from buffer <NUM> and continue to generate encoded video bitstream <NUM> from the subsequently rendered images. In various embodiments, the encoding of the frames generated by rendering unit <NUM> of game engine <NUM> is according to any of a variety of video formats and standards as desired (e.g., "HD", "<NUM>", "<NUM> Ultra", H. <NUM>, etc.).

In one embodiment, encoder <NUM> is configured to generate encoded bitstream <NUM> with a given bit budget. For example, encoder <NUM> can have a specific bit budget based on an available or specified network bandwidth (e.g., <NUM> megabits per second (Mbps)) for bitstream <NUM>, and encoder <NUM> encodes the frames to meet this specific bit budget. As encoded bitstream <NUM> is generated by encoder <NUM>, encoded bitstream <NUM> is conveyed from server <NUM> to the client (not shown). The client receives encoded bitstream <NUM>, decodes bitstream <NUM> into individual frames, and then drives the frames to the display or to a display compositor.

Referring now to <FIG>, a block diagram of one embodiment of a computing system <NUM> is shown. Computing system <NUM> includes an application <NUM> which receives player movements or actions. Computing system <NUM> also includes a game engine <NUM> with an integrated rendering unit <NUM>. In the embodiment shown in <FIG>, computing system <NUM> is configured to generate a rendered image <NUM> for display on display device <NUM> and a rendered image <NUM> rendered specifically for video encoder <NUM>. Rendering unit <NUM> also provides attributes <NUM> to video encoder <NUM> in buffer <NUM> to assist with the encoding process. Video encoder <NUM> retrieves the rendered image <NUM> and attributes <NUM> from buffer <NUM> for each image generated by rendering unit <NUM> of game engine <NUM>, and then video encoder <NUM> generates encoded video bitstream <NUM> from the rendered image <NUM> and attributes <NUM>. In one embodiment, rendered image <NUM> is generated in a first color space, and rendered image <NUM> is generated in a second color space. In one embodiment, the first color space is RGB and the second color space is YUV. Depending on the embodiment, encoded video bitstream <NUM> can be conveyed to a remote system via network <NUM>, or encoded video bitstream <NUM> can be recorded and stored in memory <NUM>.

Turning now to <FIG>, a block diagram of another embodiment of a computing system <NUM> is shown. System <NUM> includes game engine <NUM> with an integrated rendering unit <NUM>. In one embodiment, game engine <NUM> can operate in different modes depending on the specific software application being executed and/or depending on the operating conditions of system <NUM>. Table <NUM> includes a list of the different operating modes which game engine <NUM> can implement in accordance with one embodiment. For example, in mode 417A, game engine <NUM> is configured to render to video encoder <NUM> only. In mode 417A, rendering unit <NUM> renders images to the YUV color space and generates attributes <NUM> to be utilized by video encoder <NUM> when encoding each rendered image <NUM> into a video bitstream. Rendering unit <NUM> collects data to be stored in the image attributes section <NUM>. Besides the data that rendering unit <NUM> natively uses to produce images (depth, motion, textures), rendering unit <NUM> can also hint ROI and temporal changes to the previous image. For example, if only a small portion of the current rendered image <NUM> has changed compared to a previously rendered image, then rendering unit <NUM> generates an indication (which is included in attributes <NUM>) for video encoder <NUM> to reuse the unchanged portions from the previous rendered image when encoding the current rendered image <NUM>.

When operating in mode 417B, game engine <NUM> is configured to render to display <NUM> only. In mode 417B, rendering unit <NUM> generates each rendered image <NUM> in the RGB color space and then the rendered RGB image <NUM> is driven to display <NUM> or to a display compositor. When operating in mode 417C, game engine <NUM> is configured to render to video encoder <NUM> and to display <NUM>. In mode 417C, for each frame of the host gaming application, rendering unit <NUM> generates rendered image <NUM> in the RGB color space for display <NUM>, and rendering unit <NUM> generates rendered image <NUM> in the YUV color space and attributes <NUM> for video encoder <NUM>. In other embodiments, game engine <NUM> can operate in other types of modes and/or game engine <NUM> can omit one of the modes shown in table <NUM>.

In one embodiment, when game engine <NUM> is rendering images to be consumed directly by video encoder <NUM>, game engine <NUM> stores rendered image <NUM> and attributes <NUM> in a buffer <NUM> for retrieval by video encoder <NUM>. In one embodiment, translation unit <NUM> translates attributes <NUM> into a format that video encoder <NUM> can consume when encoding rendered image <NUM> into an encoded video bitstream. Translation unit <NUM> can retrieve the metadata (i.e., attributes <NUM>) stored in buffer <NUM> by game engine <NUM>, and translation unit <NUM> can translate the metadata to enable the metadata to be consumed by the various different types of video encoders that can be implemented by system <NUM>. For example this data will help encoders to understand temporal differences, texture, ROI and to fully or partially skip the image analysis. With that, encoders will save time compressing the image and provide better image quality given the time constraints (low latency). In some embodiments, translation unit <NUM> provides additional parameters for video encoder <NUM>, such as a ROI map, a compression parameter map, and a decision on whether to insert a skip frame into the bitstream.

For example, if video encoder <NUM> is encoding a video bitstream in accordance with the H. <NUM> video standard, then translation unit <NUM> can translate attributes <NUM> into a format that is compatible with the H. <NUM> standard. Alternatively, if video encoder <NUM> is encoding a video bitstream in accordance with the H. <NUM> video standard (i.e., High Efficiency Video Coding (HEVC)), then translation unit <NUM> can translate attributes <NUM> into a format that is compatible with the H. <NUM> standard. Still further, if video encoder <NUM> is encoding a video bitstream in accordance with the VP9 codec, then translation unit <NUM> can translate attributes <NUM> into a format that is compatible with the VP9 codec. Translation unit <NUM> can also translate attributes <NUM> into other formats that are compatible with other video compression standards depending on which standards are supported by video encoder <NUM>.

In one embodiment, video encoder <NUM> is configured to provide feedback to rendering unit <NUM> of game engine <NUM> and/or to translation unit <NUM>. For example, if the available bandwidth on the network connection over which video encoder <NUM> is sending the video bitstream decreases, video encoder <NUM> can notify rendering unit <NUM> of game engine <NUM> to reduce the resolution of rendered image <NUM>. Also, in some embodiments, video encoder <NUM> can generate video bitstreams for multiple clients, and if video encoder <NUM> has fewer computation resources for encoding the video bitstream, video encoder <NUM> can inform rendering unit <NUM> of game engine <NUM> to reduce the frame rate, reduce the resolution, and/or perform one or more other actions when generating rendered image <NUM>. In another embodiment, video encoder <NUM> can also request one or more additional attributes to be added to attributes <NUM>. Also, in a further embodiment, video encoder <NUM> also provides feedback to translation unit <NUM> to specify how to format various attributes <NUM> which are provided to video encoder <NUM>.

Referring now to <FIG>, one embodiment of a method <NUM> for a game engine rendering directly to a video encoder is shown. For purposes of discussion, the steps in this embodiment and those of <FIG> are shown in sequential order. However, it is noted that in various embodiments of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method <NUM>.

A game engine with an embedded rendering unit receives inputs for rendering an image (block <NUM>). The game engine renders the image in a format directly compatible with a video encoder (block <NUM>). For example, the game engine renders the image in a color format (e.g., YUV) which the video encoder will utilize when performing the encoding process. The game engine also provides a plurality of attributes to the video encoder for the rendered image (block <NUM>). Next, the video encoder encodes the rendered image into an encoded bitstream based on the plurality of attributes (block <NUM>). Then, video encoder sends the encoded bitstream to a client or stores the encoded bitstream (block <NUM>). After block <NUM>, method <NUM> returns to block <NUM> with the game engine receiving inputs for the next image to render.

Turning now to <FIG>, one embodiment of a method <NUM> for operating a multi-mode game engine is shown. A multi-mode game engine with an embedded rendering unit executes on a computing system (block <NUM>). The system determines the operating mode of the game engine (block <NUM>). If the game engine is operating in a first mode (conditional block <NUM>, "yes" leg), then the rendering unit renders each image into a first color space (i.e., YUV color space) directly for a video encoder only (block <NUM>). Also, the game engine provides a plurality of attributes to the video encoder for each rendered image (block <NUM>). In one embodiment, the render unit attributes data is packed with the YUV rendered image. After block <NUM>, method <NUM> ends.

If the game engine is operating in a second mode (conditional block <NUM>, "yes" leg), then the rendering unit renders each image into a second color space (i.e., RGB color space) for a locally connected display only (block <NUM>). After block <NUM>, method <NUM> ends. If the game engine is operating in a third mode (conditional block <NUM>, "no" leg), then the rendering unit renders each image into both a first color space (i.e., YUV color space) directly for a video encoder and into a second color space (i.e., RGB color space) for a locally connected display (block <NUM>). After block <NUM>, method <NUM> ends. It is noted that in other embodiments, the game engine can have other operating modes. In these embodiments, method <NUM> can be adapted to utilize these other operating modes.

In various embodiments, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various embodiments, such program instructions can be represented by a high-level programming language. In other embodiments, the program instructions can be compiled from a high-level programming language to a binary, intermediate, or other form. Alternatively, program instructions can be written that describe the behavior or design of hardware. Such program instructions can be represented by a high-level programming language, such as C/C++. Alternatively, a hardware design language (HDL) such as Verilog can be used. In various embodiments, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

Claim 1:
A system comprising:
a game engine (<NUM>) comprising a rendering unit (<NUM>) configured to render images for display in one of at least three operating modes:
a render to video encoder only mode;
a render to display only mode; and
a render to video encoder and display mode; and
a video encoder (<NUM>) configured to encode rendered images (<NUM>) that have been rendered in a first color space;
wherein responsive to detecting the render to video encoder and display operating mode, the rendering unit (<NUM>) is configured to:
render each of a plurality of images in both a first color space and a second color space different from the first color space, wherein the first color space is compatible with the video encoder and the second color space is not compatible with the video encoder;
generate a plurality of attributes associated with each image rendered (<NUM>) in the first color space;
convey the attributes and each image rendered in the first color space to the video encoder (<NUM>);
convey each image rendered in the second color space to a locally connected display device (<NUM>) in parallel with conveying each image rendered in the first color space to the video encoder (<NUM>);
wherein the video encoder (<NUM>) is configured to encode each rendered image received by the video encoder (<NUM>) into a video bitstream based on the plurality of attributes provided by the rendering unit (<NUM>).