Patent Description:
Digital ink allows a user to draw on a screen of a display device using a device such as a digitized pen. Typically, the input from the device generates a command from the CPU to update the screen to include the digital ink. The screen update is provided to a graphics processor (GPU) that renders the updated screen including the digital ink. The GPU typically queues the digital ink along with other updates to the screen and renders the ink when the screen is updated. Typical screen refresh rates are on the order of <NUM>-<NUM>. A user may move a digital pen tip at high speeds (relative to the screen refresh rate) with rapid changes in direction. For example, a digital pen may provide position updates at <NUM>. Due to the length of the rendering pipeline, there may be a delay of at least <NUM> between the time an input of digital ink is received to the time that the GPU is able to render the input. Accordingly, when drawing with digital ink, a user may notice a gap between the tip of the pen input and the rendered digital ink. The user may feel the experience of drawing with digital ink is less responsive than traditional ink. In the case of specialized digital ink having more detailed features such as pencil effects, the gap between the tip of the pen input and the rendered digital ink may reduce feedback to the user regarding the detailed features of the digital ink.

One solution to processing digital ink is to use customized processing hardware in the display device that processes the input and renders the digital ink. For example, the display device may detect the location of a pen and shade pixels using the customized processing hardware. While effective, this solution is customized to specific hardware and the customized processing hardware adds significant cost.

Thus, there is a need in the art for improvements in graphics processing for updating digital ink on display devices. In <CIT> devices, systems, and, methods are disclosed for processing stylus interactions with a device and drawing the results of those interactions in a manner that reduces lag. This includes the introduction of a separate overlay module layer that can be updated separately from a normal view system/process of a computing device. In this respect, the overlay module layer may be used to remove unnecessary synchronization events to allow for quick display of stylus input events in the overlay module layer while still allowing the normal rendering process of the operating system to be followed.

<CIT> discloses touch path logic configured to receive a plurality of touch events and to generate an output based on the touch event.

<CIT> relates to performing hit testing in a graphical user interface.

It is the object of the present invention to provide a method and system for quickly and efficiently rendering digital pencil ink.

The following presents a simplified summary of one or more implementations of the present disclosure in order to provide a basic understanding of such implementations. This summary is not an extensive overview of all contemplated implementations, and is intended to neither identify key or critical elements of all implementations nor delineate the scope of any or all implementations. Its sole purpose is to present some concepts of one or more implementations of the present disclosure in a simplified form as a prelude to the more detailed description that is presented later.

In an example, a method of drawing digital pencil ink on a display may include rendering a frame via a graphics queue of a graphics processing unit (GPU). The method may include fetching updated digital pencil ink input from an input buffer at a designated time before scanning at least a portion of the frame including the digital pencil ink to the display, the digital pencil ink input including input locations and input properties associated with each input location. The method may include determining a set of input stamps based on the updated digital pencil ink input, each input stamp being associated with stamp properties. The method may include determining, using a compute shader thread for each block within a portion of the frame, whether each of the input stamps intersects the block. The method may include determining, using at least one compute shader thread for each pixel of a respective block, a cumulative effect of each of the input stamps intersecting the respective block on the pixel based on the stamp properties. The method may include outputting each pixel to the display based on the cumulative effect of each of the stamps.

In another example, a computer device for drawing digital pencil ink is provided. The computer device may include a memory storing one or more parameters or instructions for executing an operating system and one or more applications. The computer device may include a graphics processing unit (GPU) for rendering frames of the one or more applications for display on a display device coupled to the computer device, the GPU including a graphics queue and a priority queue. The computer device may include at least one processor coupled to the memory, and the GPU. The at least one processor may be configured to render a frame via the graphics queue of the GPU. The at least one processor may be configured to fetch updated digital pencil ink input from an input buffer at a designated time before scanning at least a portion of the frame including the digital pencil ink to the display, the digital pencil ink input including input locations and input properties associated with each input location. The at least one processor may be configured to determine a set of input stamps based on the updated digital pencil ink input, each input stamp being associated with stamp properties. The at least one processor may be configured to determine, using a compute shader thread for each block within a portion of the frame, whether each of the input stamps intersects the block. The at least one processor may be configured to determine, using at least one compute shader thread for each pixel of a respective block, a cumulative effect of each of the input stamps intersecting the respective block on the pixel based on the stamp properties. The at least one processor may be configured to output each pixel to the display based on the cumulative effect of each of the stamps.

In another example, a computer-readable medium includes code executable by one or more processors for drawing digital pencil ink on a display using a GPU in a computer device. The code may include code for rendering a frame via a graphics queue of the GPU. The code may include code for fetching updated digital pencil ink input from an input buffer at a designated time before scanning at least a portion of the frame including the digital pencil ink to the display, the digital pencil ink input including input locations and input properties associated with each input location. The code may include code for dispatching a first compute shader thread for each input location of the updated digital pencil ink input to determine a set of input stamps based on the updated digital pencil ink input, each input stamp being associated with stamp properties. The code may include code for dispatching a second compute shader thread group for each block within a portion of the frame to determine whether each of the input stamps intersects the block, each thread group including a thread for each input stamp. The code may include code for dispatching at least one third compute shader thread for each pixel of a respective intersected block to determine a cumulative effect of each of the input stamps intersecting the respective intersecting block on the pixel based on the stamp properties. The code may include code for outputting each pixel to the display based on the cumulative effect of each of the stamps.

Additional advantages and novel features relating to implementations of the present disclosure will be set forth in part in the description that follows, and in part will become more apparent to those skilled in the art upon examination of the following or upon learning by practice thereof.

The present disclosure provides systems and methods for rendering digital pencil ink on a display with relatively low latency (as compared to current solutions). As used herein, the term digital ink includes any coloring of pixels on a display based on an indication by a user relative to the display. For example, digital ink may be drawn by a user using a digital pen, stylus, or finger. Digital pencil ink refers to digital ink intended to create a visual effect similar to drawing with a pencil. As discussed in further detail below, digital pencil ink may be rendered as a series of stamps, each stamp having properties based on how the input device is held and characteristics of a selected pencil.

In an implementation, for example, this disclosure provides systems and methods for rendering digital pencil ink on a screen using a priority queue to add the most up to date ink input to a rendered frame. The priority queue may be a queue for a compute shader. The compute shader may be a programmable shader stage that provides high-speed general purpose computing and takes advantage of the large numbers of parallel processors on the graphics processing unit (GPU). The compute shader may perform operations in a single stage and may have a priority queue that is separate from a graphics pipeline. A characteristic of the priority queue is that graphics processing work in the priority queue may be processed more quickly than graphics processing work in the graphics pipeline. Unlike custom processing hardware, the compute shader may be a non-customized component of a GPU on many computer devices. For example, such a compute shader may be found on computer devices including GPUs capable of running at least MICROSOFT Direct3D <NUM> ®. The systems and method use the priority queue and the compute shader to process and draw the most recent updates (e.g., at least with respect to position) to the digital pencil ink to a frame that is about to be displayed, while the traditional graphics pipeline is used to render previously received digital pencil ink in subsequent frames.

In an implementation, the input for the most recent digital pencil ink may be put on a relatively fast path (as compared to a path to the graphics pipeline) to update the high priority compute shader drawing. For example, the pen input may be provided directly to an ink function, bypassing an application and 2D compositor. The ink function may provide the command to the compute shader to draw the ink update as digital pencil ink using the compute shader. The command may be added to the priority queue of the compute shader. The timing for the digital pencil ink updates may be provided by a monitored fence that provides an indication with respect to a video synchronization (V-SYNC) corresponding to a deadline for presenting a next frame.

Various procedures may be used by the compute shader to add the digital pencil ink to a frame. Generally, the compute shader does not use triangles as in traditional graphics processing. In an example procedure, input into the ink function may be a series of input points on the display. In a first pass, the compute shader may determine stamp properties for each input point. In a second pass, a portion of the display may be divided into blocks, and the compute shader may determine which stamps intersect each block. In a third pass, the compute shader may determine for each pixel in the intersected blocks a weight of each stamp. The cumulative weight of the stamps may be used to determine how to display each pixel. Because a compute shader is programmable, additional techniques for drawing high quality ink may be developed as needed. The updated ink may be provided to the traditional rendering pipeline for display in frames subsequent to the frame when the updated ink is received.

Referring now to <FIG>, an example computer system <NUM> includes a computer device <NUM> and a digital pen <NUM> (also referred to as a stylus). The computer device <NUM> may be, for example, any mobile or fixed computer device including but not limited to a desktop or laptop or tablet computer, a cellular telephone, a gaming device, a mixed reality or virtual reality device, a music device, a television, a navigation system, a camera, a personal digital assistant (PDA), a handheld device, any other computer device having wired and/or wireless connection capability with one or more other devices, or any other type of computerized device capable of receiving inputs from digital pen <NUM>. The computer device <NUM> may include a display <NUM>. The display <NUM> may be a digitized surface such as a touch screen that performs both output of images and receiving input from a user. The display <NUM> may include a digitizer <NUM> for detecting a location of an interaction between a user and the display <NUM>. For example, the digitizer <NUM> may detect the location of a finger or the digital pen <NUM> on the display <NUM> or a point of contact or near point of contact between the digital pen <NUM> and the display <NUM>. In some examples, the computer device <NUM> may detect the digital pen <NUM> hovering near the digitized surface and register a touch event upon an action such as clicking a button on the digital pen <NUM>.

As illustrated in <FIG>, the computer device <NUM> may allow a user to draw digital pencil ink <NUM> on the display <NUM>. For example, <FIG> illustrates the word "Ink" being written on the display <NUM> in a continuous line. The display <NUM> may be periodically updated at a refresh rate (e.g., <NUM> - <NUM>). The digital pen <NUM> and/or the digitizer <NUM> may provide updated position information at a higher rate (e.g., <NUM>) than the refresh rate. A displayed portion of the digital pencil ink <NUM> may end at an end point <NUM> corresponding to a last update of the position used as an input into rendering an image for the display <NUM>. Because the digital pen <NUM> may move relatively quickly, a segment <NUM> (represented as a dashed line) between the end point <NUM> and the pen tip <NUM> may not be included in the digital pencil ink <NUM> rendered to the image on display <NUM>. Accordingly, when a user is writing with the digital pen <NUM>, a gap corresponding to the segment <NUM> may appear between the digital pencil ink <NUM> and the pen tip <NUM>.

Referring to <FIG>, an example computer system <NUM> may include a computer device <NUM> that provides images for display on the display <NUM> using a graphics processing unit (GPU) <NUM> including a priority queue <NUM> for receiving position updates from digital pen <NUM> and a compute shader <NUM> for rendering the pen updates. The computer device <NUM> may also include a CPU <NUM> that executes instructions stored in memory <NUM>. For example, the CPU <NUM> may execute an operating system <NUM> and one or more applications <NUM>. The operating system <NUM> may control the GPU <NUM> and the use of the priority queue <NUM> and the compute shader <NUM> for drawing digital pencil ink in a manner that reduces latency between digital pencil ink input (e.g., from digital pen <NUM>) and drawing of the digital pencil ink on the display <NUM>, e.g., for reducing the gap (relative to current solutions) corresponding to the segment <NUM> may appear between the digital pencil ink <NUM> and the pen tip <NUM>.

Computer device <NUM> may include a memory <NUM> and CPU <NUM> configured to control the operation of computer device <NUM>. Memory <NUM> may be configured for storing data and/or computer-executable instructions defining and/or associated with an operating system <NUM> and/or application <NUM>, and CPU <NUM> may execute operating system <NUM> and/or application <NUM>. An example of memory <NUM> can include, but is not limited to, a type of memory usable by a computer, such as random access memory (RAM), read only memory (ROM), tapes, magnetic discs, optical discs, volatile memory, non-volatile memory, and any combination thereof. Memory <NUM> may store local versions of applications being executed by CPU <NUM>.

The CPU <NUM> may include one or more processors for executing instructions. An example of CPU <NUM> can include, but is not limited to, any processor specially programmed as described herein, including a controller, microcontroller, application specific integrated circuit (ASIC), field programmable gate array (FPGA), system on chip (SoC), or other programmable logic or state machine. The CPU <NUM> may include other processing components such as an arithmetic logic unit (ALU), registers, and a control unit.

The operating system <NUM> may include instructions (such as application <NUM>) stored in memory <NUM> and executable by the CPU <NUM>. The operating system <NUM> may include a display controller <NUM> for controlling the GPU <NUM>. For example, the display controller <NUM> may provide commands <NUM> to the GPU <NUM> to perform one or more specific graphics processing operations such as rendering source images or performing adjustments. The display controller <NUM> may include a compositor <NUM>, in the form of a hardware and/or software component, configured to combine multiple sources of information to create a complete image for display. For example, in a 2D environment, the compositor <NUM> may determine in which windows various applications are to be rendered.

The GPU <NUM> may include one or more processors and specialized hardware for image processing. In an implementation, the GPU <NUM> may be integrated with a CPU <NUM> on a motherboard of the computer device or may be a discrete chip. The GPU <NUM> may include a dedicated memory <NUM>. The GPU <NUM> may be connected to the display <NUM> via a display interface <NUM>. The GPU <NUM> may periodically scan out an image from an image buffer <NUM> to the display <NUM> via the display interface <NUM> according to a refresh rate of the display <NUM>. The GPU <NUM> may include a graphics queue <NUM>, a render pipeline <NUM>, a priority queue <NUM>, and a compute shader <NUM>. The graphics queue <NUM> may receive commands from the display controller <NUM> for rendering an image. The graphics queue <NUM> may generally provide the commands to the render pipeline <NUM>. The render pipeline <NUM> may perform multiple stages of image processing. For example, the render pipeline <NUM> may include an input-assembler stage, vertex shader stage, hull shader stage, tessellator stage, domain shader stage, geometry shader stage, stream output stage, rasterizer stage, pixel-shader stage, and output merger stage.

The priority queue <NUM> may receive commands from the ink function <NUM> and/or display controller <NUM>. Generally, the priority queue <NUM> may provide commands to the compute shader <NUM>. The compute shader <NUM> may operate as a single processing stage, which may allow prioritization of commands in the priority queue <NUM> over existing commands (e.g., from the graphics queue). Accordingly, passing commands to the compute shader <NUM> via the priority queue <NUM> may allow execution of the commands in a fast and predictable manner. In an implementation, the priority queue <NUM> may also provide commands to the render pipeline <NUM>. For example, the render pipeline hardware may allow interruption of ongoing commands at certain stages of the render pipeline <NUM> or may include additional resources for processing high-priority commands. Accordingly, the use of the priority queue <NUM> and/or compute shader <NUM> may allow the relatively simple operation of drawing ink to be performed more quickly than if the same operation is performed by multiple stages of the render pipeline <NUM>. For example, the compute shader <NUM> may draw updated ink in less time than the render pipeline <NUM> takes to render the entire frame. Therefore, when using the priority queue and/or compute shader <NUM>, the time for obtaining input for the ink update may be moved closer to the time that the ink is displayed.

In an example, display interface <NUM> can be communicatively coupled with the GPU <NUM> and/or memory <NUM> for communicating with the display <NUM>. A display interface, as referred to herein, may also include various types of ports, including high definition multimedia interface (HDMI) ports, display serial interface (DSI) ports, mobile industry processor interface (MIPI) DSI ports, universal serial bus (USB) ports, Firewire ports, or other embedded or external wired or wireless display ports that can allow communications between computer device <NUM> and display <NUM>.

The operating system <NUM> may include an ink function <NUM>. The ink function <NUM> may provide a fast path for ink input to reach the priority queue <NUM>. For example, the fast path may provide commands for drawing the ink input to the priority queue rather than a traditional path of providing ink input to an application to process before generating commands for rendering the ink. Accordingly, the fast path may bypass the application <NUM>. The digital pen <NUM> and/or digitizer <NUM> may provide input information (e.g., pen coordinates and properties) to an input buffer <NUM> in memory <NUM> via a wired or wireless connection <NUM>. When woken up for input, the ink function <NUM> may access the input buffer <NUM> to obtain the input information. The ink function <NUM> may determine whether the input information is ink input or another type of input (e.g., a button press), for example, based on whether the input is in a drawing area or whether an inking mode is selected. The ink function <NUM> may generate commands for the GPU <NUM> to draw digital pencil ink based on the ink input and send the commands to the priority queue <NUM>. For example, the ink function <NUM> may generate dispatch commands for the compute shader <NUM> to draw digital pencil ink <NUM> based on the ink input. The ink function <NUM> may provide ink input to other components that utilize the ink input such as the display controller <NUM> and the application <NUM>. For example, the display controller <NUM> may control the GPU <NUM> to render the ink input in a subsequent frame (relative to a frame currently being presented) using the graphics queue <NUM>.

The ink function <NUM> may also control timing of obtaining ink input and generating commands to draw the digital pencil ink. In order to minimize latency between the ink input and drawing the digital pencil ink on the display <NUM>, the ink function <NUM> may attempt to obtain the ink input as late as possible before processing the ink input for inclusion in a frame that is to be presented. In an implementation, the GPU <NUM> may scan the image buffer <NUM> to the display <NUM> in a fixed pattern (e.g., rasters). As used herein, scanning may refer to a process of updating pixels on the display <NUM>. For example, the display <NUM> may be scanned from the top to the bottom starting at a video synchronization (V-SYNC) to update each pixel. The ink function <NUM> may include a predictor <NUM> that may estimate a time at which new digital pencil ink will be scanned to the display <NUM>. For example, if the ink input is located at the top of the display <NUM>, the digital pencil ink may be scanned shortly after the V-SYNC. In contrast, if the ink input is located at the bottom of the display <NUM>, the digital pencil ink may not be scanned until closer to the end of a frame. The ink function <NUM> may be able to obtain additional ink input and draw the digital pencil ink to the image buffer <NUM> before the location of the digital pencil ink is scanned to the display <NUM>. The predictor <NUM> may predict the time at which the new digital pencil ink will be scanned based on the most recent position of the digital pencil ink input and the velocity of the digital pen <NUM>. The predictor <NUM> may use the predicted time to determine a safety margin for providing the drawing commands to the GPU <NUM> in order to complete drawing the digital pencil ink before the GPU <NUM> scans the location to the display <NUM>.

The ink function <NUM> may also include a timing function <NUM> for waking up the ink function <NUM> to obtain the input. The timing function <NUM> may set a wake up time for each frame based on the V-SYNC and the predicted safety margin using a periodic monitored fence. The periodic monitored fence may provide a signal at a configured time before a hardware event such as the V-SYNC. The periodic monitored fence may be a synchronization object that allows a processor (e.g., CPU <NUM> or GPU <NUM>) to signal or wait on a particular fence object. The synchronization object can wait on periodically occurring wait events, such as a time offset from various V-SYNCs. In an example, the timing function <NUM> may set the offset for a periodic monitored fence. The timing function <NUM> may adjust a periodic monitored fence based on the predicted safety margin such that the ink function <NUM> wakes up in time to obtain the input information and generate the commands for the GPU <NUM>.

The computer device <NUM> may also include an application <NUM> including instructions stored in memory <NUM> and executed by the CPU <NUM>. The application <NUM>, for example, may be an application that uses digital pencil ink, for example, to take notes or create a drawing. A user may provide input to the application <NUM> using digital pencil ink. The application <NUM>, for example, may perform character recognition to translate the digital pencil ink into text. It should be noted that the ink function <NUM> may bypass the application <NUM> for initially drawing the digital pencil ink. For example, the ink function <NUM> may obtain the ink input and generate commands for drawing the digital pencil ink before providing the ink input <NUM> to the application <NUM>. The ink function <NUM> may also communicate with the application <NUM> to determine properties of the digital pencil ink such as a selected pencil having a line width and color.

Referring now to <FIG>, the present disclosure may allow computer device <NUM> to draw digital pencil ink <NUM> on the display <NUM> in a manner that reduces the length of the segment <NUM> in comparison to <FIG>. For example, <FIG> illustrates the word "Ink" being written on the display <NUM> in a continuous line using the same input as in <FIG>. The end point <NUM> may still represent the last ink input that is available at the V-SYNC for rendering the frame via the graphics queue <NUM>. However, the computer device <NUM> may obtain updated ink input including the point <NUM> after the V-SYNC. The computer device <NUM> may then draw the segment <NUM> using the priority queue <NUM> and/or the compute shader <NUM>. Accordingly, the gap corresponding to the segment <NUM> may be reduced by drawing the segment <NUM>. It should be appreciated that although <FIG> illustrates drawing the updated digital pencil ink with the segment <NUM> being a straight line for simplicity, more complex curves may also be drawn using the priority queue <NUM> and/or the compute shader <NUM>.

Referring now to <FIG>, an example method <NUM> provides for the computer device <NUM> to display digital pencil ink on the display <NUM>. For example, method <NUM> may be used for displaying digital pencil ink <NUM> as it is being drawn by the digital pen <NUM> such that the end point <NUM> is kept close to the pen tip <NUM>. As a result, operation of method <NUM> may draw digital pencil ink <NUM> with less latency than current solutions. The actions illustrated in method <NUM> may overlap in time. For example, at an instant in time, two of the actions may be performed by different components. The execution of the actions may also be interleaved on a component. Additionally, the actions illustrated in method <NUM> may be performed in an order other than illustrated in <FIG>. Further details and examples of timing are discussed below with respect to <FIG>.

At <NUM>, method <NUM> may optionally include rendering a frame via a graphics queue of a graphics processing unit (GPU). For example, the display controller <NUM> may render a frame via the graphics queue <NUM> of the GPU <NUM>. At <NUM>, the action <NUM> may include rendering digital pencil ink within the frame via a rendering pipeline of the GPU, wherein the digital pencil ink is available at a video synchronization (V-SYNC) preceding the frame. For example, the display controller <NUM> may render digital pencil ink within the frame via the render pipeline <NUM> of the GPU <NUM>. The digital pencil ink may be based on ink input that was available before a video synchronization preceding the frame (e.g., digital pencil ink <NUM> up to end point <NUM>). That is, the digital pencil ink available at the V-SYNC preceding the frame may be processed through a normal frame rendering process via the graphics queue <NUM> and the render pipeline <NUM>.

At <NUM>, method <NUM> may optionally include determining a designated time for obtaining updated digital pencil ink input for the frame based on at least a previous input location. For example, the predictor <NUM> may determine the designated time for obtaining digital pencil ink input for the frame based on at least the previous input location. In an implementation, the predictor <NUM> may estimate a location of the digital pencil ink input based on the previous input location and an input velocity. The predictor <NUM> may estimate a time after a V-SYNC when the estimated location will be scanned. The predictor <NUM> may determine a safety margin before the estimated scanning time for drawing the digital pencil ink. The predictor <NUM> may set the designated time at or before the safety margin. The designated time may be either before or after the V-SYNC depend on the estimated location of the ink input. In an implementation, the predictor <NUM> may adjust the safety margin based on feedback regarding either the accuracy of the predictions or the time for the GPU to complete operations. For example, the predictor <NUM> may receive an indication of a time when the GPU <NUM> actually starts a drawing operation after being woken up and/or an indication of a time when the GPU <NUM> finishes rendering a frame or drawing ink using the compute shader <NUM>.

At <NUM>, the method <NUM> may optionally include setting a periodic monitored fence based on the designated time. For example, the timing function <NUM> may set the periodic monitored fence based on the designated time. The timing function <NUM> may set the time when the periodic monitored fence wakes up the ink function <NUM> for each frame.

At <NUM>, the method <NUM> may include fetching updated digital pencil ink input from an input buff at the designated time before scanning at least a portion of the frame including the digital pencil ink to the display. For example, the ink function <NUM> may fetch the updated digital pencil ink input from the input buffer <NUM> at the designated time before displaying the frame. The obtained digital pencil ink input may include any updates to the position of the digital pen <NUM> at the designated time. For example, the updated digital pencil ink input may include input from after the V-SYNC for the corresponding frame. Accordingly, the obtained digital pencil ink input may include additional locations (e.g., point <NUM> and other points along segment <NUM>) after a previous V-SYNC. In some implementations, the obtained digital pencil ink input may include additional locations input after the current V-SYNC but before a scanning operation reaches the input location. The digital pencil ink input may include input locations and input properties associated with each input location.

At <NUM>, the method <NUM> may include drawing the updated digital pencil ink on the rendered frame via a priority queue of the GPU based on the updated digital pencil ink input prior to displaying at least a portion of the frame including the digital pencil ink. For example, the ink function <NUM> may draw the updated digital pencil ink (e.g., segment <NUM>) on the rendered frame via the priority queue <NUM> of the GPU <NUM> based on the updated digital pencil ink input. The drawing may be performed before the GPU <NUM> scans at least a portion of the frame including the updated digital pencil ink (e.g., the portion of the frame including segment <NUM>) to the display <NUM>. For example, the compute shader <NUM> may draw the segment <NUM> on the rendered image in the image buffer <NUM> after the GPU starts scanning out the frame at the V-SYNC, but before the GPU <NUM> reaches the end point <NUM>. Accordingly, when the GPU <NUM> reaches the end point <NUM>, the GPU <NUM> may begin scanning out the segment <NUM> as if it had been rendered before the V-SYNC.

At <NUM>, the action <NUM> may include dispatching a compute shader configured to draw the digital pencil ink on the rendered frame based on the digital pencil ink input. For example, the ink function <NUM> may dispatch the compute shader <NUM> (e.g., by sending commands to the priority queue <NUM>) to draw the digital pencil ink on the rendered frame based on the updated digital pencil ink input. Further details and examples of using the compute shader <NUM> to draw digital pencil ink are discussed below regarding <FIG>.

At <NUM>, the method <NUM> may optionally include determining that a rendering pipeline will not complete the frame before a V-SYNC, the frame including digital pencil ink based on input available at a previous V-SYNC. For example, the GPU <NUM> may indicate that the render pipeline <NUM> will not complete a frame before an upcoming V-SYNC. The GPU <NUM> may instead display the previous frame again. Accordingly, digital pencil ink input between the previous frame and the previous V-SYNC may not be rendered, resulting in a gap before the end point <NUM>.

At <NUM>, the method <NUM> may include dispatching compute shaders to draw the digital pencil ink <NUM> available at the previous V-SYNC. For example, the ink function <NUM> may dispatch the compute shader <NUM> via the priority queue <NUM> to draw the digital pencil ink <NUM> available at the previous V-SYNC. The ink function <NUM> may dispatch the compute shader <NUM> via the priority queue <NUM> in response to an indication that the render pipeline <NUM> will not complete a frame before the V-SYNC. Accordingly, the compute shader <NUM> may draw both the previously available digital pencil ink <NUM> and the updated digital pencil ink (e.g., segment <NUM>) obtained after the V-SYNC. Drawing both sets of digital pencil ink with the compute shader <NUM> may result in a continuous line of digital pencil ink.

Referring now to <FIG>, an example timing diagram <NUM> illustrates rendering of frames and display of digital pencil ink using GPU <NUM>, according to conventional techniques, where latency in presenting the digital pencil ink may occur due to relatively long queuing and processing times. The display <NUM> may have a periodic V-SYNC <NUM> (including <NUM>-a, <NUM>-b, etc.), which may correspond to the refresh rate of the display <NUM>. For example, a refresh rate may be <NUM>-<NUM>, resulting in a V-SYNC <NUM> every <NUM> - <NUM> milliseconds. The computer device <NUM> may also receive input updates <NUM> (e.g., updated position information) from digital pen <NUM>. The input updates <NUM> may be more frequent than V-SYNC <NUM>. For example, the input updates <NUM> may occur at a rate of <NUM>.

In order to draw the digital pencil ink <NUM> on the display <NUM>, the CPU <NUM> may obtain the input updates <NUM> and perform a command operation <NUM> to instruct the GPU <NUM> to render the digital pencil ink as part of a graphics frame. For example, at V-SYNC <NUM>-a, the CPU <NUM> may obtain ink updates up to input update <NUM>-a. The command operation <NUM> may include generating commands and sending commands to graphics queue <NUM> for execution by render pipeline <NUM>. The commands may include commands for rendering the digital pencil ink as well as commands for rendering a display image (e.g., based on application <NUM> and compositor <NUM>).

The render pipeline <NUM> may perform a rendering operation <NUM> to render the digital pencil ink along with the graphics frame. The rendering operation <NUM> may be a computationally intense operation. The time for completing the rendering operation <NUM> may depend on the quality of the rendered images being produced. For example, the rendering time may be based on factors such as resolution and number of colors, as well as quality of various visual effects (e.g., shadows or particles). The quality of the images may be adjusted based on hardware capability such that the GPU <NUM> is typically capable of rendering new frames at the refresh rate.

The GPU <NUM> and the display <NUM> may also perform a display operation <NUM> in which the GPU <NUM> scans the rendered image out to the display <NUM>. For example, the GPU <NUM> may update the pixels of the display <NUM> line by line starting at the V-SYNC <NUM>. The display operation <NUM> may extend over a substantial portion of the frame. For example, a frame A based on input at V-SYNC <NUM>-a may begin scanning at V-SYNC <NUM>-b, and may not be completed until V-SYNC <NUM>-c. Accordingly, a portion (e.g., the bottom portion) of the frame may not be scanned until later in the frame. The time <NUM> between the input update <NUM>-a and the V-SYNC <NUM>-b may represent an average gap (e.g., for a pixel in the middle of the display <NUM>) between the drawn digital pencil ink and the location of the pen tip <NUM>. Generally, when the time of the ink update is fixed, portions of the frame that are scanned first will have a shorter time <NUM> than portions of the frame that are scanned last. The length of segment <NUM> may be proportional to the time <NUM>.

In the subsequent frame, the digital pencil ink input may be obtained at input update <NUM>-b. The digital pencil ink based on input update <NUM>-a may be considered existing digital pencil ink. The rendering operation <NUM>-b may include rendering both the existing digital pencil ink (A) and new digital pencil ink (B) as part of the next frame. Likewise, the display operation <NUM> may include scanning the rendered image including the existing digital pencil ink (A) and new digital pencil ink (B).

Referring now to <FIG>, according to the present disclosure, timing diagram <NUM> illustrates another example of rendering of frames and display of digital pencil ink using GPU <NUM>, where latency in presenting digital pencil ink may be reduced by drawing digital pencil ink updates using a priority queue <NUM> and compute shader <NUM>. The rate of V-SYNC <NUM> and input updates <NUM> may be the same as in <FIG>. As in <FIG>, the input update <NUM>-a may be obtained by the CPU <NUM> at the V-SYNC <NUM>-a, the CPU <NUM> may perform the command operation <NUM> and the render pipeline <NUM> may perform the rendering operation <NUM>.

Additionally, in order to reduce latency, the CPU <NUM> may obtain ink input at input update <NUM>-c and perform command operation <NUM>-c to generate commands for compute shader <NUM> to draw digital pencil ink. The compute shader <NUM> may draw the additional digital pencil ink (C) that became available between input update <NUM>-a and input update <NUM>-c. For example, in draw operation <NUM>-a, the compute shader <NUM> may adjust pixels of the rendered frame resulting from rendering operation <NUM>-a. Accordingly, at the V-SYNC <NUM>-b, the image buffer <NUM> may include digital pencil ink based on input until input update <NUM>-c. The GPU <NUM> and display <NUM> may perform display operation <NUM>-a to display the image including digital pencil ink up to C, that is A+C. As illustrated in <FIG>, the time <NUM> between the last input update <NUM>-c and the V-SYNC <NUM>-b is significantly shorter than the time <NUM> of <FIG> (added to <FIG> for ease of comparison). Accordingly, the length of segment <NUM>, e.g., the gap between the end of the ink and pen tip, may be reduced in comparison to the prior art process of <FIG>.

In the subsequent frame beginning at V-SYNC <NUM>-b, the CPU <NUM> may obtain ink input based on input update <NUM>-b. Accordingly, the CPU <NUM> may perform the command operation <NUM>-a to render the digital pencil ink A+C+B. At input update <NUM>-d, the CPU <NUM> may obtain the new digital pencil ink input and perform command operation <NUM>-d to draw the digital pencil ink using the priority queue <NUM> and the compute shader <NUM>. The compute shader <NUM> may perform the draw operation <NUM>-b to draw the digital pencil ink D. Accordingly, the display operation <NUM>-b may include digital pencil ink A+C+B+D.

In a third frame, additional digital pencil ink input may be obtained at input update <NUM>-e. Because the input update <NUM>-e is available before the V-SYNC <NUM>-c, the digital pencil ink may be queued for rendering via the render pipeline <NUM>. However, the render pipeline may be started late or take a longer time to complete, for example, due to a higher priority process interrupting the rendering. The render pipeline <NUM> may generate a signal indicating that the rendering operation <NUM>-c will not be completed by the V-SYNC <NUM>-d. Traditionally, the GPU <NUM> would display the previously rendered frame. In this case, the previously rendered frame would not include the ink updates from <NUM>-d or <NUM>-e. Accordingly, in response to a signal that the render process will not complete the rendering operation <NUM>-c before the V-SYNC <NUM>-d, the priority queue <NUM> and/or compute shader <NUM> may be used to draw the updated digital pencil ink based on input updates <NUM>-d and <NUM>-e. Accordingly, the display operation <NUM>-c may display all of the available digital pencil ink.

Referring now to <FIG>, timing diagram <NUM> illustrates another example of rendering and display of digital pencil ink using the priority queue <NUM> and/or compute shader <NUM> of GPU <NUM>, where further latency reductions in presenting digital pencil ink may be obtained based on a predicted time for scanning the digital pencil ink. The rate of V-SYNC <NUM> and input updates <NUM> may be the same as in <FIG>.

Instead of obtaining an updated ink input at a fixed time before the V-SYNC <NUM>, the CPU <NUM> (e.g., by executing predictor <NUM>) may determine the time for obtaining the updated ink input. For example, the CPU <NUM> may determine, based on input update <NUM>-a, that any additional ink input is likely to be located at the bottom of the display <NUM> and does not need to be scanned until time <NUM>. The CPU <NUM> may also determine a safety margin <NUM> for performing a draw operation <NUM> for drawing any new ink. The CPU <NUM> may then obtain an input update <NUM> before the safety margin <NUM>. In this example, the input update may be input update <NUM>-g. In an implementation, if the predicted portion for the digital pencil ink is not scanned until near the end of the frame, the updated digital pencil ink may be obtained after the V-SYNC (e.g., V-SYNC <NUM>-b), where the GPU <NUM> starts scanning the image to the display <NUM>. Accordingly, digital pencil ink obtained at the input update <NUM>-g, which according to <FIG> would not be rendered until after V-SYNC <NUM>-d, may be drawn and scanned more than a full frame earlier according to the technique of <FIG>. Therefore, the time <NUM> may be less than the time <NUM> in <FIG> (added to <FIG> for ease of comparison) and/or time <NUM> in <FIG> (added to <FIG> for ease of comparison). Additionally, it should be noted that times <NUM>, <NUM>, and <NUM> represent an average scenario (e.g., the digital pencil ink is near the middle). By predicting the display time of the digital pencil ink, the worst case scenario can be improved according to the technique illustrated in <FIG> such that the latency between the input from the digital pen <NUM> and drawing the digital pencil ink is consistently low (e.g., less than one frame).

Referring now to <FIG>, an example of digital pencil ink drawn using the compute shader <NUM> is illustrated. The digitizer <NUM> may detect digital pencil ink input <NUM> in a portion of the display <NUM>. The digital pencil ink input <NUM> may be interpreted as a set of stamps <NUM>. For example, the CPU <NUM> and/or ink function <NUM> may generate a stamp for each input point (i.e., the location of the digital pen <NUM> when input is recorded). For digital pencil ink, each input point may be associated with input properties of the digital pen <NUM> such as pressure and tilt. Previously rendered digital pencil ink is omitted for simplicity. A bounding box <NUM> may be determined based on the maximum and minimum values of the updated digital pencil ink input. The bounding box may be expanded by a maximum radius of a stamp to ensure all stamps are within the bounding box. The bounding box <NUM> may be used to reduce the processing load on the GPU <NUM> and the compute shader <NUM>. For example, the bounding box <NUM> may exclude all of the pixels outside of the bounding box <NUM> from being processed by the compute shader <NUM> regarding the updated digital pencil ink.

In order to quickly and efficiently render the updated digital pencil ink, the bounding box <NUM> may be segmented into a set of blocks <NUM>. Each block <NUM> may include a number of pixels. For example, a block <NUM> may be an <NUM> x <NUM> block of pixels, although other sized blocks may be used. If the bounding box <NUM> is not evenly divisible into blocks, partial blocks may be padded outside of the bounding box <NUM> to generate full blocks.

In one implementation, the compute shader <NUM> may draw the digital pencil ink input <NUM> using three passes (although other implementations may use a different number of passes). In each pass, the CPU <NUM> may dispatch multiple compute shader threads to perform an operation on multiple instances of an input. A different input may be used in each pass. In a first pass, the compute shader <NUM> may determine stamp properties of each stamp <NUM> based on the digital pencil ink input and associated digital pen properties. For example, the digital pencil ink input may include coordinates of an output pixel where the digital pen <NUM> was located when the input point was sampled, a pressure on the digital pen <NUM> when the input point was sampled, and an angle of the pen when the input point was sampled. The CPU <NUM> may obtain the digital pencil ink input from the input buffer <NUM>. The format of the digital pencil ink input may not be ideal for the GPU <NUM> to operate on. For example, the GPU <NUM> may need to convert the coordinates, pressure, and tilt into a description of an ellipse shaped stamp before determining whether the stamp affects an output pixel. The first pass of the compute shader may generate a thread for each digital pencil ink input point. Each thread may compute the stamp properties of the stamp <NUM>. For example, the stamp properties of the stamp <NUM> may include a shape, size, and texture sample. The shape may be an ellipse based on the tilt. The size may be based on the pressure. The texture sample may be based on a selected type or size of pencil and the pressure. For example, the type or size of pencil may be based on conventional graphite pencils and may produce a line having similar properties to a line drawn with a corresponding graphite pencil. The thread for the first pass may also perform any computations that may be used by later passes. For example, the thread for the first pass may generate a pre-computed table of texture samples.

In a second pass, the compute shader <NUM> may determine which stamps <NUM> affect each block <NUM>. That is, the compute shader <NUM> may determine whether block <NUM> is intersected by each stamp <NUM>. The compute shader may dispatch a thread group for each block <NUM>. Each thread group may include one thread for each stamp <NUM>. Each individual thread may determine whether the respective block <NUM> is intersected by the respective stamp <NUM>. In an implementation, determining the intersections may be simplified using a hit circle <NUM> to represent the block <NUM>. If the center of the stamp <NUM> is within a distance of the center of the block <NUM> defined by the hit circle <NUM>, the thread may determine an intersection.

The thread group may generate a bit mask <NUM>. The bit mask <NUM> for the block <NUM> may indicate which input stamps <NUM> intersect the block <NUM>. If the stamp <NUM> intersects the block <NUM>, the stamp <NUM> may be added to a bit mask for the block <NUM> (e.g., a corresponding bit of the bit mask <NUM> may be set to <NUM>). Blocks <NUM> that are not intersected by at least one stamp <NUM> of the digital pencil ink input <NUM> (e.g., bit mask equals <NUM>) may be culled from the third pass. The second pass may fill an output buffer <NUM> with a number of intersected blocks <NUM> and an entry <NUM> for each intersected block including an identifier of the intersected block (or thread group identifier) and the bit mask <NUM> for the intersected block.

In a third pass, the compute shader <NUM> may color pixels. The compute shader <NUM> may be dispatched using a dispatch indirect command that allows the output buffer <NUM> to provide input to determine the number of thread groups. For example, in the third pass, the compute shader <NUM> may spawn a thread group for each intersected block included in an entry <NUM> based on the number of intersected blocks <NUM>. Using the dispatch indirect command may avoid latency in transferring information (e.g., the number of blocks) from the GPU <NUM> to the CPU <NUM> to dispatch the compute shader <NUM>. In the third pass, each thread in the group may correspond to a pixel of an intersected block <NUM>. For example, the thread group may include <NUM> threads for an 8x8 block. Each thread may determine, for each stamp <NUM> identified by the bit mask <NUM>, an effect of the respective stamp <NUM> on the individual pixel. For example, the thread may determine to apply the texture sample to the pixel. The thread for the pixel may determine the cumulative effect of all of the intersecting stamps <NUM> on the pixel. For example, as illustrated in <FIG>, some of the pixels intersect a single stamp <NUM>, while other pixels intersect multiple stamps. Each stamp may contribute to the shading of the pixel. Accordingly, a pixel intersected by multiple stamps may have a more intense effect applied. In other words, a weight applied to the pixel may be greater as more stamps contribute to the cumulative effect. In an implementation, the effect may be applied to current values in a representation of the pixel. Source over blending may be used to blend the current values with the cumulative effect. For example, the cumulative effect on a white pixel may be more dramatic than the effect on a dark pixel.

<FIG> shows various data structures that may be used to process the updated digital pencil ink. A data structure <NUM> may be stored in the input buffer <NUM> for digital pencil ink input. The data structure <NUM> may include coordinates including an x-coordinate <NUM> and a y-coordinate <NUM>. The data structure <NUM> may also include input properties such as tilt <NUM> and pressure <NUM> associated with each input point <NUM>.

The data structure <NUM> may be generated by the compute shader <NUM> in the first pass and stored in the memory <NUM> to store GPU stamps. The data structure <NUM> may include coordinates including an x-coordinate <NUM> and a y-coordinate <NUM>. The data structure <NUM> may also include stamp properties such as shape <NUM> and texture sample <NUM> associated with each input point <NUM>.

The data structure <NUM> may be generated by the compute shader <NUM> in the second pass and stored in the memory <NUM> to store the output buffer <NUM>. The data structure <NUM> may include a number of intersected blocks <NUM>. The data structure <NUM> may also include block coordinates including an x-coordinate <NUM> and a y-coordinate <NUM>. The block coordinates may refer to the coordinates of a block <NUM> rather than a pixel. The data structure <NUM> may include the mask <NUM> associated with each entry <NUM>.

The data structure <NUM> may be generated by the compute shader <NUM> in the third pass. The data structure <NUM> may include coordinates including an x-coordinate <NUM> and a y-coordinate <NUM> of each output pixel <NUM>. The data structure <NUM> may also include a total weighted value <NUM> to be applied to the output pixel.

Referring now to <FIG>, an example method <NUM> provides for the compute shader <NUM> to draw digital pencil ink on the display <NUM>. For example, method <NUM> may be used for displaying digital pencil ink <NUM> as it is being drawn by the digital pen <NUM> such that the end point <NUM> is kept close to the pen tip <NUM>.

At <NUM>, the method <NUM> may include determining a set of input stamps based on the updated digital pencil ink input, each input stamp being associated with stamp properties. For example, the ink function <NUM> may determine the set of input stamps <NUM> based on the updated digital pencil ink input, each input stamp being associated with stamp properties such as shape <NUM> and texture sample <NUM>. In an implementation, for instance, action <NUM> may optionally include, at <NUM>, dispatching a compute shader thread for each input point of the updated digital pencil ink input. For example, the ink function (executed by the CPU) may call the compute shader <NUM> of the GPU <NUM> to dispatch a compute shader thread for each input point of the digital pencil ink input. In an implementation, for instance, action <NUM> may optionally include, at <NUM>, determining the stamp properties based on the input properties. For example, each thread may determine the stamp properties (e.g., shape <NUM> and texture sample <NUM>) based on the input properties (e.g., tilt <NUM> and pressure <NUM>).

At <NUM>, the method <NUM> may include determining, using a compute shader thread for each block within a portion of the frame, whether each of the input stamps intersects the block. For example, the compute shader <NUM> may spawn a thread group for each block <NUM>. Each thread within the thread group may correspond to a stamp <NUM>. Each thread may determine whether the stamp <NUM> will intersect the respective block <NUM>. At <NUM>, the action <NUM> may optionally include generating an output buffer including a set of intersected blocks and bitmask for each intersected block indicating which of the stamps intersect the intersected block. For example, the compute shader <NUM> may generate the output buffer <NUM> including a number of intersected blocks <NUM> and entries <NUM> for each of the set of intersected blocks. In an implementation, the output buffer <NUM> may only include blocks intersected by a stamp. In another implementation, the action <NUM> may optionally include culling blocks that are not intersected by at least one stamp. For example, the compute shader <NUM> may cull blocks that are not intersected by at least one stamp from the output buffer <NUM>. Culling blocks may reduce the work of the compute shader in a third pass.

At <NUM>, the method <NUM> may include determining, using at least one compute shader thread for each pixel within a respective block, a cumulative effect of each of the input stamps intersecting the respective block on the pixel based on the stamp properties. For example, the compute shader <NUM> may spawn a thread group for each block <NUM> included in the output buffer <NUM>. Each thread group may include a thread for each pixel within the block (e.g., <NUM> threads for the block <NUM>). Each thread may loop through the stamps intersecting the block <NUM> to determine a weight to apply to the pixel. For example, action <NUM> may optionally include, at <NUM>, generating a compute shader thread group for each input stamp intersecting the respective block, the thread group including the compute shader thread for each pixel within the respective block. For instance, the compute shader <NUM> may generate the compute shader thread groups based on the mask <NUM> for the respective block indicating which stamps <NUM> intersect the block <NUM>.

Each compute shader thread may determine a weight of each stamp <NUM> and a total weighted value <NUM> for all of the stamps <NUM>. For example, at <NUM>, the action <NUM> may optionally include determining a location of the pixel within the stamp. For example, the compute shader thread may look up the stamp in the data structure <NUM> and determine the location of the pixel within the stamp based on the x-coordinate <NUM>, y-coordinate <NUM>, and shape <NUM>. At <NUM>, the action <NUM> may optionally include determining a weight of the stamp on the pixel. For example, the compute shader <NUM> may use the location determined at action <NUM> and the texture sample <NUM> to determine the weight of the stamp on the pixel. The actions <NUM> and <NUM> may be repeated for each pixel within the block <NUM>. At <NUM>, the action <NUM> may optionally include writing a total weighted value to the pixel. The compute shader <NUM> may sum the weight of each of the stamps and blend the total weight with a current value of the pixel to determine the total weighted value for the pixel. The compute shader <NUM> may write the total weighted value <NUM> to the data structure <NUM>.

At <NUM>, the method <NUM> may include outputting each pixel to the display based on the cumulative effect of each of the stamps. For example, the GPU <NUM> may output each pixel to the display <NUM> based on the cumulative effect of each of the stamps. In an implementation, the GPU <NUM> may scan out the image buffer <NUM> at the V-SYNC after applying the respective total weighted value <NUM> to each pixel in the image buffer <NUM>. Accordingly, the display <NUM> may include the segment <NUM> representing the updated digital pencil ink rendered as digital pencil ink.

Referring now to <FIG>, illustrated is an example computer device <NUM> in accordance with an implementation, including additional component details as compared to <FIG>. In one example, computer device <NUM> may include processor <NUM> for carrying out processing functions associated with one or more of components and functions described herein. Processor <NUM> can include a single or multiple set of processors or multi-core processors. Moreover, processor <NUM> can be implemented as an integrated processing system and/or a distributed processing system. In an implementation, for example, processor <NUM> may include CPU <NUM> and/or GPU <NUM>. In an example, computer device <NUM> may include memory <NUM> for storing instructions executable by the processor <NUM> for carrying out the functions described herein. In an implementation, for example, memory <NUM> may include memory <NUM> and/or memory <NUM>.

Further, computer device <NUM> may include a communications component <NUM> that provides for establishing and maintaining communications with one or more parties utilizing hardware, software, and services as described herein. Communications component <NUM> may carry communications between components on computer device <NUM>, as well as between computer device <NUM> and external devices, such as devices located across a communications network and/or devices serially or locally connected to computer device <NUM>. For example, communications component <NUM> may include one or more buses, and may further include transmit chain components and receive chain components associated with a transmitter and receiver, respectively, operable for interfacing with external devices. In an implementation, for example, communications component <NUM> may include connection <NUM> for communicatively connecting digital pen <NUM> to CPU <NUM> and memory <NUM>.

Additionally, computer device <NUM> may include a data store <NUM>, which can be any suitable combination of hardware and/or software, that provides for mass storage of information, databases, and programs employed in connection with implementations described herein. For example, data store <NUM> may be a data repository for operating system <NUM> (<FIG>) and/or applications <NUM> (<FIG>).

Computer device <NUM> may also include a user interface component <NUM> operable to receive inputs from a user of computer device <NUM> and further operable to generate outputs for presentation to the user. User interface component <NUM> may include one or more input devices, including but not limited to a keyboard, a number pad, a mouse, a touch-sensitive display, a digitizer, a navigation key, a function key, a microphone, a voice recognition component, any other mechanism capable of receiving an input from a user, or any combination thereof. Further, user interface component <NUM> may include one or more output devices, including but not limited to a display (e.g., display <NUM>), a speaker, a haptic feedback mechanism, a printer, any other mechanism capable of presenting an output to a user, or any combination thereof.

In an implementation, user interface component <NUM> may transmit and/or receive messages corresponding to the operation of operating system <NUM> and/or application <NUM>. In addition, processor <NUM> executes operating system <NUM> and/or application <NUM>, and memory <NUM> or data store <NUM> may store them.

As used in this application, the terms "component," "system" and the like are intended to include a computer-related entity, such as but not limited to hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computer device and the computer device can be a component. One or more components can reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets, such as data from one component interacting with another component in a local system, distributed system, and/or across a network such as the Internet with other systems by way of the signal.

Various implementations or features may have been presented in terms of systems that may include a number of devices, components, modules, and the like. It is to be understood and appreciated that the various systems may include additional devices, components, modules, etc. and/or may not include all of the devices, components, modules etc. discussed in connection with the figures. A combination of these approaches may also be used.

The various illustrative logics, logical blocks, and actions of methods described in connection with the embodiments disclosed herein may be implemented or performed with a specially-programmed one of a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor may also be implemented as a combination of computer devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Additionally, at least one processor may comprise one or more components operable to perform one or more of the steps and/or actions described above.

Further, the steps and/or actions of a method or procedure described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor, such that the processor can read information from, and write information to, the storage medium. Further, in some implementations, the processor and the storage medium may reside in an ASIC. Additionally, the ASIC may reside in a user terminal. Additionally, in some implementations, the steps and/or actions of a method or procedure may reside as one or any combination or set of codes and/or instructions on a machine readable medium and/or computer readable medium, which may be incorporated into a computer program product.

In one or more implementations, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium. A storage medium may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs usually reproduce data optically with lasers.

Claim 1:
A method (<NUM>) of drawing digital pencil ink on a display for input received after rendering a frame via a graphics queue of a graphics processing unit, GPU, comprising:
fetching updated digital pencil ink input from an input buffer before scanning at least a portion of the frame including the digital pencil ink to the display, the updated digital pencil ink input including input locations and input properties associated with each input location, wherein the portion of the frame is segmented into a set of blocks, and wherein each block includes a plurality of pixels;
determining (<NUM>) a set of input stamps based on the updated digital pencil ink input, each input stamp being associated with stamp properties;
determining (<NUM>), using a compute shader thread for each block within the portion of the frame, whether each of the input stamps intersects the block;
determining (<NUM>), using at least one compute shader thread for each pixel of a respective intersected block, a cumulative effect of each of the input stamps intersecting the respective block on the pixel based on the stamp properties; and
outputting (<NUM>) each pixel to the display based on the cumulative effect of each of the stamps.