Perceptual importance maps for image processing

The present disclosure is directed to techniques for determining a perceptual importance map. The perceptual importance map indicates the relative importance to the human visual system of different portions of an image. The techniques include obtaining cost values for the blocks of an image, where cost values are values used in determining motion vectors. For each block, a confidence value is derived from the cost values. The confidence value indicates the confidence with which the motion vector is believed to be correct. A perceptual importance value is determined based on the confidence value via one or more modifications to the confidence value to better reflect importance to the human visual system. The generated perceptual importance values can be used for various purposes such as allocating bits for encoding, identifying regions of interest, or selectively rendering portions of an image with greater or lesser detail based on relative perceptual importance.

BACKGROUND

Image processing is the process of analyzing an image to determine additional information pertinent to the image. A very wide variety of image processing techniques are known. Additional technical developments in the area of image processing are constantly being made.

DETAILED DESCRIPTION

The present disclosure is directed to techniques for determining a perceptual importance map for an image or sequence of images. The perceptual importance map indicates the relative importance to the human visual system of different portions of an image. The techniques include obtaining cost values for the blocks of an image, where cost values are values used in determining motion vectors for, e.g., motion estimation in an image. Specifically, cost values are determined for a set of candidate motion vectors for a block of an image and the “best” cost determines the motion vector assigned to the block. For each block, a confidence value is derived from the cost values. The confidence value indicates the confidence with which the motion vector is believed to be correct. A perceptual importance value is determined based on the confidence value via one or more modifications to the confidence value. These modifications are intended to modify the confidence value to more accurately reflect importance to the human visual system. A map of perceptual importance values, that reflects the values for different blocks of an image, is generated and can be used for various purposes such as allocating bits to different blocks for encoding, identifying regions of interest of an image, or selectively rendering portions of an image with greater or lesser detail based on relative perceptual importance.

FIG. 1is a block diagram of an example device100in which aspects of the present disclosure are implemented. The device100includes, for example, a computer (such as a server, desktop, or laptop computer), a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. The device100includes a processor102, a memory104, a storage device106, one or more input devices108, and one or more output devices110. The device100optionally includes an input driver112and an output driver114. It is understood that the device100optionally includes additional components not shown inFIG. 1.

The processor102includes one or more of: a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, wherein each processor core is a CPU or a GPU. The memory104is located on the same die as the processor102or separately from the processor102. The memory104includes a volatile or non-volatile memory, for example, random access memory (RAM), dynamic RAM, or a cache.

The storage device106includes a fixed or removable storage, for example, a hard disk drive, a solid state drive, an optical disk, or a flash drive. The input devices108include one or more of a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, or a biometric scanner. The output devices110include one or more of a display, a speaker, a printer, a haptic feedback device, one or more lights, or an antenna.

The input driver112communicates with the processor102and the input devices108, and permits the processor102to receive input from the input devices108. The output driver114communicates with the processor102and the output devices110, and permits the processor102to send output to the output devices110.

An image processor120is shown in two different forms. The image processor120obtains images, processes the images, and outputs image processor output. In a first form, the image processor120is software that is stored in the memory104and that executes on the processor102as shown. In a second form, the image processor is at least a portion of a hardware graphics engine that resides in output drivers114. In other forms, the image processor120is a combination of software and hardware elements, with the hardware residing, for example, in output drivers114, and the software executed on, for example, the processor102. The image processor120analyzes images stored in memory, such as in memory104, or any other memory such as a buffer stored in or associated with a hardware implementation of the image processor120, or any other memory. In various examples, the image processor120analyzes sequences of images output by a graphics processor (such as a three-dimensional graphics processing pipeline), or sequences of images stored, for example, as a movie in memory104, storage106, or some other location.

Motion vectors are analytical constructs used in many forms of image processing. Typically, an image processor, such as image processor120, determines motion vectors for various blocks (groupings of pixels) of an image and uses the motion vectors for other processing.FIG. 2illustrates an example image200that includes a plurality of pixel blocks202. As illustrated, the pixel blocks202are subdivisions of the image200. The image processor120may obtain motion vectors for any number of pixel blocks202of the image, but many image processing techniques obtain motion vectors for pixel blocks202that comprise the entire image.

The image processor120determines motion vectors for a particular pixel block202as follows. The image processor120identifies multiple candidate motion vectors and determines cost values for each candidate motion vector. Each candidate motion vector represents a displacement between the block under analysis and a block of a reference frame. The image processor120selects the motion vector for a block based on a comparison of the costs of the different motion vectors (in some techniques, this results in selection of the motion vector corresponding to the lowest cost). In many instances, the selected motion vector represents an estimated amount of motion between the block in the current frame and the block in the reference frame.

FIG. 3illustrates an example instance of application of a technique to obtain motion vectors for a single pixel block302for analysis. The pixel block302is within an image area300that represents both the area of the pixels of the current frame and the area of the pixels of the reference frame. Pixels of both the current frame and the reference frame are illustrated within the image area300so that the motion vector analysis can be illustrated. For the reference frame, only pixels within the search area306are illustrated. For the current frame, only pixels of the pixel block302are illustrated.

To determine the motion vector for the block for analysis302, the image processor120determines costs for each of a variety of candidate motion vectors, which have a variety of directions and amplitudes, and then selects one of the costs based on a cost-selection criteria. The candidate motion vector associated with the selected cost is the motion vector for the block for analysis.

The set of candidate motion vectors that are analyzed may have any definable relationship to the block for analysis302. In one example, the set includes motion vectors that point to each different pixel in the search area306. Thus, the set of candidate motion vectors searched would point from one of the pixels of the block for analysis302to each of the pixels in the search area306. In other examples, the set of candidate motion vectors points to a subset of pixels within a search area, with some pixels not pointed to.

The analysis that is performed to determine cost may be any technically feasible analysis, but generally involves a comparison of the pixels of the block for analysis302with the pixels of comparison blocks310defined by the candidate motion vectors308. In an example, the cost analysis involves calculating some mathematical relationship between pixel pairs that include a pixel from the block for analysis302and a pixel from the comparison block310. In other examples, single metrics are derived from the pixels in a block and those metrics are compared to corresponding single metrics of pixels of a block of a reference frame. Again, any other technically feasible technique is possible. The resultant cost values are analyzed and one is selected based on any technically feasible criteria to select the motion vector for the block for analysis302. This technique is repeated for a variety (such as all) of blocks in the image to be analyzed. Regardless of which technique is used, the cost analysis compares pixels of the block for analysis302, which is in a current frame, with pixels defined by the comparison blocks310, which are part of a reference frame. Note, it is possible for a comparison block310to partially or fully overlap with the block for analysis302, and even for a comparison block310to be in the exact same location as the block for analysis302(which would correspond to a motion vector of zero motion).

There are many techniques that can be used for determining cost of a block202. One example is the sum of absolute differences technique. With the sum of absolute differences technique, cost is determined according to the following expression:

∑i=0n-1⁢⁢∑j=0m-1⁢Cij-Rij
where n is the horizontal dimension of the block in pixels, m is the vertical dimension of the block in pixels, Cijis a pixel value (or sample) at location i, j in the block for analysis302and Rijis the corresponding pixel at location i, j in the comparison block310. The Cijor Rijvalues can be brightness values if pixels are represented in the YUV color space or can be any of the red, green, or blue components, or a sum thereof, if the pixel is represented in the RGB color space.

Another example technique for determining cost is the mean squared error. With the mean squared error technique, cost is determined according to the following expression:

1N2⁢∑i=0n-1⁢⁢∑j=0n-1⁢(Cij-Rij)2
where, as with the mean absolute difference technique, Cijis a pixel at location i, j in the block for analysis302and Rijis the corresponding pixel at location i, j in the comparison block310. Any other technically feasible technique for determining cost of a block302may alternatively be used.

Generally, the “final product” of this analysis is a motion vector for each block302of an image. In both of the above example techniques, a lower cost is associated with a “more desirable” motion vector. For example, in the sum of absolute differences technique, the cost value can be thought of as representing the total similarity between the block for analysis302and the comparison block310. A lower sum of absolute differences represents a more similar block. Thus, the lowest cost would be associated with the most desirable motion vector, and the image processor120would select that motion vector for the block for analysis302. It is possible, however, that cost values would be determined according to different techniques, in which a most desirable cost is the greatest cost or is determined in some other manner.

It is possible to derive other metrics from the above analysis. A “confidence metric” is useful for other aspects of image processing. This confidence metric conceptually represents the confidence that the determined motion vector is “correct” or otherwise the most desirable motion vector to use. Techniques are presented herein for determining the confidence metric, determining a perceptual importance metric based on the confidence metric, and using the perceptual importance metric for various purposes related to image processing.

Some example techniques for determining a confidence metric for a block302of an image are now described. In general, these techniques reflect the following facts. Blocks for analysis302that have distinctive traits will be similar to only one or a few blocks in the reference frame. Because similar blocks have a low cost while different blocks have a high cost, a block for analysis302that has distinctive visual traits will lead to only a few low costs with many high costs. This type of spread out distribution of costs would thus indicate a high confidence that the selected motion vector is correct. On the other hand, blocks for analysis302that do not have distinctive visual traits will be similar to more surrounding blocks, resulting in a less spread out distribution of costs. This less spread out distribution of costs would indicate a lower confidence that the selected motion vector is correct.

In one example technique, the image processor120determines the confidence metric as a ratio of the “best” cost to the “worst” cost (or the reciprocal thereof, if appropriate). Taking the sum of absolute differences technique as an example, the lowest cost would be the cost for the selected motion vector while the highest cost would represent the similarity of the block for analysis302to the least similar block. This ratio would be indicative of the degree to which the cost values are “spread out” for the block for analysis302, which, as described above, would correlate to the distinctiveness of the block for analysis302. More specifically, a higher such metric would indicate a higher confidence and a lower metric would indicate a lower confidence. Because the motion vector is more likely to be correct for a distinctive block, this ratio would be a good representation of the confidence metric.

In another example technique, the image processor120determines the confidence metric as the ratio of the second best cost to the worst cost. This technique could be used similarly to the best to worst cost technique. More specifically, a high value would indicate a more distinctive block and thus a higher confidence while a low value would indicate a less distinctive block and thus a lower confidence.

In yet another example technique, the image processor120determines the confidence metric as the ratio of the best cost to the second best cost. Again, a higher value would indicate a more distinctive block and thus a higher confidence while a lower value would indicate a less distinctive block and thus a lower confidence. In still another example technique, the image processor120determines the confidence metric as the ratio of the best cost to the average of all of the costs. Again, a higher value would indicate a more distinctive block and thus a higher confidence while a lower value would indicate a less distinctive block and thus a lower confidence.

The image processor120determines a perceptual importance metric based on the confidence metric. In various implementations, the image processor120implements one or more of a variety of techniques alone or in combination for determining the perceptual importance metric based on the confidence metric.

A variety of techniques that can be implemented by the image processor120to obtain the perceptual importance metric are now described. The techniques include applying any of the following sets of operations alone or in combination.

According to one set of operations for obtaining the perceptual importance metric, the image processor120reduces the range of costs that are considered for determining the confidence metric. More specifically, as described above, the image processor120may determine the costs for the purpose of determining a motion vector for a block. In doing so, the image processor determines costs within a certain area of the image (such as the search area306). Instead of using each of those costs for obtaining the confidence metric, the image processor120instead uses costs for a more restricted search area. In other words, according to this set of operations, the image processor120determines the motion vector for a block using costs from a larger area of the image and determines the confidence value using costs from a relatively smaller area of the image. This restricted area means that costs used for the confidence value would not be derived from high amplitude motion vectors. This set of operations accounts for the fact that objects having a high degree of motion are less perceptible to the human visual system.

According to another set of operations for obtaining the perceptual importance metric, the image processor120applies one or more linear or non-linear transformations to the confidence metric to obtain the perceptual importance metric. In an example, the transformations applied include one or more of scaling or shifting (i.e., applying an offset to) the confidence metric, applying a piecewise transformation function to the confidence metric, or applying look-up tables to the confidence metric. In an example, the transformation includes adding an offset and clamping the values to a range. This transformation is a non-linear transform that seeks to treat all values above a maximum the same and/or below a minimum the same, and to give some distinctive significance to values between the range. In the context of confidence metrics, this transformation has the effect of treating values that should be treated conceptually the same, but that might vary greatly numerically, the same. In an example, confidence values of 0.1% and 10% represent very low confidence values, but vary by a factor of 100 numerically. Despite this numerical disparity, it may be desirable to treat these confidence values the same. Any of the transformations described may be applied alone or together, and other transformations not listed may be applied alternatively or additionally.

According to another set of operations for obtaining the perceptual importance metric, the image processor120applies one or both of spatial or temporal filters. Applying spatial filters includes applying some spatial filtering function to multiple confidence values in a single frame. The result is that confidence values in a filtering area are affected by other confidence values in that filtering area. In an example, spatially filtering a particular confidence value includes multiplying the confidence value by a weight and adding weighted versions of neighboring confidence values. Any type of spatial filtering may be applied. Applying temporal filtering includes applying some temporal filtering to multiple confidence values for the same block or nearby blocks over different frames. The result is that confidence values are affected by confidence values of other frames. In an example, temporally filtering a particular confidence value includes multiplying the confidence value by a weight and adding weighted versions of confidence values forwards or backwards in time (subsequent or previous frames). Temporal filtering provides smoothing to reduce the visual impact of abrupt changes in perceptual importance values across different frames. In some examples, the image processor120prevents temporal filtering from occurring across scene change boundaries (which can be detected, for example, via an infinite impulse response (IIR) filter).

According to yet another set of operations for obtaining the perceptual importance metric, the image processor120modifies the confidence value based on one or more terms that take amplitude of motion into account. Higher motion reduces perceptual importance of a block because the human visual system is less able to perceive detail in fast moving subjects than in slow moving or stationary subjects. For this reason, the set of operations includes modifying the confidence value for a block based on the motion vector determined for that block. In various examples, modifying the confidence value based on one or more terms that take amplitude of motion into account include dividing the confidence value by one or more of the following terms:
|dx|+|dy|;
dx2+dy2; or
√{square root over (dx2+dy2)}.
where dx is the x component of the motion vector and dy is the y component of the motion vector determined for the block for which the perceptual importance metric is being determined. Although certain specific terms are shown, it should be understood that other terms not shown could be used alternatively or in conjunction with the terms provided.

FIGS. 4A-4Cillustrate an example sequence of images for which perceptual importance values are determined. In each of the images, a grid is shown. Each square of the grid represents a different block of pixels.FIG. 4Aillustrates a first image400(referred to as the “previous” image because it is prior to the image ofFIG. 4B) that includes horizon scenery, a barn, and a road with a vehicle on it.FIG. 4Billustrates a second image420(referred to as the “current image”) that has similar content as the first image, except that the camera has panned to the left, and thus the scenery has shifted to the right. In addition, the car has traveled down the road.

FIG. 4Cdepicts an overlay450of the previous image400and the current image420, and includes additional information related to the techniques for determining perceptual importance that are described herein. The previous image is illustrated with a dotted line and the current image is illustrated with a solid line. Aspects of the analysis described herein are illustrated.

Motion vectors (arrows originating at the center of each block) along with confidence values (numbers adjacent to the arrows) are illustrated for each block. As described elsewhere herein (e.g., with respect toFIG. 3), the motion vectors represent estimated motion from the previous image to the current image. The confidence values illustrate the confidence that the motion vectors are “correct,” and are generated according to any of the techniques described herein. The confidence values range from 1 to 10 for simplicity of illustration, but it should be understood that confidence values could be within any other numerical range.

The top section (top two rows of blocks) represents the sky. Motion vectors are illustrated, but the associated confidence values are low because of the uniformity among the blocks in the sky. In other words, it is difficult to select a direction associated with the best match because the visual features are uniform.

The next section down (next two rows of blocks) includes the horizon line with mountain detail. This section of the image is relatively detailed and thus the confidence of the motion vectors is high. Each of the motion vectors for the blocks including mountain detail has approximately the same direction and magnitude, and the confidence value for each such motion vector is illustrated as being 9. Several blocks in these two rows have little to no visual detail and thus have a much lower confidence value. The next row, below the two rows including the mountains, has little to no detail. Thus, the motion vectors for the blocks in this row are semi-random and have relatively low confidence values.

The three bottom rows include detailed features including the barn, the car, and the road. Thus the motion vectors for these blocks reflect the panning of the camera and also have high confidence values. Some blocks, such as those only including small details of the roads, have medium confidence values (e.g., values of 6). The blocks in the current image including the car have a different motion vector, reflecting a combination of the pan and the motion of the car. This motion vector has a greater amplitude than the motion vectors for other blocks, representing only panning of the image. Also, due to the distinctive visual features of the car, the motion vectors for the blocks including the car have a high confidence value.

The image processor120generates perceptual importance values for the blocks shown based on the confidence values and the motion vectors. In the particular technique illustrated, the image processor120performs the following operations. First, the image processor120performs a transform on the confidence value to obtain an intermediate perceptual importance value. The transform is a lookup table transform that groups different ranges of confidence values into different perceptual importance groups. For confidence values from 1-4, the image processor120assigns an intermediate perceptual importance value of “1,” for confidence values from 5-6, the image processor120assigns an intermediate perceptual importance value of “2,” and for confidence values from 7-9, the image processor120assigns an intermediate perceptual importance value of “3.” The image processor120weights these intermediate perceptual importance values based on the magnitude of the motion vector, decreasing the intermediate perceptual importance values as that magnitude increases (to account for the fact that the human visual system is less able to perceive detail on fast moving subjects than on slow moving or stationary subjects). This weighting is reflected in the blocks having the car, in the bottom right of the screen, which each has a confidence value of 9. The weighted confidence values are then sorted into bins: low, medium, and high perceptual importance values. The blocks with confidence values of 7-9 are assigned to the high perceptual importance value bin, except for the blocks having the car, which are assigned to the medium perceptual importance value bin, due to the motion-related weighting. The blocks having 5-6 confidence values are also assigned to the medium perceptual importance bin, and the blocks having 1-4 confidence values are assigned to the low perceptual importance bin. Although represented as conceptual values, the “high,” “medium,” and “low” perceptual importance values may be assigned numerical values, and these numerical values may be used in further operations. It should be understood thatFIGS. 4A-4Crepresent merely an example technique for deriving perceptual importance values based on motion estimation operations and that any technique consistent with the present disclosure may be used.

Once a perceptual importance metric is determined, the image processor120performs one or more operations based on the perceptual importance metric. One such operation includes assigning compression factors to different blocks of the image based on the perceptual importance metric. According to this operation, given a particular bitrate budget, the image processor120assigns bits to different blocks based on relative perceptual importance. In an example, blocks having a higher perceptual importance would be assigned more bits and blocks having a lower perceptual importance would be assigned fewer bits. In an example, the image processor120normalizes perceptual importance values to assign bitrate for each block. This normalization allows bits to be assigned proportionally to the relative perceptual importance values, which would allow, for example, an image with all high importance blocks or all low importance blocks to have an even distribution of bits among blocks but for an image that has different perceptual importance values for different blocks to have different bits assigned to different blocks. In an example of normalization, the image processor120would sum all numerical perceptual importance values for all blocks to obtain a total perceptual importance value. For each block, the image processor120would divide the perceptual importance value for that block by the total perceptual importance value and multiply the total bit budget for the image by the resulting number. Any other technique for assigning bits of a bitrate budget to blocks based on perceptual importance could alternatively or additionally be used. After bits of the bitrate budge are assigned to the different blocks, the image processor120compresses the different blocks utilizing the assigned number of bits for the compression of each block. This technique allows blocks having a higher perceptual importance to be encoded with more bits and blocks having a lower perceptual importance to be encoded with fewer bits, effectively redistributing encoding bits from less “important” areas to more “important” areas of the image (where, again, “important” areas are those considered to be highly discernably by the human visual system, such as stationary blocks with high levels of detail).

In another example, the image processor120uses perceptual importance in combination with other metrics to identify regions of interest for other operations. Several examples are now provided. In one example, perceptual importance is enhanced with the addition of flesh tone detection. More specifically, instead of modifying the number of bits assigned to particular blocks of an image based on perceptual importance, perceptual importance is first modified by a flesh tone metric (for example, indicating how similar the color of a particular block is to a flesh tone) and then the modified value is used to determine the number of bits to assign to a particular block of an image. In another example, face detection is used in conjunction with the perceptual importance to determine the number of bits to use to compress particular blocks of an image. As with the flesh tone detection technique, with the face detection technique, the perceptual importance value is further modified based on how likely a particular block is to be at least part of a face. Blocks with higher likelihood of being a face would be assigned more bits than blocks with a lower likelihood of being a face. In yet another example, other metadata that indicates the importance of objects is used to further modify the perceptual importance values (for example, game metadata identifying important game objects, or other types of metadata are used).

In yet another example, the image processor120uses perceptual importance to assist with graphics rendering. In an example, perceptual importance helps identify areas for greater rendering detail and areas for lower rendering detail, which allows for rendering resources to be focused on more important areas of an image, thereby improving efficiency. Once a frame is rendered, the image processor120would perform the above processing to determine a perceptual importance map. This map could be used to encode the image for a cloud gaming scenario (i.e., the encoded image being transmitted to a player). In addition, the perceptual importance map could be used to create a “rendering resolution map” to adjust the resolution of different areas of the image (using similar techniques as, or combined with, e.g., foveated rendering or checkerboard rendering) such that higher perceptual importance areas are given higher resolution and lower perceptual importance images are given lower resolution. This rendering resolution map would be applied to the next frame, as long as the next frame belong to the same “scene.” The rendering resolution map can be adjusted based on motion vectors so that the map accounts for the movement of visual features between frames. In other words, once the perceptual importance map is determined for a frame, the locations of the perceptual importance values are spatially shifted based on motion vectors for the frame to account for predicted motion between frames.

FIG. 5is a flow diagram of a method500for determining perceptual importance values for an image, according to an example. Although described with respect to the system shown and described with respect toFIGS. 1-3 and 4A-4C, it should be understood that any system configured to perform the method, in any technically feasible order, falls within the scope of the present disclosure.

The method500begins at step502, where the image processor120determines costs for a block of an image. As described elsewhere herein, the image processor120determines the cost for a block by identifying a plurality of candidate motion vectors, determining associated costs for each of the candidate motion vectors, and identifying a cost for the block by selecting one of the associated costs based on a cost selection criteria. In an example, the image processor120calculates a cost corresponding to a motion vector by comparing pixels of the block to pixels of a reference frame in an area indicated by the motion vector. In an example, the lowest cost represents the best matched motion vector, so the image processor120selects the motion vector associated with the lowest cost as the motion vector for the block. The image processor120identifies that lowest cost as the cost determined for step502.

As described above, the perceptual importance value may be determined based on a confidence that is determined based on all costs considered in determining the motion vector. Alternatively, the perceptual importance value may be determined based on a confidence that is determined based on a smaller set of costs than all those considered in determining the motion vector, such as a set including all costs for candidate motion vectors pointing to an area that is more restricted than the area covering the set of candidate motion vectors from which the cost is determined. Thus, at step504, if the image processor120is to determine the confidence values based on costs in a restricted search area, then the method500proceeds to step506where the image processor120selects costs from a more limited area than those used for motion vector determination; then the method500proceeds to step510. If at step504, the image processor120is to determine the confidence values based on costs in the same search area as that used for the motion vector, the method proceeds to step508. At step508, the image processor120selects the costs used for motion vector determination as the costs for determining the confidence value.

At step510, the image processor120determines the confidence value based on the selected costs. Any technically feasible technique for determining the confidence value could be used. Some examples, such as calculating ratios between different costs of candidate motion vectors as described elsewhere herein, could be used.

At step512, if the perceptual importance value is to be determined based on one or more transformations, then the method500proceeds to step514, and if the perceptual importance is not to be determined based on one or more transformations, then the method500proceeds to step516. At step514, the image processor120applies one or more transformations to the confidence value. Any technically feasible transformation may be applied, and some examples are described herein, for example with respect toFIG. 3.

At step516, if the perceptual importance value is to be determined based on one or more of temporal or spatial filters, then the method500proceeds to step518, where the image processor120applies one or more spatial and/or temporal filters to the confidence values, and if the perceptual importance value is not to be determined based on one or more of temporal or spatial filters, then the method500proceeds to step520. Some example techniques for applying spatial and/or temporal filters are described elsewhere herein, for example, in the description associated withFIG. 3, and generally include modifying confidence values based on those of neighboring blocks (spatial) or on blocks at different frames (temporal) in order to reduce abrupt changes in perceptual importance between frames or within frames.

At step520, if the perceptual importance value is to be determined based on motion vector magnitude, then the method500proceeds to step522and if the perceptual importance value is not to be determined based on motion vector magnitude, then the method500proceeds to step524. At step522, the image processor120modifies the confidence value based on motion vector magnitude, in order to give blocks with higher degrees of motion less perceptual importance than blocks with lower degrees of motion. Techniques described herein include modifying the confidence value by a term that includes aspects of the motion vector determined for the block for which perceptual importance is being determined, and are described with respect toFIG. 3.

The method500includes application of any combination of four sets of techniques (steps506,514,518, and522), and one, more than one, or none of those techniques may be used in converting the confidence value to a perceptual importance value. Steps504,512,516, and520are illustrated as decision blocks. Any of these decisions, which determine what contributes to the perceptual importance values, may be made at runtime during decoding or may represent different hard-coded configurations for the technique. In other words, method500may represent a set of different possible combinations of techniques for converting from confidence value to perceptual importance value.

At step524, the perceptual importance value is output. This perceptual importance value may be accumulated with other perceptual importance values into a perceptual importance map, which may be used for other purposes as described elsewhere herein, such as assigning bits of a bit budget to different blocks based on relative perceptual importance, determining region of interest of an image, or performing processing related to 3D rendering.