Retaining a surface detail

Among other disclosure, a computer-implemented method for retaining a surface detail includes identifying a surface that is to be used for generating an image in a rendering process. The surface includes polygons to be changed from an initial size to a larger size as the surface is changed to a lower resolution as part of tessellating the surface. The surface includes at least one surface detail smaller than the larger polygon size. The method includes storing information corresponding to at least a portion of the surface that includes polygons forming the surface detail. The method includes tesselating the surface, wherein the surface assumes the lower resolution. The method includes determining, while the image is at the lower resolution and using the stored information, a shading sample for at least one of the polygons of the larger size that includes the surface detail. The method includes storing the shading sample.

TECHNICAL FIELD

This document relates to image processing.

BACKGROUND

Computer-based images are used in many situations. For example, servers, personal computers and video game consoles all use digital images in one or more applications. In some situations, images are used in animation, such as when a motion picture is produced or a video game is operated. In the former example, it is usually acceptable that image rendering takes a certain amount of time, because the rendering is done in advance of presenting the actual motion picture. In the latter example, by contrast, images must be rendered essentially in real time if the game is to be interactive and provide a realistic experience for the user.

The resolution of an object in a computer-generated image changes as the object becomes smaller. For example, this happens as the object moves away from the viewer in screenspace. This can result in a loss of detail. For example, if the object had been provided with small-scale visual characteristics at the higher resolution (such as with a displacement map), such details can be averaged out or entirely disappear as the object gets smaller.

SUMMARY

The invention relates to retaining a surface detail.

In a first aspect, a computer-implemented method for retaining a surface detail includes identifying a surface that is to be used for generating an image in a rendering process. The surface includes polygons to be changed from an initial size to a larger size as the surface is changed to a lower resolution as part of tessellating the surface. The surface includes at least one surface detail smaller than the larger polygon size. The method includes storing information corresponding to at least a portion of the surface that includes polygons forming the surface detail. The method includes tesselating the surface, wherein the surface assumes the lower resolution. The method includes determining, while the image is at the lower resolution and using the stored information, a shading sample for at least one of the polygons of the larger size that includes the surface detail. The method includes storing the shading sample.

Implementations can include any, all or none of the following features. The method can further include generating a map of where on the surface the surface detail is located; and storing the map separate from the surface. The method can further include accessing the map before storing the information; and identifying, using the accessed map, the information to be stored. The method can further include accessing the map after storing the information and before determining the shading sample; and identifying, using the accessed map, the stored information to be used in determining the shading sample. The method can further include defining a boundary that encloses the at least one of the polygons of the larger size; and using the boundary in accessing the stored information for determining the shading sample. The method can further include modifying at least one of a size and a shape of the boundary before using the boundary. The method can further include compressing the stored information; storing the compressed stored information in a hierarchy of multidimensional tree structures; and accessing the compressed stored information in the hierarchy of multidimensional tree structures as part of using the stored information in determining the shading sample. At least the compressing, storing of the compressed stored information and accessing can be performed in a real-time image generation process. The method can further include selecting the hierarchy of multidimensional tree structures before the storing of the compressed stored information based on at least one of a screen extent of the surface and available bandwidth. The compression can include at least one of: merging similarly oriented polygons; culling points where the surface contains no detail; and combinations thereof. The method can further include determining where the surface contains no detail by identifying where normals of the polygons satisfy a similarity criterion compared with a normal of the surface. The method can further include determining probability terms for each of multiple regions in the hierarchy to indicate a likelihood that the region should produce reflected light; and using the probability terms in accessing the compressed stored information. The method can further include determining a variation of surface orientation in the similarly oriented polygons; and taking the variation of surface orientation into account in determining the probability terms. The method can further include providing the surface at the lower resolution with the polygons forming the surface detail, wherein the polygons forming the surface detail are used in determining the shading sample.

In a second aspect, a computer program product is tangibly embodied in a computer-readable medium and includes instructions that when executed by a processor perform a method for retaining a surface detail. The method includes identifying a surface that is to be used for generating an image in a rendering process. The surface includes polygons to be changed from an initial size to a larger size as the surface is changed to a lower resolution as part of tessellating the surface. The surface includes at least one surface detail smaller than the larger polygon size. The method includes storing information corresponding to at least a portion of the surface that includes polygons forming the surface detail. The method includes tesselating the surface, wherein the surface assumes the lower resolution. The method includes determining, while the image is at the lower resolution and using the stored information, a shading sample for at least one of the polygons of the larger size that includes the surface detail. The method includes storing the shading sample.

In a third aspect, a system includes a polygon managing module providing a surface that is to be used for generating an image in a rendering process. The surface includes polygons to be changed from a current size to a larger size as the surface is changed to a lower resolution as part of tessellating the surface. The surface includes at least one surface detail smaller than the larger polygon size. The system includes a cache that is provided with information corresponding to at least a portion of the surface that includes polygons forming the surface detail. The system includes a light module that determines, while the image is at the lower resolution and using the cached information, a shading sample for at least one of the polygons of the larger size that includes the surface detail.

Implementations can include any, all or none of the following features. The system can further include a map of where on the surface the surface detail is located, stored separate from the surface. The system can be configured for use in a real-time image generation process, wherein the system compresses the cached information, stores the compressed cached information in a hierarchy of multidimensional tree structures, and accesses the compressed cached information in the hierarchy of multidimensional tree structures as part of using the cached information in determining the shading sample. The system can further include a rendering module providing a rendered version of the surface generated using at least the shading sample.

Advantages of implementations can include any, all or none of the following. An improved image rendering process can be provided. Surface details can be retained when an object becomes smaller in screenspace. Image rendering in real-time implementations can be provided with retention of surface details.

DETAILED DESCRIPTION

FIGS. 1A-Eshow a scene100that is part of generating an image in an animation process. The scene100schematically represents a computer-based image generation process. Ultimately, the image will be rendered into a viewable image and used for some purpose, such as to create an animated motion picture or the output of an interactive video game.

As shown inFIG. 1A, the scene100includes a surface102that is to be at least partially visible in the image. The surface can, for example, represent part of a character, object, landscape or other image feature. The portion of the surface102that is visible is determined by the position of a virtual camera104. The camera104is not to be visible in the image and it is not part of the image. Rather, the camera104represents the elevation, distance and angle from which the surface102is to be seen in the finished image. That is, an observer of the finished image will see the surface102from the same viewpoint as does the camera104.

The view of the camera104in this example is schematically illustrated using arrows106A and106B. Arrow106A shows the direction from which the camera104is aimed at the surface102. Arrow106B, in turn, shows that the camera's view reaches further than just the point where it intersects the surface102. This corresponds to light reflection in the surface102. That is, incoming light from a remote light source (not shown) that travels toward the surface along the direction of the arrow106B, will reflect upon reaching the surface. Some or all of the reflected light will then continue along the direction of the arrow106A toward the camera104. In some implementations, light characteristics can change as part of the reflection; for example, intensity and spectral distribution of the reflected light that hits the camera can be different from the incoming light. These and other changes depend on the nature of the surface102, for example.

Light that impinges on the camera104will be registered as being part of the image. That is, the image as seen by the camera104includes not only the visual components of the surface102itself, but also contributions from light reflected on this surface. In other words, the camera registers the reflection seen on the surface.

The scene100here includes another light source108, located approximately directly above the point where the arrow106A impinges on the surface102. The light source108generates outgoing light in one or more directions. For example, the light source can represent the sun or an object that is illuminated by the sun.

The camera104does not see the light source108in this example. In other words, the light from the light source that reflects on the surface102does not take the direction of the arrow106A. This is because the light from the light source does not impinge on the camera along the direction of the arrow106A. The reason for this, in turn, is that the surface102in the present example is relatively flat and does not contain any surface irregularities or other features. That is, the orientation of the surface102at all points is essentially perpendicular to a normal110of the surface102. This means that no contribution from the light source108will be registered as part of the image according to the configuration shown inFIG. 1A.

However, if the surface102were provided with one or more surface details such that some of its points were oriented in a different direction relative to the normal110, the camera104could see a contribution from the light source108.FIG. 1Bshows one such example. The surface has here been provided with a surface detail112that includes a slight indentation. For example, the surface detail112can represent a characteristic in the skin of a character, a property of the material that makes up an inanimate object, or a detail on a ground surface. In computer-based image generation, features such as the surface detail112can be used to make a surface more realistic. The surface detail112is sometimes considered to be a small-scale bump or scratch. The size, shape and orientation of the surface detail112are only exemplary and many other sizes, shapes and/or orientations can be used.

The surface102is here made up of polygons114situated next to each other. For the most part, the polygons are oriented perpendicularly to the normal110, as mentioned earlier. However, polygons114A that make up the surface detail112can have different orientations. Here, the surface detail is formed by three polygons114A, of which the outer two are slanted relative to the normal110and the third middle one is approximately perpendicular to the normal, albeit at a different level than the rest of the polygons114. The surface detail112can be provided by displacing those of the polygons114that occupy the area where the surface detail should be located, thus forming the polygons114A, or by removing those original ones of the polygons114and replacing them with the polygons114A, to name two examples.

The differently oriented polygons can provide light contributions to the image that are not obtained with the other polygons114. For example, one or more of the outer polygons114A can reflect light from the light source108such that it travels along the direction of the arrow106A and thus is registered by the camera104as being included in the image. This is illustrated by an arrow106C that represents the possibility that the camera will see the light source108as a reflection in at least part of the surface detail112. This can be considered a glint of light that appears in the surface detail112.

The polygons114and114A are of the same size in the present example. Moreover, all of the polygons are relatively small and there is a relatively high number of polygons forming the surface. The polygon sizes correspond to the situation that the image here has a relatively high resolution. For example, this can occur when the character, item or landscape feature having the surface102is relatively close to the viewer, and therefore occupies a substantial portion of the image.

However, resolution can change over time, depending on surface location and other circumstances. Foe example, when tessellating the surface102, the polygons will be made larger than their present size. Stated another way, two or more polygons having a relatively smaller size can be replaced with one relatively larger polygon when tessellation is performed. When the surface102recedes away from the viewer in screenspace, its resolution will be decreased so that it consists of fewer polygons.FIG. 1Cshows an example of the surface after performing tessellation, when the surface has assumed the lower resolution. Here, there are fewer polygons than before and each of them is larger. That is, the surface102is now made up of polygons116having a larger size than the polygons114and114A.

The change of the surface102from the higher resolution to the lower resolution can cause the surface details112to be lost. That is, a map or function that defines the location and appearance of the surface detail can be filtered as the surface102becomes smaller in screenspace, and the result can be to partially or completely average out the details. This causes the surface to be smoothened and any glints of light to be lost. If the image were to be rendered in the condition represented by the larger polygons116, the camera102(and therefore the viewer) would not see the light source108.

In some implementations, the surface detail112can be retained at the lower resolution using stored information. For example, the polygons114A corresponding to the surface detail112can be stored while the surface is at the higher resolution and thereafter be provided to the surface at the lower resolution.FIG. 1Dshows an example of this. Here, a boundary118has been defined that encloses one of the larger polygons116. In some implementations, the boundary118represents the area in which to determine whether there existed any additional surface detail at the higher resolution that has now been replaced with just the one larger polygon116. In this example, the surface detail112existed, at the higher resolution, within the area defined by the boundary118. The surface can therefore be provided with the polygons114A to replace the larger polygon.

With the different orientations of the smaller polygons114A, the camera102can again see the light source108, as indicated by the arrow106C. This means that a shading sample generated for one of the larger polygons116that includes the smaller polygons114A can be generated so that the shading sample includes a light contribution from the surface detail. The light contribution from the surface detail can be determined by looking up in the cache based on the larger polygon116that is currently being processed. The light contribution from the surface detail thus determined can be combined with a remainder of the light contribution for the larger polygon. For example, the surface detail contribution can be averaged with the remainder contribution to generate an overall shading sample (e.g., a color value) for the larger polygon. The surface detail can also provide glints from one or more other directions, as schematically illustrated by arrows106D.

Accordingly, the image will now include details visible in the surface that were not present when only the larger polygons116were used. A rendered version of the image can then be generated based on the generated shading samples, which rendered version will show the light contributions (e.g., the glints) from one or more restored surface details.

The boundary118can be provided with different size, shape and/or orientation. This can reduce or eliminate the risk that irrelevant surface detail is inadvertently encompassed by the boundary118.FIG. 1Eshows an example where the object that is enclosed by the surface102is relatively thin. For example, the object represents a thin wall, plate of glass or a membrane. This means that not only the upper surface of the object can be visible (represented by large-size polygons116), but also a lower surface. The lower surface is here defined by large-size polygons120. Each of the polygons116and120may be located at a position where the surface had surface details at a higher resolution.

Assume, for example, that large-size polygon120A is one such polygon; e.g., the surface initially had a bump or scratch also on the lower surface that has now been lost at the lower resolution. If the boundary118were to be applied, as part of restoring lost surface detail to a polygon116A, with the size and shape the boundary had inFIG. 1D, there is a chance that previous surface detail corresponding to the large-size polygon120A (which is located nearby the polygon116A) would inadvertently be included in such detail restoration.

To counteract this and/or other problems, the size and/or shape of the boundary118can be modified before the surface detail is restored. InFIG. 1Eit is seen that the boundary118has been provided with a flatter shape than before. Any kinds of size and/or shape adjustments can be made.

FIG. 2shows an example of a system200that can be used for performing the processes described herein, to name some examples. The system includes a computer device202which can be any processor-based device, such as a server computer, a personal computer or a video game console. The computer device can be connected to one or more input devices204, such as a keyboard, mouse or a game control. The computer device can provide output to one or more output devices206, such as a display screen, a printer or a television set. For example, the system200can be used for generating images for an animated motion picture or for an interactive video game.

The computer device202here includes a polygon managing module208that manages polygons as part of an image-generating process. For example, the polygons shown in any or all ofFIGS. 1A-Ecan be managed.

The computer device202here includes a light module210that can determine light contributions to an image and generate one or more shading samples based on the light contribution. For example, light contributions to the image captured by the camera104can be determined for each large-size polygon, and a corresponding shading sample can be stored.

The computer device202here includes a rendering module212that renders the image so that it can be seen by the viewer. The rendering module212can receive the shading samples as input and use them in the rendering. For example, a frame for an animated motion picture or a frame of an interactive game can be rendered.

The computer device202here includes a cache214that can be used for storing information at a higher resolution. For example, any or all surface detail information inFIGS. 1A-Ecan be stored in the cache212. Any kind of computer-readable storage device can be used for the cache214. For example, a point cloud cache can be defined and used in an implementation based on the shading language compatible with the PRMan product from Pixar Animation Studios. As another example, a multidimensional tree structure can be used.

The polygon managing module208can work with one or more models216as part of managing polygons. That is, the model216can define the geometry of a scene, such as any of the scenes inFIGS. 1A-E, and the module208can generate and manage the polygons making up such geometry through various object movements and transformations as required.

A resolution module218indicates the resolution of a surface at a given point. For example, the resolution inFIG. 1Bis relatively high and the resolution inFIGS. 1C-Eis relatively low. The sizes of the polygons and the resolution can vary in proportion to each other, as has been mentioned.

The computer device202here includes a map220of where the surface detail is located. The map220is stored separate from the surface. The map220can for example be used to ensure that only the surface detail (and not areas without surface detail) are cached. As another example, when the entire surface has been cached (i.e., also areas without surface detail), the map can be used to ensure that the lower-resolution surface is only provided with cached information for areas where surface detail actually occurs.

A boundary module222can provide a boundary for determining presence or absence of surface detail. For example, the boundary118as shown inFIGS. 1D-Ecan be provided. Resizing and/or reshaping of the boundary can be provided.

In some implementations, for example those intended to be used for a real-time image generation process, such as in an interactive video game, the computer device202can be provided with one or more hierarchies224of multidimensional tree structures226. The structures226can be used to read from the cache214, such as from a point cloud cache. The hierarchies224can receive information from the cache214that has been compressed, for example. Any compression technique can be used, in some implementations. Such compressed information in the hierarchy224can then be accessed as part of using the cached information, relating to the surface detail(s), in determining a light contribution. Any kind of multidimensional tree structure can be used. One or more multidimensional tree structures can be used also in implementations that are not configured for real-time image generation.

The computer device202can include a probability module228that can determine probability terms to be used in accessing cached information. For example, the probability terms can be used to indicate a likelihood that a particular surface detail should produce a glint of reflected light. This can reduce the chance that too many and/or too strong glints are produced.

The organizations of components in the computer device202is exemplary only. In other implementations, two or more components can be united into a common component. Also, one or more of the components can provide substantially similar functionality while being located at another device (not shown).

FIG. 3is a flow chart showing an exemplary method300. Some or all of the method300can be performed in an image-generating process. Any or all steps of the method300can be performed by a processor executing instructions stored in a computer-readable storage medium, for example in the system200.

The method300can begin in step302with identifying a surface that is to be used for generating an image in a rendering process. The surface includes polygons to be changed from an initial size to a larger size as the surface is changed to a lower resolution as part of tessellating the surface. The surface includes at least one surface detail smaller than the larger polygon size. For example, the surface102as shown inFIG. 1Bcan be identified.

In step304, the method300can include generating a map of where on the surface the surface detail is located. For example, the map220can be generated to indicate the surface detail112.

In step306, the method300can include storing the map separate from the surface. For example, the polygon managing module208can store both the map220and the surface102.

Steps308and310can be considered a caching-using-map approach. In step308, the method300can include accessing the map before storing the information about a surface detail. For example, the map220can be accessed before information is stored in cache214. In step310, the method300can include identifying, using the accessed map, the information to be stored. For example, only the surface detail information indicated by the map220is stored in cache214.

In step312, the method300can include storing information corresponding to at least a portion of the surface that includes polygons forming the surface detail. For example, information about some or all of the surface102, including the surface detail112, can be stored in the cache214.

Steps314and316can be considered an accessing-cache-using-map approach. In some implementations, the steps314and316are alternative to the steps308and310. In step314, the method300can include accessing the map after storing the information and before determining the light contribution. In step316, the method300can include identifying, using the accessed map, the stored information to be used in determining the light contribution. For example, only the surface detail information in cache214, as indicated by the map220, is accessed.

In step318, the method300can include tessellating the surface to the lower resolution. For example, the surface102can change from the resolution inFIG. 1Bto the resolution inFIG. 1Cas a result of tessellation. The tessellation can be performed by the polygon managing module208.

In step320, the method300can include modifying at least one of a size and a shape of the boundary before using the boundary. For example, the boundary module222can modify the boundary118.

In step322, the method300can include defining a boundary that encloses at least one of the polygons at the larger size. For example, the boundary188as shown inFIGS. 1Dand/or1E can be defined.

In step324, the method300can include using the boundary in accessing the stored information for determining the light contribution. For example, the boundary118can be used to obtain surface detail information for the large-size polygon116A from the cache214.

Steps326-342can, among other examples, be used in implementations for more demanding processing, such as the rapid image generation and rendering necessary for real-time interactive operations. In step326, the method300can include selecting a hierarchy of multidimensional tree structures before the storing of the compressed stored information based on at least one of a screen extent of the surface and available bandwidth. For example, the hierarchy224can be chosen so that it fits the surface102and/or based on the bandwidth available in the system200or elsewhere.

In step328, the method300can include compressing the stored information. Compressing can include merging similarly oriented polygons, culling points where the surface contains no detail, and combinations thereof, to name a few examples. For example, those of the large-size polygons116that are similarly oriented can be merged. For example, some of the polygons114outside the surface detail112can be culled. The computer device202can compress information in the cache214.

In step330, the method300can include storing the compressed stored information in a hierarchy of multidimensional tree structures. For example, the hierarchy224can be used.

In step332, the method300can include accessing the compressed stored information in the hierarchy of multidimensional tree structures as part of using the stored information in determining the light contribution. For example, the multidimensional tree structures226can be accessed in a real-time implementation. While steps328-332in this example are described in a real-time implementation, it is noted that multidimensional tree structures can be used in non-real-time implementations.

In step334, the method300can include determining where the surface contains no detail by identifying where normals of the polygons satisfy a similarity criterion compared with a normal of the surface. For example, normals of the polygons114can be compared with the surface normal110according to any similarity criterion, such as whether the normals do not differ significantly.

In step336, the method300can include determining a variation of surface orientation in the similarly oriented polygons. For example, the variation of surface orientation of some of the large-size polygons116can be determined.

Steps338and340relate to determining probability terms. In step344, the method300can include taking the variation of surface orientation into account in determining the probability terms. In step346, the method300can include determining probability terms for each of multiple regions in the hierarchy to indicate a likelihood that the region should produce reflected light. For example, the probability module228can be used.

In step342, the method300can include using the probability terms in accessing the compressed stored information. For example, the probability terms can regulate the number and/or strength of glints produced.

In step344, the method300can include providing the surface at the lower resolution with the polygons forming the surface detail, wherein the polygons forming the surface detail are used in determining the light contribution. For example, in the surface102shown inFIG. 1D, one or more of the larger polygons116can be replaced with the polygons114A.

In step346, the method300can include determining, while the image is at the lower resolution and using the stored information, a shading sample for at least one of the polygons of the larger size that includes the surface detail. For example, the light module210can determine a glint visible to the camera104as indicated by the arrow106C.

In step348, the method300can include storing the shading sample. For example, the rendering module212can render an image that includes the surface102at a lower resolution, wherein the light contribution of the light source108is present due to the surface detail112.

The order and description of the steps in the method300are exemplary only. More or fewer steps, and/or performed in different order, can be used in some implementations.

FIG. 4is a schematic diagram of a generic computer system400. The system400can be used for the operations described in association with any of the computer-implement methods described previously, according to one implementation. The system400includes a processor410, a memory420, a storage device430, and an input/output device440. Each of the components410,420,430, and440are interconnected using a system bus450. The processor410is capable of processing instructions for execution within the system400. In one implementation, the processor410is a single-threaded processor. In another implementation, the processor410is a multi-threaded processor. The processor410is capable of processing instructions stored in the memory420or on the storage device430to display graphical information for a user interface on the input/output device440.

The storage device430is capable of providing mass storage for the system400. In one implementation, the storage device430is a computer-readable medium. In various different implementations, the storage device430may be a floppy disk device, a hard disk device, an optical disk device, or a tape device.

The input/output device440provides input/output operations for the system400. In one implementation, the input/output device440includes a keyboard and/or pointing device. In another implementation, the input/output device440includes a display unit for displaying graphical user interfaces.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.