Source: http://www.google.fr/patents/US8144153
Timestamp: 2013-05-22 01:32:59
Document Index: 243210801

Matched Legal Cases: ['art 1600', 'art 1700', 'art 1600', 'Application No. 561570', 'Application No. 561570', 'Application No. 581496', 'Application No. 582356', 'application No. 2009240847']

Brevet US8144153 - Model production for animation libraries - Google�BrevetsRecherche Images Maps Play YouTube Actualit�s Gmail Drive Plus » Recherche avanc�e dans les brevets | Historique Web | Connexion Recherche avanc�e dans les brevets BrevetsA computer-implemented method includes selecting a subset of images from a set of captured images. A surface feature of one object is represented in the content of each selected subset image. The method also includes decomposing the surface feature content of each selected image to produce a model of...http://www.google.fr/patents/US8144153?utm_source=gb-gplus-shareBrevet US8144153 - Model production for animation libraries Num�ro de publicationUS8144153 B1Type de publicationOctroi Num�ro de demande11/943,556 Date de publication27 mars 2012 Date de d�p�t20 nov. 2007 Date de priorit�20 nov. 2007 InventeursSteve SullivanFrancesco Callari Cessionnaire d'origineLucasfilm Entertainment Company Ltd. Classification aux �tats-Unis345/473345/585345/419 Classification internationaleG06T13/00 Classification coop�rativeG06T2207/30201G06T2207/30204G06T7/2046G06T13/40 Classification europ�enneG06T 13/40R�f�rencesCitations de brevets (71)Citations hors brevets (32) R�f�renc� par (2)Liens externesUSPTO Cession USPTO EspacenetModel production for animation librariesUS 8144153 B1 R�sum� A computer-implemented method includes selecting a subset of images from a set of captured images. A surface feature of one object is represented in the content of each selected subset image. The method also includes decomposing the surface feature content of each selected image to produce a model of representations of the object.
selecting, in a processor, a subset of images from a set of captured images, wherein a surface feature of a first object is represented in content of the selected image subset;
producing a model, at a first time, of at least one representation of the first object by decomposing the content of the selected image subset into principal components;
using the model to track surface features in at least one image that is separate from the selected image subset used to produce the model, wherein tracking the surface features provides an estimate of the surface features represented in the at least one image that is separate from the selected image subset used to produce the model; and
combining with the model, at a second time, content that represents the estimated surface features from the tracked surface features of the at least one separate image by decomposing content into principal components to produce an enhanced model of additional representations.
storing the enhanced model in a library of models.
3. The computer-implemented method of claim 1, in which combining the content includes decomposing the content with a linear transformation.
4. The computer-implemented method of claim 3, in which decomposing the content with a linear transformation includes computing principle components.
5. The computer-implemented method of claim 1, in which the image subset selection is based upon the surface feature represented in the content of the subset images.
6. The computer-implemented method of claim 1, in which the at least one separate image includes content that represents a second object, different from the first object.
7. The computer-implemented method of claim 1, in which the at least one separate image includes content that represents the first object.
8. The computer-implemented method of claim 1, in which the content is represented in a mesh.
9. The computer-implemented method of claim 1, in which the surface feature includes an artificial surface feature.
10. The computer-implemented method of claim 1, in which the surface feature includes a natural surface feature.
a memory for storing instructions of a model initiator and a model updater;
the model initiator to select a subset of images from a set of captured images, wherein a surface feature of a first object is represented in content of the selected image subset, the model initiator also decomposing the content of the selected image subset into principal components to produce a model, at a first time, of at least one representation of the first object; and
the model updater to use the model to track surface features in at least one image that is separate from the selected image subset used to produce the model, wherein tracking the surface features provides an estimate of the surface features represented in at least one image that is separate from the selected image subset used to produce the model, the model updater is further configured to combine, at a second time, the model and content that represents the estimated surface features from the tracked surface features of the at least one separate image by decomposing content into principal components to produce an enhanced model of additional representations.
12. The system of claim 11, in which the model initiator is capable of storing the model in a library of models and the model updater is capable of storing the enhanced model in the library of models.
13. The system of claim 11, in which the model updater is configured to compute a linear decomposition to combine the content with the model.
14. The system of claim 13, in which the linear decomposition computation includes computing principle components.
15. The system of claim 11, in which the image subset selection is based upon the surface feature represented in the content of the subset images.
16. The system of claim 11, in which the at least one separate image includes content that represents a second object, different from the first object.
17. The system of claim 11, in which the at least one separate image includes content that represents the first object.
18. A computer program product tangibly embodied in a storage device and comprising instructions that when executed by a processor perform a method comprising:
selecting a subset of images from a set of captured images, wherein a surface feature of a first object is represented in content of the selected image subset;
producing a model, at a first time, of at least one representation of the first object by decomposing content of the selected image subset into principal components;
using the model to track surface features in at least one image that is separate from the selected image subset used to produce the model, wherein tracking the surface features provides an estimate of the surface features represented in at least one image that is separate from the selected image subset used to produce the model; and
combining, at a second time, the model and content that represents the estimated surface features from the tracked surface features of the at least one separate image by decomposing content into principal components to produce an enhanced model of additional representations.
19. The computer program product of claim 18, further comprising instructions that when executed by a processor perform a method comprising:
20. The computer program product of claim 18, in which combining the content includes decomposing the content with a linear transformation.
21. The computer program product of claim 20, in which decomposing the content with a linear transformation includes computing principle components.
22. The computer program product of claim 18, in which the image subset selection is based upon the surface feature represented in the content of the subset images.
23. The computer program product of claim 18, in which the at least one separate image includes content that represents a second object, different from the first object.
24. The computer program product of claim 18, in which the at least one separate image includes content that represents the first object.
25. The computer program product of claim 18, in which wherein the content is represented in a mesh. Description
TECHNICAL FIELD This document relates to initiating the production of models for animation libraries.
BACKGROUND Computer-based animation techniques often involve capturing a series of images of an actor (or other object) with multiple cameras each having a different viewing perspective. The cameras are synchronized such that for one instant in time, each camera captures an image. These images are then combined to generate a three-dimensional (3D) graphical representation of the actor. By repetitively capturing images over a period of time, a series of 3D representations may be produced that illustrate the actor's motion (e.g., body movements, facial expressions, etc.).
To produce an animation that tracks the actor's motion, a digital mesh may be generated from the captured data to represent the position of the actor for each time instance. For example, a series of digital meshes representing an actor's face may be used to track facial expressions. To define mesh vertices for each motion, markers (e.g., make-up dots) that contrast with the actor's skin tone may be applied to the actor's face to provide distinct points and highlight facial features.
SUMMARY The systems and techniques described here relate to producing models for animation libraries.
In one implementation, images of an actor's performance are captured and a subset of representative images are selected to produce a model. From the model, representations (e.g., body movements, facial expressions, etc.) included in the images, or similar representations of the actor may be reproduced. The model may also be progressively refined by estimating representations that may be present in additional images. The model also be used as a basis for other models. For example, estimated representations of another actor's or an actress's performance may be added to the model to produce another model that may be used for reproducing more representations (e.g., facial expressions).
In one aspect, a computer-implemented method includes selecting a subset of images from a set of captured images. A surface feature of one object is represented in the content of the selected image subset. The method also includes producing a model of at least one representation of the object by using the surface feature content of the selected image subset. The method further includes using the model to estimate surface features represented in at least one image that is separate from the selected image subset used to produce the model. The method also include combining the estimated surface feature content of the at least one separate image with the model to produce an enhanced model of additional representations.
Implementations may include any or all of the following features. The computer-implemented method may also include storing the enhanced model in a library of models. Combining the content may include decomposing the content with a linear transformation such as computing principle components. Image subset selection may be based upon the surface feature represented in the content of the subset images. One or more of the separate images may include content that represents another object, which may be different from the first object or the same. The surface feature content may be represented in a mesh. The surface feature may include an artificial or natural surface feature.
In another aspect, a system includes a computer system that includes a model initiator to select a subset of images from a set of captured images. A surface feature of an object is represented in the content of the selected image subset. The model initiator also uses the surface feature content of the selected image subset to produce a model of at least one representation of the object. The system also includes a model updater to use the model to estimate surface features represented in at least one image that is separate from the selected image subset used to produce the model. The model updater is further configured to combine the estimated surface feature content of the one or more separate images with the model to produce an enhanced model of additional representations. Implementations may include any or all of the following features. The model initiator may be capable of storing the model in a library of models and the model updater may be capable of storing the enhanced model in the library of models. The model updater may be configured to compute a linear decomposition, such as computing principle components, to combine the content with the model. Image subset selection may be based upon the surface feature represented in the content of the subset images. One or more of the separate images may include content that represents another object or the same object.
In still another aspect, a computer program product tangibly embodied in an information carrier includes instructions that when executed by a processor perform a method that includes selecting a subset of images from a set of captured images. A surface feature of an object is represented in the content of the selected image subset.
The method also includes producing a model of at least one representation of the object by using the surface feature content of the selected image subset. The method further includes using the model to estimate surface features represented in one or more images separate from the selected image subset used to produce the model. The method also includes combining the estimated surface feature content of the one or more separate images with the model to produce an enhanced model of additional representations.
Implementations may include any or all of the following features. The method may also include storing the enhanced model in a library of models. Combining the content may include decomposing the content with a linear transformation such as by computing principle components. Image subset selection may be based upon the surface feature represented in the content of the subset images. At least one separate image may include content that represents another object or the same object. The surface feature content may be represented in a mesh.
DESCRIPTION OF DRAWINGS FIG. 1 is a diagram of an exemplary motion capture system.
FIGS. 3A-D includes a shape mesh, an image, a motion mesh overlaying the image and the motion mesh.
FIGS. 4A-C includes an animation mesh overlaying a captured image, the animation mesh and a rendering of the animation mesh.
FIGS. 6A-C include a portion of a motion mesh, a portion of an animation mesh and a portion of a shape mesh.
FIG. 11 is a diagram that illustrates initiating production of the motion model.
FIG. 12 is a diagram that illustrates portions of a model initiator.
FIG. 13 is a diagram that illustrates enhancing a previously produced motion model.
FIG. 14 is a diagram that illustrates portions of a model updater.
FIG. 15 is a diagram that illustrates producing one motion from another motion model.
FIG. 16 is a flow chart of operations of a model initiator.
FIG. 17 is a flow chart of operations of a model updater.
DETAILED DESCRIPTION Referring to FIG. 1, a motion capture system 100 includes a group of cameras 102 a-e that are capable of capturing images of an actor's face 104 or other type of deformable object. To highlight facial features, a series of markers 106 (e.g., makeup dots) are applied to the actor's face 104. Dependent upon lighting conditions and the facial expressions to be captured, the markers 106 may be distributed in various patterns. For example, the markers may be uniformly distributed across the face 104 or some of the markers may be concentrated in particular areas (e.g., corners of the mouth) that tend to deform with detailed shapes for many facial expressions. Along with artificial highlight points (e.g., markers 106), natural points of the actor's face 104 may be used to represent facial surface features. For example, the texture of the actor's face may provide distinct features. Contours, curves, or other similar types of shapes in an actor's face may also represent facial features. For example, the contour of a lip, the curve of an eyebrow or other portions of an actor's face may represent useful features.
The cameras 102 a-e are temporally synchronized such that each captures an image at approximately the same time instant. Additionally, the cameras 102 a-e are spatially positioned (in know locations) such that each camera provides a different aspect view of the actor's face 104. In this illustration, the cameras are arranged along one axis (e.g., the �Z� axis of a coordinate system 108), however, the cameras could also be distributed along another axis (e.g., the �X� axis or the �Y� axis) or arranged in any other position in three dimensional space that may be represented by the coordinate system 108. Furthermore, while cameras 102 a-e typically capture optical images, in some arrangements the cameras may be capable of capturing infrared images or images in other portions of the electromagnetic spectrum. Thereby, along with optical cameras, infrared cameras, other types of image capture devices may be implemented in the motion capture system 100. Cameras designed for collecting particular types of information may also be implemented such as cameras designed for capturing depth information, contrast information, or the like. Image capturing devices may also be combined to provide information such as depth information. For example, two or more cameras may be bundled together to form an image collection device to capture depth information.
A motion mesh generator 118 is also executed by the computer system 112 to produce relatively lower resolution meshes that represent the position of the markers as provided by images 100 a-e. As described in detail below, these meshes (referred to as motion meshes) track the movement of the markers 106 as the actor performs. For example, the actor may produce a series of facial expressions that are captured by the cameras 102 a-e over a series of sequential images. The actor may also provide facial expressions by delivering dialogue (e.g., reading from a script) or performing other actions associated with his character role. While the actor changes expressions (and when first provided), the markers 106 may change position. By capturing this motion information, the facial expressions may be used to animate a computer-generated character. However, the resolution of the motion meshes is dependent upon the number of markers applied to the actor's face and the image capture conditions (e.g., lighting), for example. Similarly, shape meshes may be produced by the shape mesh generator 116 that represent the shape of the facial expressions over the actor's performance.
Referring to FIG. 2, a series of images 200 a-e respectively captured by cameras 102 a-e are illustrated. By temporally synchronizing the cameras 102 a-e, corresponding images may be captured during the same time instance. For example, each of the images labeled �T=1� may have been captured at the same time. The images 200 a-e are provided to the computer system 112 for processing by the shape mesh generator 116, the motion mesh generator 118, the animation mesh generator 120 and the motion transferor 122. For processing, the content of the images captured at the same time instance may be combined. For example, the shape mesh generator 116 may use stereo reconstruction (or other similar methodology) to construct a 3D shape mesh 202 for each time instance from the corresponding captured images. Generally each shape mesh has a relatively high resolution and provides a detailed representation of the shape of the captured object (e.g., the actor's face 104). As shown in FIG. 3A, an exemplary shape mesh 300 illustrates the shape (e.g., an actor's facial expression) produced from images (e.g., images 110 a-e) captured at the same time instant. While large in number, the vertices of the shape mesh 300 may not be distinguishable (compared to the markers) and may not be quantified by a coordinate system. As such, the motion of individual vertices may not be tracked from one shape mesh (e.g., T=1 shape mesh) to the next sequential shape meshes (e.g., T=2 shape mesh, . . . , T=n shape mesh).
Returning to FIG. 2, the motion transferor 122 includes a motion linker 208 that transfers the motion information of the motion meshes 204 to the animation mesh 206. Additionally, the motion linker 210 may use shape information from the shape meshes 202 to transfer the motion information to the animation mesh 206. In this example, the motion is transferred to a single animation mesh, however, in some implementations the motion may be transferred to multiple animation meshes. As illustrated in FIG. 5, the motion information associated with a sequence of motion meshes 500 a-d is transferred to the animation mesh 400. Additionally, a corresponding sequence of shape meshes 502 a-d provides shape information that may be used in the motion transfer. To transfer the motion, the position of each vertex may be mapped from the motion meshes 500 a-d to the animation mesh 400. For example, the position of each vertex included in motion mesh 500 a (and associated with time T=1) may be transferred to appropriate vertices included in the animation mesh 400. Sequentially, the vertex positions may then be transferred from motion meshes 500 b, 500 c and 500 d (for times T=2, 3 and N) to animate the animation mesh 400.
As mentioned, by collecting images over one or multiple sessions, one or more motion models may be produced for reconstructing facial expressions captured in the images (and produce similar expressions). To produce a motion model (or models), content from a set of images collected during a session may be decomposed (e.g., into principal components). In some arrangements, content from each of the collected images is used to produce a model. However, for a large number of collected images, the corresponding amount of content may be vast and need a considerable amount of computing resources to be processed in a reasonable time period.
By selecting a representative subset of images from the collected images, a motion model may be efficiently produced in a reasonable time period. Once created, the model may be enhanced with additional content to progressively expand the range of expressions that may be reconstructed from the motion model. For example, content from one or more of the other images captured during the session may be decomposed and combined to enhance the motion model. Images captured during multiple sessions may also be used for model enhancement. For example, a motion model may be created from a subset of images captured during one session in which an actor performs (e.g., a movie, television, show, commercial, etc.). To enhance the motion model, additional images associated with another performance session of the actor (e.g., next day's shooting, shooting at a different location, etc.) may be used to refine the model for a wider range of facial expressions or other representations of the actor. As described below, one or more techniques and methodologies may be implemented for choosing the subset of images.
A created motion model may also be used to develop another motion model. For example, one model (produced from a selected subset of images of an actor's performance) may be used to as a basis for creating a model of another performance of the same actor, a motion model of a performance of another actor (e.g., an actress) or the like. Motion models may be created (from a previously created motion model) that represent performances associated with a particular project (e.g., movie, television show, commercial, etc.), or a particular role (e.g., a character) or other similar event.
Additional content for developing other motion models or enhancing a current motion model may be attained by estimating image content. For example, a motion model may be used to track surface features (e.g., natural features, artificial features, etc.) in a series of images (e.g., separate from the images used to create the motion model) representing an actor performing a number of facial expressions. The estimated image content may be used to develop another motion model or to enhance the motion model used to track the surface features.
By producing one or more motion models from a subset of selected images, computational resources needed to produce the models is reduced along with processing time. As the models are progressively refined and enhanced by estimating surface features, e.g. included in additional images, computational loading and resource needs may remain in a reasonable and sustainable range.
Referring to FIG. 11, production of a motion model from a set of selected images is illustrated. A series of images 1100 are captured with a motion capture system such as the system 100 shown in FIG. 1. The images 1100 are provided to a model initiator 1102 that may be executed on a single computer system (e.g., computer system 112) or distributed across multiple computer systems for execution. The model initiator selects a subset (e.g., two or more) of the images 1100 to produce a motion model 1104 that may be used to reconstruct a range of representations (e.g., an actor's facial expressions) captured within the selected images. The motion model 1104 may also serve as a basis for one or more other motion models (e.g., an enhanced version of the motion model 1104, a new motion model, etc.).
In this implementation the motion model 1104 is stored in a motion library 1106 and may be retrieved for use at a later time. For example the motion library 1106 may be similar to the motion library 124 and be stored in a storage device (e.g., a hard-drive, a CD-ROM, RAID drive, etc.) such as storage device 114.
Referring to FIG. 12, some operations of the model initiator 1102 are illustrated for producing the motion model 1104 by using a selected subset of images from the image series 1100. To select images from the series, an image selector 1200 is included in the model initiator 1102. One or more techniques and methodologies may be implemented by the image selector 1200 for selecting an image subset. For example, a predetermined criterion may be used to control image selection. In one implementation, the first and last images in the series 1100 may be selected. Along with first and last positions within the series 1100, other positions in the series may also be used for a selection criterion. Every fifth image of the image series 1100, for example, may be selected for inclusion in a subset of images for model production. Image selection may also be based upon content included in individual images.
For example, one or more objects represented in the images 1100 may be used as a selection criterion. Objects such as an actor's face or body are some examples of deformable objects that may be represented in the images. Surface features of an object may also be used for image selection. For example, natural surface features (e.g., eye or lip contours, skin texture, etc.) or artificial surface features (e.g., makeup patches, applied markers, etc.) may be used to identify images for selection. Surface features that appear in common among two or more images may serve as the basis for including images in the selected subset. For example, one image may include a lip contour in a representation of a smiling actor and another image may include a lip contour in a representation of a frowning actor. In another example, one image may include a marker applied to the nose of a screaming actor while another image (that includes the same marker) may be of the actor being quiet.
Image selection may also be based upon the relative position change of surface features among two or more images. For example, two images may be selected that each include surface features in nearly equivalent positions (e.g., an actor's facial expression are similar in each image). Producing a motion model from two or more images with such similar representations may provide a limited range of reconstructed representations. To provide a relatively wider range of representations, the image selector 1200 selects images that include surface features that represent a wide range of motion. For example, an image of a laughing actor may be selected along with an image (that includes at least one surface feature present in the first image) that presents the actor in an angered mood. Due to the surface feature range of motion within the two images, a motion model may be produced that can reconstruct representations of these emotional extremes along with other representations (e.g., the actor in an agitated mood) of similar emotions. A pairing of a neutral expression and a relatively extreme emotion may also be used as a selection criterion. For example, an image of actor presenting a calm demeanor may be selected with an image of the actor acting manic. The positions of the surface features in the calm demeanor image may be similar to feature positions for many frequently occurring facial expressions (e.g., mildly amused, somewhat sad, etc.) while feature positions in the manic image may occur with less frequently occurring expressions (e.g., terror, euphoria, etc.). A motion model produced from such a pair of images may provide a wide range of representations (e.g., facial expressions) that may be reconstructed. Other image pairs or combinations of images may also be used to produce a motion model.
Selection criteria may also be based upon user input. For example, a technician may view the images series 1100 and direct image selection by the image selector 1200. User input may also provide one or more rules for image selection. For example, a user may provide a particular count of surface features that needs to be present in an image to be selected. Similarly, a user may identify one or more particular surfaces (e.g., markers) that must be present in an image to be selected.
In this arrangement, the image selector 1200 selects two images 1202, 1204 from the series of images 1100. The selected images are provided to a decomposer 1206 that produces the motion model 1104 by decomposing the image content. For example, surface features represented in the images 1202, 1204 may be used to produce a motion mesh for each image to represent the location of the surface features. The motion meshes may be decomposed by the decomposer 1206 into principal components to produce the motion model 1104. Along with being stored for later retrieval and use, the motion model 1104 may be used to substantially reconstruct the representations (e.g., facial expressions) present in the images 1202, 1204 along with similar representations (e.g., similar facial expressions).
Referring to FIG. 13, once produced from a selected subset of captured images, content of other images may be used to refine a motion model such that the model provides an expanded range of producible representations (e.g., facial expressions). As illustrated, the motion model 1104 may be retrieved from the motion library 1106 and provided to a model updater 1300 that may be executed on a single computer system (e.g., computer system 112) or distributed across multiple computer systems for execution. One or more types of additional content may be used by the model updater 1300 to refine the motion model 1104. For example, content may be provided from additional images captured during the same session as the images 1202, 1204 used to produce the motion model 1104. Images captured during a different session (e.g., at a later time, at a different location, etc.) may also be used to enhance the motion model 1104. As illustrated, a series of images 1302 are provided to model updater 1300 that illustrate the same actor captured in the image series 1100 (from which the image subset was selected to produce the motion model 1104). While images of the same actor are used in this scenario, images of one or more other actors may be used by the model updater 1300. For example, another actor playing the same role (performed by the first actor in the images 1100) may be used by the model updater 1300 to refine the motion model 1104. Images may also be selected based upon processing performed with the motion model 1104. For example, a series of facial expressions may be tracked with the motion model 1104 for enhancing the motion model. In a similar manner, selected images may be used (e.g., tracked) with the motion model 1104 to produce another motion model. The motion model 1104 may also use one or more other methodologies and techniques to produce additional motion models.
The model updater 1300 uses the additional content provided by the image series 1302 along with the previously produced motion model 1104 to produce an updated motion model 1304. Similar to the motion model 1104, the updated motion model 1304 may be stored in a library such as the motion library 1106. The updated motion model 1304 may also be used to reconstruct the representations (e.g., the actor's facial expressions) of the images (e.g., images 1202 and 1204) used to produce the motion model 1104 and the images (e.g. image series 1302) used produce the updated motion model 1304. The updated motion model 1304 may also be used to produce representations not included in the images (e.g., images 1202 and 1204, the series of images 1302). For example, by applying one or more weighting factors to the updated motion model 1304, a facial expression may be produced that is similar (but not exactly matching) the facial expressions represented in the images 1202, 1204 and the series of images 1302. As such, the number of producible facial expressions may be expanded by incorporating additional content into the motion model 1104 to produce the updated motion model 1304.
Referring to FIG. 14, one implementation of the model updater 1300 includes a decomposer 1400 and a combiner 1402 for updating motion models to expand the range of reproducible representations. Similar to the decomposers discussed above, the decomposer 1400 is capable of decomposing the content of images (e.g., image series 1302) using one or more linear transformation techniques such as PCA. The decomposer 1400 may also be capable of selecting a subset of images (e.g., from the series of images 1302) for refining the motion model 1104. For example, an image may be selected if one or more surface features (e.g., markers) are identified in the image. Absent the appropriate surface features, an image may be not selected for inclusion in the subset. By using the image series 1302 (or selected images from the series), the motion model 1104 may estimate surface features present in the corresponding images to provide content to enhance the motion model. The content of the images may also be transformed prior to being decomposed by the decomposer 1400. For example, one or more motion meshes or other type of representation may be produced to represent surface features included in the images 1302.
Once decomposed (e.g., into principal components), the combiner 1402 combines the decomposed information and the motion model 1104. One or more techniques and methodologies may be implemented by the combiner 1402 to produce the updated motion model 1304. For example, principal components produced by the decomposer 1400 may be combined with principal components of the motion model 1104. The combiner 1402 may also apply weighting factors or scale factors to the principal components of the motion model 1104 and the components computed from the image series 1302 for adjustment. User input may also be accepted by the model updater 1300, for example, image selection, decomposition technique selection and the like may be based on user input. Both similar and dissimilar estimated image content may be used to update a motion model. As illustrated in the figure, images 1302 of the same actor may be used to update the motion model 1104. However, images of another actor (e.g., performing as the same character as the first actor) may be used to update the motion model 1104. Or, in another scenario, the same actor performing another role may be used to update the motion model 1104. In still another scenario, the motion model 1104 may be used as a basis for producing an entirely new motion model.
Referring to FIG. 15, a motion model 1500 is created for one actor based upon the previously created motion model 1104 associate with another actor. As illustrated in FIG. 12, a subset of selected images (e.g., two images) may be used to produce the motion model 1104 that is capable of providing a relatively limited range of facial expressions or other types of representations similar to the captured images. Surface feature movement (for facial expressions) tend to be somewhat correlated since the position of the surface features for one expression may be similar to the surface feature positions for another facial expression. As such, a motion model produced from a limited number of facial expressions may be used as a basis for a motion model of another actor's face.
In this example, the motion model 1104 produced from the selected images 1202, 1204 (of a male actor) is used as the basis for producing the motion model 1500 associated with captured images of an actress. Upon being retrieved from the motion library 1106, the motion model 1104 is provided to a model updater 1502 that operates in a similar manner to the model updater 1300 (shown in FIGS. 13 and 14).
Along with the motion model 1104, a series of images 1504 may be provided to the model updater 1502. The model updater 1502 may use the images in a manner similar to the model updater 1300 (shown in FIG. 13). For example, the motion model 1104 may be used to estimate surface features included one or more of the images. The estimated surface features may be decomposes, for example, into principal components. As mentioned above, a subset of images may be selected from the image series 1504 prior to decomposition. As such, images lacking appropriate definition (e.g., blurred) in the surface features may be removed prior to decomposition. The decomposed content may be combined with the motion model 1104 to produce the motion model 1500 that may be stored (e.g., in the motion library 1106) and used (e.g., a later time) to reconstruct the actress's facial expressions represented in the series of images 1504 along with other facial expressions within the range of the motion model.
As illustrated, one motion model (e.g., motion model 1500) is produced from a series of images and a previously created motion model (e.g., motion model 1104). In some arrangements, other types of information may be used to produce a motion model from a previously produced motion model. For example, two or more previously produced motion models may be combined to produce a new motion model. A portion of a previously produced motion model may be adjusted to produce a new motion model. For example, one or more weighting factors and mathematical functions may be applied to a motion model to produce a new motion model. A new motion model may also be produced by deleting a portion of a previously produced motion model. While deleting a portion of a motion model may reduce the range of representations (e.g., facial expressions) that may be reconstructed, the new motion model may be combined with content from a series of images or one or more other motion models to expand the representation range of the model.
Referring to FIG. 16, a flowchart 1600 represents some of the operations of the model initiator 1102. The operations may be executed by a single computer system (e.g., computer system 112) or multiple computing devices. Along with being executed at a single site (e.g., at one computer system), operation execution may be distributed among two or more sites.
Operations include receiving 1602 a set of captured images. For example, a set of images of an actor performing a particular character role may be received. Operations also include selecting 1604 a subset of images from the set of received images. For example, a subset of two images may be selected from the received images. As mentioned above, various criteria may be implemented for subset selection. For example, two images of the actor may be selected that each include a common surface feature. Operations also include decomposing 1606 content of the image subset. For example, representations of the surface features may be decomposed into principal components to produce a motion model. Upon decomposing the content to produce the motion model, operations also include storing 1608 the motion model in a motion library.
Referring to FIG. 17, a flowchart 1700 represents some of the operations of the model updater 1300. The operations may be executed by a single computer system (e.g., computer system 112) or multiple computing devices. Along with being executed at a single site (e.g., at one computer system), operation execution may be distributed among two or more sites.
Operations include receiving 1702 a motion model. In some arrangements the received motion model is produced from a subset of images by a model initiator such as the model initiator 1102. Operations also include receiving 1704 additional content for enhancing the received motion model, producing another motion model, etc. For example, a set of captured images may provide additional content for expanding the expressions that may be produced by the received motion model. In some arrangements, the additional content may include estimated surface features provided by the motion model. Another motion model may also be received for enhancing the first received motion model. Operations also include decomposing 1706 the additional content. For example, content that represents surface features in a set of images may be decomposed into principal components. Operations also include combining 1708 the decomposed content with the received motion model. By combing the decomposed content, representations that may be produced from the enhanced motion model may be expanded. Combining the decomposed content may also produce a new motion model. For example, a received motion model of an actor may be combined with decomposed image content that represents a performance of an actress to create a motion model of the actress's performance. Upon combining the decomposed content with the received motion model, operations include storing 1710 the updated or new motion model.
To perform the operations described in flow chart 1600 and 1700, model initiator 1102 and model updater 1300 may respectively perform any of the computer-implement methods described previously, according to one implementation. For example, a computer system such as computer system 112 (shown in FIG. 1) may execute the model initiator 1102 and the model updater 1300. The computer system may include a processor (not shown), a memory (not shown), a storage device (e.g., storage device 114), and an input/output device (not shown). Each of the components may be interconnected using a system bus or other similar structure. The processor is capable of processing instructions for execution within the computer system. In one implementation, the processor is a single-threaded processor. In another implementation, the processor is a multi-threaded processor. The processor is capable of processing instructions stored in the memory or on the storage device to display graphical information for a user interface on the input/output device.
Citations de brevets Brevet cit� Date de d�p�t Date de publication D�posant TitreUS579012420 nov. 19954 ao�t 1998Silicon Graphics, Inc.System and method for allowing a performer to control and interact with an on-stage display deviceUS583126010 sept. 19963 nov. 1998Ascension Technology CorporationHybrid motion trackerUS593241715 oct. 19963 ao�t 1999The Regents Of The University Of CaliforniaMethod of screening compounds for controlling capacitative calcium ion entry into mammalian cellsUS60724968 juin 19986 juin 2000Microsoft CorporationMethod and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objectsUS611505212 f�vr. 19985 sept. 2000Mitsubishi Electric Information Technology Center America, Inc. (Ita)System for reconstructing the 3-dimensional motions of a human figure from a monocularly-viewed image sequenceUS616681112 ao�t 199926 d�c. 2000Perceptron, Inc.Robot-based gauging system for determining three-dimensional measurement dataUS620834827 mai 199827 mars 2001In-Three, Inc.System and method for dimensionalization processing of images in consideration of a pedetermined image projection formatUS63242964 d�c. 199727 nov. 2001Phasespace, Inc.Distributed-processing motion tracking system for tracking individually modulated light pointsUS635342231 mars 20005 mars 2002Stephen G. PerlmanVirtual display system and methodUS643825514 avr. 200020 ao�t 2002Stress Photonics, Inc.Transient thermal marking of objectsUS651565919 d�c. 20014 f�vr. 2003In-Three, Inc.Method and system for creating realistic smooth three-dimensional depth contours from two-dimensional imagesUS652233226 juil. 200018 f�vr. 2003Kaydara, Inc.Generating action data for the animation of charactersUS66060957 juin 199912 ao�t 2003Microsoft CorporationCompression of animated geometry using basis decompositionUS66144074 janv. 20022 sept. 2003Stephen G. PerlmanVirtual display system and methodUS66144287 juin 19992 sept. 2003Microsoft CorporationCompression of animated geometry using a hierarchical level of detail coderUS66332949 mars 200114 oct. 2003Lucasfilm Entertainment Company LtdMethod and apparatus for using captured high density motion for animationUS668692626 mars 20013 f�vr. 2004In-Three, Inc.Image processing system and method for converting two-dimensional images into three-dimensional imagesUS691989214 ao�t 200219 juil. 2005Avaworks, IncorporatedPhoto realistic talking head creation system and methodUS697763018 juil. 200020 d�c. 2005University Of MinnesotaMobility assist deviceUS702705414 ao�t 200211 avr. 2006Avaworks, IncorporatedDo-it-yourself photo realistic talking head creation system and methodUS70354366 ao�t 200225 avr. 2006The University Of TokyoMethod of generating poses and motions of a tree structure link systemUS709892027 juil. 200429 ao�t 2006Microsoft CorporationMethods and systems for animating facial features and methods and systems for expression transformationUS710263315 mai 20025 sept. 2006In-Three, Inc.Method for conforming objects to a common depth perspective for converting two-dimensional images into three-dimensional imagesUS711632310 d�c. 20023 oct. 2006In-Three, Inc.Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional imagesUS711632430 sept. 20033 oct. 2006In-Three, Inc.Method for minimizing visual artifacts converting two-dimensional motion pictures into three-dimensional motion picturesUS71299492 nov. 200431 oct. 2006Microsoft CorporationMethods and systems for animating facial features, and methods and systems for expression transformationUS71647186 sept. 200116 janv. 2007France TelecomMethod for segmenting a video image into elementary objectsUS718404727 juin 200027 f�vr. 2007Crampton Stephen JamesMethod and apparatus for the generation of computer graphic representations of individualsUS721265626 janv. 20061 mai 2007Microsoft CorporationRapid computer modeling of faces for animationUS729226120 ao�t 19996 nov. 2007Intellectual Ventures I LlcVirtual reality cameraUS74338071 sept. 20057 oct. 2008Microsoft CorporationFacial image processingUS745012624 avr. 200611 nov. 2008Microsoft CorporationMethods and systems for animating facial features, and methods and systems for expression transformationUS75545498 nov. 200430 juin 2009Sony CorporationSystem and method for tracking facial muscle and eye motion for computer graphics animationUS760586110 mars 200520 oct. 2009Onlive, Inc.Apparatus and method for performing motion capture using shutter synchronizationUS801913714 sept. 200913 sept. 2011Lucasfilm Entertainment Company Ltd.Three-dimensional motion captureUS2001002451222 f�vr. 200127 sept. 2001Cybernet Systems CorporationOptical body trackerUS2001003367531 mai 200125 oct. 2001Google Inc.Wavelet-based facial motion capture for avatar animationUS2002004128522 juin 200111 avr. 2002Lifef/X Technologies, Inc.Non-linear morphing of faces and their dynamicsUS200200606494 janv. 200223 mai 2002Rearden, LlcVirtual display system and methodUS2004006348110 juin 20031 avr. 2004Wang XiaolingApparatus and a method for more realistic interactive video games on computers or similar devices using visible or invisible light and an input computing deviceUS2004011971628 mai 200324 juin 2004Electronics And Telecommunications Research InstituteApparatus and method for high-speed marker-free motion captureUS2004015596211 f�vr. 200312 ao�t 2004Marks Richard L.Method and apparatus for real time motion captureUS2004016113216 janv. 200419 ao�t 2004Cohen Charles J.Gesture-controlled interfaces for self-service machines and other applicationsUS200401790081 mai 200316 sept. 2004Sony CorporationSystem and method for capturing facial and body motionUS2005007812414 oct. 200314 avr. 2005Microsoft CorporationGeometry-driven image synthesis renderingUS2005009941415 mai 200212 mai 2005Best Charles J.L.Method for conforming objects to a common depth perspective for converting two-dimensional images into three-dimensional imagesUS2005010487810 d�c. 200219 mai 2005Kaye Michael C.Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional imagesUS2005010487930 sept. 200319 mai 2005Kaye Michael C.Method for minimizing visual artifacts converting two-dimensional motion pictures into three-dimensional motion picturesUS200501465212 mars 20047 juil. 2005Kaye Michael C.Method for creating and presenting an accurate reproduction of three-dimensional images converted from two-dimensional imagesUS2005023150530 juin 200420 oct. 2005Best Charles J LMethod for creating artifact free three-dimensional images converted from two-dimensional imagesUS2006005569915 sept. 200416 mars 2006Ol2, Inc.Apparatus and method for capturing the expression of a performerUS2006005570615 sept. 200416 mars 2006Ol2, Inc.Apparatus and method for capturing the motion of a performerUS2006006757319 sept. 200530 mars 2006Cyberextruder.Com, Inc.System, method, and apparatus for generating a three-dimensional representation from one or more two-dimensional imagesUS200601269282 d�c. 200515 juin 2006Image Metrics LimitedMethod and system for cleaning motion capture dataUS2006015764018 janv. 200520 juil. 2006Rearden, LlcApparatus and method for capturing still images and video using coded aperture techniquesUS2006019278524 avr. 200631 ao�t 2006Microsoft CorporationMethods and systems for animating facial features, and methods and systems for expression transformationUS2006019285425 f�vr. 200531 ao�t 2006Ol2, Inc.Apparatus and method improving marker identification within a motion capture systemUS2006022810116 mars 200612 oct. 2006Lucasfilm Entertainment Company Ltd.Three-dimensional motion captureUS2007005271125 ao�t 20068 mars 2007Gordon DemianReconstruction render farm used in motion captureUS2007009117820 oct. 200526 avr. 2007Cotter Tim SApparatus and method for performing motion capture using a random pattern on capture surfacesUS200701338418 f�vr. 200714 juin 2007Siemens Corporate Research, Inc.Coded visual markers for tracking and camera calibration in mobile computing systemsUS2008017007716 janv. 200717 juil. 2008Lucasfilm Entertainment Company Ltd.Generating Animation LibrariesUS2008017007813 avr. 200717 juil. 2008Lucasfilm Entertainment Company Ltd.Using animation libraries for object identificationUS2008017077713 avr. 200717 juil. 2008Lucasfilm Entertainment Company Ltd.Combining multiple session content for animation librariesUS2008018044825 juil. 200731 juil. 2008Navy, Secretary Of The United States Of AmericaShape completion, animation and marker-less motion capture of people, animals or charactersUS2009020934315 f�vr. 200820 ao�t 2009Indigo Technologies, LlcMotion-tracking game controllerUS2010000293414 sept. 20097 janv. 2010Davidson ColinThree-Dimensional Motion CaptureUS2010016486221 juil. 20091 juil. 2010Lucasfilm Entertainment Company Ltd.Visual and Physical Motion Sensing for Three-Dimensional Motion CaptureEP1946243A1 Titre non disponibleJP10084691A Titre non disponibleWO2004041379A230 oct. 200321 mai 2004Nike Inc.Clothes with tracking marks for computer gamesCitations hors brevetsR�f�rence1Agarwala, Aseem, "Keyframe-Based Tracking for Rotoscoping and Animation," ACM SIGGRAPH '04 conference proceedings, 2004, pp. 1-8.2Bascle, B., Blake, A., Separability of pose and expression in facial tracking and animation, Jan. 1998, IEEE Computer Society, Proceedings of the Sixth International Conference on Computer Vision, pp. 323-328.3Buenaposada, Jose M., "Performance driven facial animation using illumination independent appearance-based tracking," In Proceedings of ICPR, Hong Kong, Aug. 2006, 4 pages.4Chang et al., "Automatic 3D Facial Expression in Analysis in Videos," Proc. IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures (AMFG '05), vol. 3723, pp. 293-307, 2005.5Curio, Crist�bal et al., "Semantic 3D Motion Retargeting for Facial Animation." ACM Jul. 28-29, 2006, pp. 77-84, 176.6Debevec, et al. "Acquiring the Reflectance Field of a Human Face", SIGGRAPH 2000 Conference Proceedings, pp. 1-12, 2000.7Deng et al., "Animating blendshape faces by cross-mapping motion capture data," In Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games (Redwood City, California, Mar. 14-17, 2006). I3D '06. ACM, New York, NY, 43-48.8Examination Report from European Patent Office for Application No. GB0718003.7, dated Aug. 3, 2009, 2 pages.9Examination Report from New Zealand Patent Office for Application No. 561570, dated Dec. 18, 2009, 1 page.10Examination Report from New Zealand Patent Office for Application No. 561570, dated Jul. 7, 2009, 2 pages.11Examination Report from New Zealand Patent Office for Application No. 581496,dated Dec. 2, 2009, 2 pages.12Examination Report from New Zealand Patent Office for Application No. 582356, dated Jan. 7, 2010, 2 pages.13Examination Report from United Kingdom Office for Application No. PCT/US2006/009787, dated Aug. 3, 2009, 2 pages.14Furukawa et al., "Dense 3D Motion Capture from Synchronized Video Streams," [online], [retrieved on Aug. 26, 2009]. Retrieved from the Internet , 8 pages.15Furukawa et al., "Dense 3D Motion Capture from Synchronized Video Streams," [online], [retrieved on Aug. 26, 2009]. Retrieved from the Internet <URL: http://www.cs.washington.edu/homes/furukawa/papers/cvpr08b.pdf>, 8 pages.16Gleicher, Michael, "Animation From Observation: Motion Capture and Motion Editing," Computer Graphics 33(4), 1999, pp. 1-5.17Grung et al. "Missing values in principal component analysis", Chemometrics and Intelligent Laboratory Systems, vol. 42, No. 1-2, pp. 125-139, Aug. 1998.18Heap et al. "Towards 3D hand tracking using a deformable model", IEEE Computer Society, pp. 140-145, Oct. 1996.19Herda et al. "Skeleton-based motion capture for robust reconstruction of human motion", Computer Animation 2000, Proceedings, pp. 77-83, May 2000.20Horprasert et al., "Real-time 3D Motion Capture," Second workshop on Perceptual Interfaces, San Francisco. Nov. 1998, 4 pages.21International Preliminary Report on Patentability for PCT Application No. PCT/US2006/009787, dated Nov. 13, 2007, 9 pages.22International Search Report and Written Opinion for PCT Application No. PCT/US2006/009787, dated Oct. 29, 2007, 12 pages.23Lin et al. "Extracting 3D facial animation parameters from multiview video clips", IEEE Computer Graphics and Applications, vol. 22, No. 6, pp. 72-80, Nov. 2002.24Majkowska, A. et al., "Automation splicing for hand and body animations." 2006. In Proceedings if the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation (SCA '06). Eurographics Associate, Aire-la-Ville, Switzerland, Switzerland, pp. 309-316 and 369.25Markoff, "Camera System Creates Sophisticated 3-D Effects" [online], New York Times, Jul. 31, 2006, [retrieved on Jul. 31, 2006]. Retrieved from the Internet: , 4 pages.26Markoff, "Camera System Creates Sophisticated 3-D Effects" [online], New York Times, Jul. 31, 2006, [retrieved on Jul. 31, 2006]. Retrieved from the Internet: <URL: www.nytimes.com/2006/0731/technology/31motion.html >, 4 pages.27Park, S. et al., "Capturing and Animating Skin Deformation in Human Motion." ACM 2006, pp. 881-889.28Parke, F., Computer Generated Animation of Faces, Aug. 1972, SIGGRAPH 1972, pp. 451-457.29Search Report from United Kingdom Intellectual Property Office for application No. GB0922650.7, dated Apr. 20, 2010, 2 pages.30Stopford, Office Action issued in AU application No. 2009240847 mailed Nov. 23, 2011, 4 pages.31Wren et al, "Dynamic Models of Human Motion," [online], 1998, [retrieved on Aug. 26, 2009]. Retrieved from the Intern et , 6 pages.32Wren et al, "Dynamic Models of Human Motion," [online], 1998, [retrieved on Aug. 26, 2009]. Retrieved from the Intern et <URL: http://www.drwren.com/chris/dyna/TR-415.pdf>, 6 pages. R�f�renc� par Brevet citant Date de d�p�t Date de publication D�posant TitreUS201002316925 juin 200916 sept. 2010Onlive, Inc.System and method for performing motion capture and image reconstruction with transparent makeupUS2011017592230 mars 201121 juil. 2011At&T Corp.Method for defining animation parameters for an animation definition interfaceFaire pivoterImage d'origineAccueil Google - Plan du site - T�l�chargements par lot sur l'USPTO - R�gles de confidentialit� - Conditions d'utilisation - � propos de Google�Brevets - Envoyer des commentairesDonn�es fournies par IFI CLAIMS Patent Services©2012 Google