Source: http://www.google.com/patents/US8130225?dq=6175559
Timestamp: 2016-06-26 01:37:37
Document Index: 375575500

Matched Legal Cases: ['art 900', 'art 900', 'art 900', 'Application No. 561570', 'Application No. 561570', 'Application No. 581496', 'Application No. 582356', 'application No. 2009240847']

Patent US8130225 - Using animation libraries for object identification - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inPatentsA computer-implemented method includes comparing one or more surface features to a motion model. The surface feature or surface features represent a portion of an object in an image. The method also includes identifying a representation of the object from the motion model, based upon the comparison....http://www.google.com/patents/US8130225?utm_source=gb-gplus-sharePatent US8130225 - Using animation libraries for object identificationAdvanced Patent SearchPublication numberUS8130225 B2Publication typeGrantApplication numberUS 11/735,283Publication dateMar 6, 2012Filing dateApr 13, 2007Priority dateJan 16, 2007Fee statusPaidAlso published asUS8681158, US20080170078Publication number11735283, 735283, US 8130225 B2, US 8130225B2, US-B2-8130225, US8130225 B2, US8130225B2InventorsSteve Sullivan, Francesco G. CallariOriginal AssigneeLucasfilm Entertainment Company Ltd.Export CitationBiBTeX, EndNote, RefManPatent Citations (70), Non-Patent Citations (33), Referenced by (22), Classifications (11), Legal Events (3) External Links: USPTO, USPTO Assignment, EspacenetUsing animation libraries for object identification
US 8130225 B2Abstract
A computer-implemented method includes comparing one or more surface features to a motion model. The surface feature or surface features represent a portion of an object in an image. The method also includes identifying a representation of the object from the motion model, based upon the comparison.
comparing in a processor at least one surface feature to representations produced by a motion model that is produced from a plurality of captured images that each include at least one surface feature, wherein the surface feature being compared represents a portion of an object in an image, wherein the image is different from the plurality of captured images that produced the motion model;
identifying a representation of the object from the representations produced by the motion model based upon the comparison;
adjusting an animation mesh, different from the motion model, by using the motion model such that the animation mesh represents the portion of the object in the image, the image being different from the plurality of captured images that produced the motion model, wherein the resolution of the animation mesh is greater than the resolution of the motion model; and producing an image from the adjusted animation mesh.
2. The computer-implemented method of claim 1, wherein comparing includes processing decomposed data of the motion model to substantially match the at least surface feature.
3. The computer-implemented method of claim 2, wherein the processing includes applying at least one weight to the decomposed data.
4. The computer-implemented method of claim 2, wherein the decomposed data includes principal components.
5. The computer-implemented method of claim 1, wherein comparing includes processing the surface feature.
6. The computer-implemented method of claim 5, wherein the processing includes decomposing least surface feature.
7. The computer-implemented method of claim 1, wherein the surface feature represents an artificial feature on a surface of the object.
8. The computer-implemented method of claim 1, wherein the surface feature represents a natural feature on a surface of the object.
9. The computer-implemented method of claim 1, wherein the surface feature represents a contour on a surface of the object.
10. The computer-implemented method of claim 1, wherein the surface feature represents a marker on a surface of the object.
11. The computer-implemented method of claim 1, wherein the image is captured by a single device.
12. The computer-implemented method of claim 1, wherein the image is captured by a first number of devices and the motion model is produced from a plurality of images captured by a second number of devices, wherein the first number of devices is less than the second number of devices.
13. The computer-implemented method of claim 1, wherein the object is a deformable object.
14. The computer-implemented method of claim 1, wherein the object is an actor's face.
15. The computer-implemented method of claim 1, wherein the representation is an actor's facial expression.
a data comparer to compare at least one surface feature to representations produced by a motion model that is produced from a plurality of captured images that each include at least one surface feature, wherein the surface feature being compared represents a portion of an object in an image, wherein the image is different from the plurality of captured images that produced the motion model, the data comparer is also configured to identify a representation of the object from the representations produced by the motion model based upon the comparison;
a data transferor to adjust an animation mesh, different from the motion model, by using the motion model such that the animation mesh represents the portion of the object in the image, the image being different from the plurality of captured images that produced the motion model, wherein the resolution of the animation mesh is greater than the resolution of the motion model; and
a renderer to produce an image from the adjusted animation mesh.
17. The system of claim 16, wherein the comparison includes the data comparer processing decomposed data of the motion model to substantially match the at least one surface feature.
18. The system of claim 17, wherein the comparison includes the data comparer applying at least one weight to the decomposed data.
19. The system of claim 17, wherein the decomposed data includes principal components.
20. The system of claim 16, wherein the surface feature represents an artificial feature on a surface of the object.
21. The system of claim 16, wherein the surface feature represents a natural feature on a surface of the object.
22. The system of claim 16, wherein the surface feature represents a contour on a surface of the object.
23. The system of claim 16, wherein the surface feature represents a marker on a surface of the object.
24. The system of claim 16, further comprising a single device to capture the image.
25. The system of claim 16, wherein the image is captured by a first number of devices and the motion model is produced from a plurality of images captured by a second number of devices, wherein the first number of devices is less than the second number of devices.
26. The system of claim 16, wherein the object is an actor's face and the representation is a facial expression.
27. A computer program product tangibly embodied in a storage device and comprising instructions that when executed by a processor perform a method comprising:
comparing at least one surface feature to representations produced by a motion model that is produced from a plurality of captured images that each include at least one surface feature, wherein the surface feature being compared represents a portion of an object in an image, wherein the image is different from the plurality of captured images that produced the motion model;
adjusting an animation mesh, different from the motion model, by using the motion model such that the animation mesh represents the portion of the object in the image, the image being different from the plurality of captured images that produced the motion model, wherein the resolution of the animation mesh is greater than the resolution of the motion model; and
producing an image from the adjusted animation mesh.
28. The computer program product of claim 27, wherein comparing includes processing decomposed data of the motion model to substantially match the at least one surface feature.
29. The computer program product of claim 28, wherein the processing includes applying at least one weight to the decomposed data.
30. The computer program product of claim 28, wherein the decomposed data includes principal components.
31. The computer program product of claim 27, wherein the surface feature represents an artificial feature on a surface of the object.
32. The computer program product of claim 27, wherein the surface feature represents a natural feature on a surface of the object.
33. The computer program product of claim 27, wherein the surface feature represents a contour on a surface of the object.
34. The computer program product of claim 27, wherein the image is captured by a single device.
35. The computer program product of claim 27, wherein the image is captured by a first number of devices and the motion model is produced from a plurality of images captured by a second number of devices, wherein the first number of devices is less than the second number of devices.
36. The computer program product of claim 27, wherein the object is an actor's face and the representation is a facial expression.
37. An expression identification system comprising:
compare at least one surface feature to representations produced by a motion model that is produced from a plurality of captured images that each include at least one surface feature, wherein the surface feature being compared represents a portion of the object in the image, wherein the image is different from the plurality of captured images that produced the motion model,
identify a representation of the object from the representations produced by the motion model based upon the comparison,
adjust an animation mesh, different from the motion model, by using the motion model such that the animation mesh represents the portion of the object in the image, the image being different from the plurality of captured images that produced the motion model, wherein the resolution of the animation mesh is greater than the resolution of the motion model; and
produce an image from the adjusted animation mesh.
38. The expression identification system of claim 37, wherein comparing includes processing decomposed data of the motion model to substantially match the at least one surface feature.
39. The expression identification system of claim 38, wherein the processing includes applying at least one weight to the decomposed data.
40. The expression identification system of claim 38, wherein the decomposed data includes principal components.
41. The expression identification system of claim 37, wherein the surface feature represents an artificial feature on a surface of the object.
42. The expression identification system of claim 37, wherein the surface feature represents a natural feature on a surface of the object.
43. The expression identification system of claim 37, wherein the surface feature represents a contour on a surface of the object.
44. The expression identification system of claim 37, wherein the image is captured by a single device.
45. The expression identification system of claim 37, wherein the image is captured by a first number of devices and the motion model is produced from a plurality of images captured by a second number of devices, wherein the first number of devices is less than the second number of devices. Description
This application is a continuation-in-part and claims the benefit of priority under U.S. application Ser. No. 11/623,707, filed Jan. 16, 2007. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. This application is related to U.S. application Ser. No. 11/735,291, filed Apr. 13, 2007, which is incorporated herein by reference.
This document relates to using libraries of animation information.
Computer-based animation techniques often involve capturing a series of images of an actor (or other object) with multiple cameras each of which has a different viewing perspective. The cameras are synchronized such that for one instant in time, each camera captures an image. These images are then combined to generate a three-dimensional (3D) graphical representation of the actor. By repetitively capturing images over a period of time, a series of 3D representations may be produced that illustrate the actor's motion (e.g., body movements, speech, etc.).
To produce an animation that tracks the actor's motion, a digital mesh may be generated from each 3D representation such that each mesh represents the position of the actor at the time of image capture. Together, the digital meshes represent the movement of the actor over the image capture time period. For example, the actor's face may be represented in a series of digital meshes that track facial expressions of the actor. Markers (e.g., make-up dots) that contrast with the actor's skin tone may be applied to the actor's face to highlight facial features and provide points to align vertices of the meshes.
Once generated, the digital meshes may be rendered as a computer-generated object (e.g., a character's body) to produce an animated character that includes, for example, the facial expressions of the actor. However, to provide sufficient detail such that the actor's face is recognizable, each mesh includes a significant number of vertices that correspond to significant number of applied markers that need to be captured under optimum lighting conditions. Furthermore, for each image capture session, the actor must endure the application of these many facial markers.
In one implementation, a library of previously captured animation information (together with data captured during a current session) can be used to identify, for example, an actor's facial expression. Once identified, a model stored in the library may be used to transfer a representation of the facial expression to a relatively high resolution mesh for animation. By using libraries of previously computed models, an actor's facial expression or other type of performance mannerism, characteristic or motion may be recognized from a reduced set of identifying surface features.
In one aspect, a computer-implemented method includes comparing at one or more surface features to a motion model. The surface feature or surface features represent a portion of an object in an image. The method also includes identifying a representation of the object from the motion model based upon the comparison.
Implementations may include any or all of the following features. The method may also include adjusting an animation mesh to incorporate the representation of the object. The motion model may be produced from captured images that each include one or more surface features. The comparison may include processing decomposed data of the motion model to substantially match the one or more surface features. For example, processing may include applying one or more weights to the decomposed data (e.g., principal components). The one or more surface features may also be processed, for example, the surface features may be decomposed into principal components.
The one or more surface features associated with the object may represent an artificial feature (e.g., a marker), a natural feature, a contour, other type of feature on a surface of the object. The image may be captured by a single device such as a camera. Furthermore, more devices (e.g., cameras) may be needed to capture images to produce the motion model than the number of devices needed to capture the image that includes the one or more surface features. The object may be a deformable object such as an actor's face and the representation may be a facial expression of the actor.
In another aspect, a system includes a data comparer to compare one or more surface features to a motion model. The surface features represent a portion of an object in an image. The data comparer also identifies a representation of the object from the motion model based upon the comparison.
In still another aspect, a computer program product tangibly embodied in an information carrier and comprises instructions that when executed by a processor perform a method that includes comparing one or more surface features to a motion model. The surface features represent a portion of an object in an image. The method also includes identifying a representation of the object from the motion model based upon the comparison.
In still another aspect, an expression identification system includes one or more devices to capture at least one image of an object. The system also includes a computer system to execute one or more processes to compare one or more surface features to a motion model. The surface feature or surface features represent a portion of the object in the image. The executed processes also identify a representation of the object from the motion model based upon the comparison.
FIG. 6A-C include a portion of the motion mesh, a portion of an animation mesh and a portion of a shape mesh.
FIG. 8 is a diagram of an exemplary expression identification system.
FIG. 9 is a flow chart of operations of one or more processes executed by the expression identification system.
To process the received camera images 110 a-e (along with exchanging associated commands and data), an shape mesh generator 116 is executed by the computer system 112. The shape mesh generator 116 combines the cameras images 110 a-e into a three-dimensional (3D) shape mesh (for that capture time instance) by using stereo reconstruction or other similar methodology. The shape mesh has a relatively high resolution and provides the 3D shape of the captured object (e.g., actor's face 104). For a series of time instances, the shape mesh generator 116 can produce corresponding shape meshes that match the movement of the actor's face 104.
The motion transferor 122 may also be capable of processing the animation meshes and motion information for efficient storage and reuse. For example, as described below, the motion transferor 122 may decompose the motion information. Decomposition techniques such as Principle Component Analysis (PCA) may be implemented. Generally, PCA is an analysis methodology that identifies patterns in data and produces principle components that highlight data similarities and differences. By identifying the patterns, data may be compressed (e.g., dimensionality reduced) without much information loss. Along with conserving storage space, the principle components may be retrieved to animate one or more animation meshes. For example, by combining principle components and/or applying weighting factors, the stored principle components may be used to generate motion information that represent other facial expressions. Thus, a series of actor facial expressions may be captured by the cameras 102 a-e to form a motion library 124 that is stored in the storage device 114. The motion library 124 may use one or more types of data storage methodologies and structures to provide a storage system that conserves capacity while providing reliable accessibility.
Referring to FIG. 2, a series of images 200 a-e respectively captured by cameras 102 a-e are illustrated. By temporally synchronizing the cameras 102 a-e, corresponding images may be captured during the same time instance. For example, each of the images labeled “T=1” may have been captured at the same time. The images 200 a-e are provided to the computer system 112 for processing by the shape mesh generator 116, the motion mesh generator 118, the animation mesh generator 120 and the motion transferor 122. For processing, the content of the images captured at the same time instance may be combined. For example, the shape mesh generator 116 may use stereo reconstruction (or other similar methodology) to construct a 3D shape mesh 202 for each time instance from the corresponding captured images. Generally each shape mesh has a relatively high resolution and provides a detailed representation of the shape of the captured object (e.g., the actor's face 104). As shown in FIG. 3A, an exemplary shape mesh 300 illustrates the shape (e.g., an actor's facial expression) produced from images (e.g., images 110 a-e) capture at the same time instant. While large in number, the vertices of the shape mesh 300 may not be distinguishable (compared to the markers) and may not be quantified by a coordinate system. As such, the motion of individual vertices may not be tracked from one shape mesh (e.g., T=1 shape mesh) to the next sequential shape meshes (e.g., T=2 shape mesh, . . . , T=n shape mesh).
In this implementation, to generate a motion mesh from the images 200 a-e, the motion mesh generator 118 determines the position of each marker in three dimensional space and the positions of the cameras 102 a-e. Each marker position is assigned to a vertex, which in combination form facets of a motion mesh. In some arrangements, the position determination is provided as described in U.S. patent application Ser. No. 11/384,211 (published as United States Patent Application Publication 2006/0228101), herein incorporated by reference. Referring to FIG. 3C, a motion mesh 304 is presented overlaying the image 302 (shown in FIG. 3B). The vertices of the motion mesh 304 are assigned the respective positions of the markers applied to the actor's face and interconnect to adjacent vertices to form triangular facets. Referring to FIG. 3D, the motion mesh 304 is shown absent the high resolution image 302. As is apparent, the resolution of the motion mesh 304 is relatively low and the actor's face is not generally recognizable. However, by correlating the vertices with the positions of the markers in subsequent high resolution images, the position of the vertices may be tracked over time and thereby allow motion tracking of the actor's facial expressions. Furthermore, by quantifying the positions of the vertices over time, the associated motion information may be used to transfer the actor's facial expressions to an animated character or other type of computer-generated object.
As mentioned, while the vertices of the motion mesh 304 allow tracking of the motion of the actor's face, the relatively low resolution of the motion mesh does not provide a recognizable face. To improve resolution, some conventional methodologies increase the number of markers applied to the actor's face, thereby increasing the number of motion mesh vertices and mesh resolution. However, additional markers require more of the actor's time for application along with additional processing and storage space to generate and store the motion mesh. Furthermore, optimal lighting conditions may be needed to resolve the closely position markers. Thus, image capture may be always need to be confined to a controlled lighting environment such as a studio and not be applicable in low light environments or naturally lit environments (e.g., outside).
Referring to FIG. 6C, a portion of the shape mesh 502 a that corresponds to the actor's mouth is illustrated. The equivalent location of the vertex (highlighted by the ring 600 in FIG. 6A) and the vertices (highlighted by the ring 602 in FIG. 6B) is highlighted by a ring 604 in the shape mesh portion. As the figure illustrates, one or more shapes are included within the ring 604. As mentioned above, the shape(s) may be used to influence the motion transfer. For example, the shapes included in the ring 604 may define a range that limits the motion transferred to the vertices in the ring 602. As such, motion transferred to the vertices within the ring 602 may be constrained from significantly deviating from shapes in the ring 604. Similar shapes in the sequence of shape meshes 502 b-d may correspondingly constrain the transfer of motion information from respective motion meshes 500 b-d. In some situations, a shape mesh may include gaps that represent an absence of shape information. As such, the shape mesh may only be used to transfer motion information corresponding to locations in which shape information is present. For the locations absent shape information, motion information from one or more motion meshes may be transferred using shape information from the animation mesh. For example, the current shape or a previous shape of the animation mesh (for one or more locations of interest) may be used to provide shape information.
By collecting images of facial expressions and decomposing motion information associated with the expressions, a numerical model may be produced that allows each expression (or similar expressions) to be reconstructed. For example, principal components (produced from decomposed motion information) may be retrieved and applied with weights (e.g., numerical factors) to reconstruct one or more facial expressions (e.g., facial expressions used in the decomposition, new facial expressions, etc.). These numerical models, referred to as motion models, may be produced for one or more applications. For example, one motion model may be produced for reconstructing an actor's facial expressions for a particular performance. Other motion models may represent other performances of the actor or other actors. Performances may include the actor's participation in a project (e.g., movie, television show, commercial, etc.), or playing a particular role (e.g., a character) or other similar event.
Referring to FIG. 8, an expression identification system 800 captures images with fewer devices (compared to the motion capture system 100 shown in FIG. 1) and identifies an actor's facial expressions from the captured images by using one or more motion models. In some arrangements, the facial expression may be identified from only one captured image (and one motion model). Once identified, the expression may be reconstructed (for a previously known expression) or constructed (for a new facial expression) and used to animate a character (or other type of animation object).
For illustration, four motion models 802, 804, 806, 808 are shown as being included in a motion library 810 and stored in a storage device 812 (e.g., hard drive, CD-ROM, etc.) for retrieval by a computer system 814 (or other type of computing device). Each individual motion model may correspond to a particular actor, performance, character, etc. and include decomposed data (e.g., principal components) produced (e.g., by motion capture system 100) from captured images.
By using one or more of the motion models 802-808, a small amount of captured image data is needed for recognizing the actor's facial expression. For example, this implementation uses a single camera 816 to capture individual images (e.g., an image 818) of the actor's performance. Contents of the captured image 818 may be used to identify the actor's facial expression, for example, surface features of the actor's face may be used for comparing with expressions that may be represented by a motion model (e.g., motion model 802). Relatively few features may be needed to match the captured facial expression with an expression represented by the motion model 802. For example, only a small number (e.g., one or more) of artificial points (e.g., markers 820) may be needed for expression identification. A small number (e.g., one or more) of natural points provided by, for example, facial texture 822 (e.g., blemishes) or naturally occurring facial contours 824 (e.g., wrinkles near the corner of an eye or the corner of the mouth, etc.) may also be used individually or in combination (e.g., with other natural points, artificial points, etc.) as surface features for expression identification.
To process the image 818 and the associated motion model 802, the computer system 814 executes processes (e.g., applications, routines, etc.) associated with receiving and comparing the respective content. Processes are also executed by the computer system 814 to transfer the identified facial expression to an object (e.g., a computer generated character) for animation. In this implementation, a data receiver 826 is executed by the computer system 814 to receive content of the captured image 818 and the motion model 802 (or other motion models associated with the actor's performance being captured). A data comparer 828 is also executed by the computer system 814 to compare the contents of the captured image 818 and the motion model 802. By comparing surface features of an object (e.g., an actor's face) in the captured image 818 to the contents of the motion model 802, the data comparer 828 may identify the actor's facial expression (in the image 818) from the expressions represented by the motion model 802. Upon identification, the motion model 802 may be used to adjust the shape of an animation mesh to animate a computer generated character (or other type of object). For example, one or more vertices, shapes or types of structures (e.g., animated curved facial features) of the animation mesh may be adjusted to represent the actor's facial expression. In this arrangement, a data transferor 830 is executed to transfer the facial expression or other type of motion information (associated with the motion model 802) to an animation mesh. While this exemplary expression identification system 800 includes three separate processes (e.g., the data receiver 826, the data comparer 828, and the data transferor 830) to provide the functionality of receiving, comparing and transferring data, in some implementations the functionality may be combined into fewer processes or separated into additional processes.
The expression identification system 800 uses one camera 816 to collect one or more images for triggering the use of an appropriate motion model for expression identification and animation mesh adjustment. However, in some systems, additional cameras or other types of image capture devices may be implemented. For example, the system 800 may include an additional camera that may (or may not) be time-synchronized with the camera 816. By capturing multiple images (e.g., from different perspectives), more content may be provided to the system 800 to possibly reduce the probability of identification error along with reducing the time needed to identify a facial expression.
Referring to FIG. 9, a flowchart 900 represents some of the operations of one or more of the processes (e.g., the data receiver 826, the data comparer 828 and the data transferor 830) individually or in combination. The operations may be executed by a single computer system (e.g., computer system 814) or multiple computing devices. Along with being executed at a single site (e.g., at one computer system), operation execution may be distributed among two or more sites.
Operations include receiving 902 a captured image from one or more cameras (or other type of image capture devices). Operations also include receiving 904 a motion model such as one of the motion models 802-808 stored in the motion library 810 in the storage device 812. Typically, the received motion model is produced from a collection of facial expressions of an actor's performance that may be incorporated into an animated character. In some implementations, the motion model includes decomposed data such as principle components. By processing the decomposed data (e.g., applying weighting factors to the principle components), the previously collected facial expressions associated with the actor's performance may be reconstructed (or new expressions may be constructed).
Operations also include identifying 906 one or more surface features included the received image for identifying the expression in the image. For example one or more artificial points (e.g., markers) or natural points (e.g., batch of textured skin, face contours, etc.) may be used as identification surface features. In some arrangements one or more of the identification surface features may directly map to surface features of previously collected images (used to produce the motion model), however, in other arrangements some or all of the points may not directly map. For example, artificial markers may be used during image capture (e.g., by the image capture system 100) to produce the motion model while nature contours (e.g., curve of the actor's lip or eyebrow, etc.) may be used (e.g., by the expression identification system 800) as identification points.
Operations also include comparing 908 the motion model to the one or more identification surface features of the captured image. Through this comparison, the expression identification system 800 may identify the facial expression present in the captured image. Accordingly, data representing the identified expression may be transferred to an animation mesh for character animation. One or more techniques may be implemented to provide the comparison. For example, weighting factors may be applied to decomposed data (e.g., principle components) included in the motion model to produce one or more facial expressions that may match the expression in the captured image. Thus, captured facial expression data may be compared to facial expression data produced by a motion model. In other implementations, the comparison may be performed with other types of data. For example, decomposed data (e.g., principle components) associated with the motion model may be compared to decomposed data computed from the captured image. Thus, comparisons may be based upon decomposition or other types of data processing of the captured image.
Upon comparing the surface features of the captured image and the motion model, operations include determining 910 if a match has been detected. In this implementation, if a match is not detected, operations may include returning to receive 902 another image (e.g., from the camera 816). Other operations, not illustrated in flowchart 900 may also be executed absent a match. For example, an alert (e.g., a visual and/or audio message or signal, etc.) may be issued from the computer system 814 if a match is not detected.
If a match is detected, operations may include adjusting 912 an animation mesh or other type of animation object to represent the identified facial expression. For example, data associated with the motion model may be used to move vertices, shapes or other types of structures included in an animation mesh that illustrate motion.
To perform the operations described in flow chart 900, computer system 814 (shown in FIG. 8) may perform any of the computer-implement methods described previously, according to one implementation. The computer system 814 may include a processor (not shown), a memory (not shown), a storage device (e.g., storage device 812), and an input/output device (not shown). Each of the components may be interconnected using a system bus or other similar structure. The processor is capable of processing instructions for execution within the computer system. In one implementation, the processor is a single-threaded processor. In another implementation, the processor is a multi-threaded processor. The processor is capable of processing instructions stored in the memory or on the storage device to display graphical information for a user interface on the input/output device.
The features described can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The apparatus can be implemented in a computer program product tangibly embodied, e.g., in a machine-readable storage device, for execution by a programmable processor; and method steps can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5790124Nov 20, 1995Aug 4, 1998Silicon Graphics, Inc.System and method for allowing a performer to control and interact with an on-stage display deviceUS5831260Sep 10, 1996Nov 3, 1998Ascension Technology CorporationHybrid motion trackerUS5932417Oct 15, 1996Aug 3, 1999The Regents Of The University Of CaliforniaMethod of screening compounds for controlling capacitative calcium ion entry into mammalian cellsUS6072496Jun 8, 1998Jun 6, 2000Microsoft CorporationMethod and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objectsUS6115052Feb 12, 1998Sep 5, 2000Mitsubishi Electric Information Technology Center America, Inc. (Ita)System for reconstructing the 3-dimensional motions of a human figure from a monocularly-viewed image sequenceUS6166811Aug 12, 1999Dec 26, 2000Perceptron, Inc.Robot-based gauging system for determining three-dimensional measurement dataUS6208348May 27, 1998Mar 27, 2001In-Three, Inc.System and method for dimensionalization processing of images in consideration of a pedetermined image projection formatUS6324296Dec 4, 1997Nov 27, 2001Phasespace, Inc.Distributed-processing motion tracking system for tracking individually modulated light pointsUS6353422Mar 31, 2000Mar 5, 2002Stephen G. PerlmanVirtual display system and methodUS6438255Apr 14, 2000Aug 20, 2002Stress Photonics, Inc.Transient thermal marking of objectsUS6515659Dec 19, 2001Feb 4, 2003In-Three, Inc.Method and system for creating realistic smooth three-dimensional depth contours from two-dimensional imagesUS6522332Jul 26, 2000Feb 18, 2003Kaydara, Inc.Generating action data for the animation of charactersUS6606095 *Jun 7, 1999Aug 12, 2003Microsoft CorporationCompression of animated geometry using basis decompositionUS6614407Jan 4, 2002Sep 2, 2003Stephen G. PerlmanVirtual display system and methodUS6614428 *Jun 7, 1999Sep 2, 2003Microsoft CorporationCompression of animated geometry using a hierarchical level of detail coderUS6633294Mar 9, 2001Oct 14, 2003Seth RosenthalMethod and apparatus for using captured high density motion for animationUS6686926Mar 26, 2001Feb 3, 2004In-Three, Inc.Image processing system and method for converting two-dimensional images into three-dimensional imagesUS6919892 *Aug 14, 2002Jul 19, 2005Avaworks, IncorporatedPhoto realistic talking head creation system and methodUS6977630Jul 18, 2000Dec 20, 2005University Of MinnesotaMobility assist deviceUS7027054 *Aug 14, 2002Apr 11, 2006Avaworks, IncorporatedDo-it-yourself photo realistic talking head creation system and methodUS7035436 *Aug 6, 2002Apr 25, 2006The University Of TokyoMethod of generating poses and motions of a tree structure link systemUS7098920 *Jul 27, 2004Aug 29, 2006Microsoft CorporationMethods and systems for animating facial features and methods and systems for expression transformationUS7102633May 15, 2002Sep 5, 2006In-Three, Inc.Method for conforming objects to a common depth perspective for converting two-dimensional images into three-dimensional imagesUS7116323Dec 10, 2002Oct 3, 2006In-Three, Inc.Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional imagesUS7116324Sep 30, 2003Oct 3, 2006In-Three, Inc.Method for minimizing visual artifacts converting two-dimensional motion pictures into three-dimensional motion picturesUS7129949 *Nov 2, 2004Oct 31, 2006Microsoft CorporationMethods and systems for animating facial features, and methods and systems for expression transformationUS7164718 *Sep 6, 2001Jan 16, 2007France TelecomMethod for segmenting a video image into elementary objectsUS7184047 *Jun 27, 2000Feb 27, 2007Stephen James CramptonMethod and apparatus for the generation of computer graphic representations of individualsUS7212656 *Jan 26, 2006May 1, 2007Microsoft CorporationRapid computer modeling of faces for animationUS7292261Aug 20, 1999Nov 6, 2007Patrick TeoVirtual reality cameraUS7433807 *Sep 1, 2005Oct 7, 2008Microsoft CorporationFacial image processingUS7450126 *Apr 24, 2006Nov 11, 2008Microsoft CorporationMethods and systems for animating facial features, and methods and systems for expression transformationUS7554549 *Nov 8, 2004Jun 30, 2009Sony CorporationSystem and method for tracking facial muscle and eye motion for computer graphics animationUS7605861Oct 20, 2009Onlive, Inc.Apparatus and method for performing motion capture using shutter synchronizationUS8019137Sep 13, 2011Lucasfilm Entertainment Company Ltd.Three-dimensional motion captureUS20010024512Feb 22, 2001Sep 27, 2001Nestor YoronkaOptical body trackerUS20010033675May 31, 2001Oct 25, 2001Thomas MaurerWavelet-based facial motion capture for avatar animationUS20020041285Jun 22, 2001Apr 11, 2002Hunter Peter J.Non-linear morphing of faces and their dynamicsUS20020060649Jan 4, 2002May 23, 2002Perlman Stephen G.Virtual display system and methodUS20040063481Jun 10, 2003Apr 1, 2004Xiaoling WangApparatus and a method for more realistic interactive video games on computers or similar devices using visible or invisible light and an input computing deviceUS20040119716May 28, 2003Jun 24, 2004Chang Joon ParkApparatus and method for high-speed marker-free motion captureUS20040155962Feb 11, 2003Aug 12, 2004Marks Richard L.Method and apparatus for real time motion captureUS20040161132Jan 16, 2004Aug 19, 2004Cohen Charles J.Gesture-controlled interfaces for self-service machines and other applicationsUS20040179008May 1, 2003Sep 16, 2004Sony CorporationSystem and method for capturing facial and body motionUS20050078124Oct 14, 2003Apr 14, 2005Microsoft CorporationGeometry-driven image synthesis renderingUS20050099414May 15, 2002May 12, 2005Kaye Michael C.Method for conforming objects to a common depth perspective for converting two-dimensional images into three-dimensional imagesUS20050104878Dec 10, 2002May 19, 2005Kaye Michael C.Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional imagesUS20050104879Sep 30, 2003May 19, 2005Kaye Michael C.Method for minimizing visual artifacts converting two-dimensional motion pictures into three-dimensional motion picturesUS20050146521Mar 2, 2004Jul 7, 2005Kaye Michael C.Method for creating and presenting an accurate reproduction of three-dimensional images converted from two-dimensional imagesUS20050231505Jun 30, 2004Oct 20, 2005Kaye Michael CMethod for creating artifact free three-dimensional images converted from two-dimensional imagesUS20060055699Sep 15, 2004Mar 16, 2006Perlman Stephen GApparatus and method for capturing the expression of a performerUS20060055706Sep 15, 2004Mar 16, 2006Perlman Stephen GApparatus and method for capturing the motion of a performerUS20060067573Sep 19, 2005Mar 30, 2006Parr Timothy CSystem, method, and apparatus for generating a three-dimensional representation from one or more two-dimensional imagesUS20060126928Dec 2, 2005Jun 15, 2006Image Metrics LimitedMethod and system for cleaning motion capture dataUS20060157640Jan 18, 2005Jul 20, 2006Perlman Stephen GApparatus and method for capturing still images and video using coded aperture techniquesUS20060192785Apr 24, 2006Aug 31, 2006Microsoft CorporationMethods and systems for animating facial features, and methods and systems for expression transformationUS20060192854Feb 25, 2005Aug 31, 2006Perlman Stephen GApparatus and method improving marker identification within a motion capture systemUS20060228101Mar 16, 2006Oct 12, 2006Steve SullivanThree-dimensional motion captureUS20070052711Aug 25, 2006Mar 8, 2007Demian GordonReconstruction render farm used in motion captureUS20070091178Oct 20, 2005Apr 26, 2007Cotter Tim SApparatus and method for performing motion capture using a random pattern on capture surfacesUS20070133841Feb 8, 2007Jun 14, 2007Xiang ZhangCoded visual markers for tracking and camera calibration in mobile computing systemsUS20080170077Jan 16, 2007Jul 17, 2008Lucasfilm Entertainment Company Ltd.Generating Animation LibrariesUS20080170777Apr 13, 2007Jul 17, 2008Lucasfilm Entertainment Company Ltd.Combining multiple session content for animation librariesUS20080180448Jul 25, 2007Jul 31, 2008Dragomir AnguelovShape completion, animation and marker-less motion capture of people, animals or charactersUS20090209343Feb 15, 2008Aug 20, 2009Eric FoxlinMotion-tracking game controllerUS20100002934Sep 14, 2009Jan 7, 2010Steve SullivanThree-Dimensional Motion CaptureUS20100164862Jul 21, 2009Jul 1, 2010Lucasfilm Entertainment Company Ltd.Visual and Physical Motion Sensing for Three-Dimensional Motion CaptureEP1946243A2Oct 4, 2006Jul 23, 2008Intersense, Inc.Tracking objects with markersJPH1084691A Title not availableWO2004041379A2Oct 30, 2003May 21, 2004Nike Inc.Clothes with tracking marks for computer games* Cited by examinerNon-Patent CitationsReference1 *Agarwala et al. "Keyframe-BAsed Tracking for Rotoscoping and Animation". ACM, 2004.2Bascle, Blake, "Separability of pose and expression in facial tracking and animation," Jan. 1998, IEEE Computer Society, Proceedings of the Sixth International Conference on Computer Vision, pp. 323-328.3 *Buenaposada et al. "Performance driven facial animation using illumination independent appearance-based tracking". Aug. 2006.4Chang et al., "Automatic 3D Facial Expression in Analysis in Videos," Proc. IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures (AMFG '05), vol. 3723, pp. 293-307, 2005.5 *Cristobal Curio et al. "Semantic 3D Motion Retargeting for Facial Animation". ACM, Jul. 2006.6Debevec, et al. "Acquiring the Reflectance Field of a Human Face", SIGGRAPH 2000 Conference Proceedings, pp. 1-12, 2000.7Deng et al., "Animating blendshape faces by cross-mapping motion capture data," In Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games (Redwood City, California, Mar. 14-17, 2006). I3D '06. ACM, New York, NY, 43-48.8Examination Report from European Patent Office for Application No. GB0718003.7, dated Aug. 3, 2009, 2 pages.9Examination Report from New Zealand Patent Office for Application No. 561570, dated Dec. 18, 2009, 1 page.10Examination Report from New Zealand Patent Office for Application No. 561570, dated Jul. 7, 2009, 2 pages.11Examination Report from New Zealand Patent Office for Application No. 581496,dated Dec. 2, 2009, 2 pages.12Examination Report from New Zealand Patent Office for Application No. 582356, dated Jan. 7, 2010, 2 pages.13Examination Report from United Kingdom Office for Application No. PCT/US2006/009787, dated Aug. 3, 2009, 2 pages.14Furukawa et al., "Dense 3D Motion Capture from Synchronized Video Streams," [online], [retrieved on Aug. 26, 2009]. Retrieved from the Internet , 8 pages.15Furukawa et al., "Dense 3D Motion Capture from Synchronized Video Streams," [online], [retrieved on Aug. 26, 2009]. Retrieved from the Internet <URL: http://www.cs.washington.edu/homes/furukawa/papers/cvpr08b.pdf>, 8 pages.16 *Gleicher "Animation From Observation: Motion Capture and Motion Editing". ACM 1999.17Grung et al. "Missing values in principal component analysis", Chemometrics and Intelligent Laboratory Systems, vol. 42, No. 1-2, pp. 125-139, Aug. 1998.18Heap et al. "Towards 3D hand tracking using a deformable model", IEEE Computer Society, pp. 140-145, Oct. 1996.19Herda et al. "Skeleton-based motion capture for robust reconstruction of human motion", Computer Animation 2000, Proceedings, pp. 77-83, May 2000.20Horprasert et al., "Real-time 3D Motion Capture," Second workshop on Perceptual Interfaces, San Francisco, Nov. 1998, 4 pages.21International Preliminary Report on Patentability for PCT Application No. PCT/US2006/009787, dated Nov. 13, 2007, 9 pages.22International Search Report and Written Opinion for PCT Application No. PCT/US2006/009787, dated Oct. 29, 2007, 12 pages.23Lin et al. "Extracting 3D facial animation parameters from multiview video clips", IEEE Computer Graphics and Applications, vol. 22, No. 6, pp. 72-80, Nov. 2002.24Majkowska, A. et al., "Automatic splicing for hand and body animations." 2006. In Proceedings if the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation (SCA '06). Eurographics Associate, Aire-la-Ville, Switzerland, Switzerland, pp. 309-316 and 369.25Markoff, "Camera System Creates Sophisticated 3-D Effects" [online], New York Times, Jul. 31, 2006, [retrieved on Jul. 31, 2006]. Retrieved from the Internet: , 4 pg.26Markoff, "Camera System Creates Sophisticated 3-D Effects" [online], New York Times, Jul. 31, 2006, [retrieved on Jul. 31, 2006]. Retrieved from the Internet: <URL: www.nytimes.com/2006/0731/technology/31motion.html >, 4 pg.27 *Michael Gleicher "Animation From Observation: Motion Capture and Motion Editing". Computer Graphics 33(4), pp. 51-54.28 *Park "Capturing and Animating Skin Deformation in Human Motion". ACM 2006.29Parke, F., "Computer Generated Animated of Faces," Aug. 1972, SIGGRAPH 1972, pp. 451-457.30Search Report from United Kingdom Intellectual Property Office for application No. GB0922650.7, dated Apr. 20, 2010, 2 pages.31Stopford, Office Action issued in AU application No. 2009240847 mailed Nov. 23, 2011, 4 pages.32Wren et al, "Dynamic Models of Human Motion," [online], 1998, [retrieved on Aug. 26, 2009]. Retrieved from the Intern et , 6 pages.33Wren et al, "Dynamic Models of Human Motion," [online], 1998, [retrieved on Aug. 26, 2009]. Retrieved from the Intern et <URL:http://www.drwren.com/chris/dyna/TR-415.pdf>, 6 pages.* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS8472700 *Dec 15, 2008Jun 25, 2013Electronics And Telecommunications Research InstituteMethod and apparatus for creating 3D face model by using multi-view image informationUS8542236Jan 16, 2007Sep 24, 2013Lucasfilm Entertainment Company Ltd.Generating animation librariesUS8681158Mar 5, 2012Mar 25, 2014Lucasfilm Entertainment Company Ltd.Using animation libraries for object identificationUS8805002 *Dec 23, 2010Aug 12, 2014Metalo GmbHMethod of determining reference features for use in an optical object initialization tracking process and object initialization tracking methodUS8908960Sep 9, 2011Dec 9, 2014Lucasfilm Entertainment Company Ltd.Three-dimensional motion captureUS8913792 *Jun 13, 2014Dec 16, 2014Metaio GmbhMethod of determining reference features for use in an optical object initialization tracking process and object initialization tracking methodUS8928674Jun 8, 2012Jan 6, 2015Lucasfilm Entertainment Company Ltd.Combining multiple session content for animation librariesUS8941665Mar 21, 2012Jan 27, 2015Lucasfilm Entertainment Company Ltd.Model production for animation librariesUS9020192Mar 13, 2013Apr 28, 2015Access Business Group International LlcHuman submental profile measurementUS9142024Jul 21, 2009Sep 22, 2015Lucasfilm Entertainment Company Ltd.Visual and physical motion sensing for three-dimensional motion captureUS9177410 *Aug 9, 2013Nov 3, 2015Ayla MandelSystem and method for creating avatars or animated sequences using human body features extracted from a still imageUS9208608Feb 25, 2013Dec 8, 2015Glasses.Com, Inc.Systems and methods for feature trackingUS9235929Feb 22, 2013Jan 12, 2016Glasses.Com Inc.Systems and methods for efficiently processing virtual 3-D dataUS9236024Dec 6, 2012Jan 12, 2016Glasses.Com Inc.Systems and methods for obtaining a pupillary distance measurement using a mobile computing deviceUS9286715Feb 25, 2013Mar 15, 2016Glasses.Com Inc.Systems and methods for adjusting a virtual try-onUS9311746Feb 22, 2013Apr 12, 2016Glasses.Com Inc.Systems and methods for generating a 3-D model of a virtual try-on productUS20080170077 *Jan 16, 2007Jul 17, 2008Lucasfilm Entertainment Company Ltd.Generating Animation LibrariesUS20090153553 *Dec 15, 2008Jun 18, 2009Electronics And Telecommunications Research InstituteMethod and apparatus for creating 3D face model by using multi-view image informationUS20100164862 *Jul 21, 2009Jul 1, 2010Lucasfilm Entertainment Company Ltd.Visual and Physical Motion Sensing for Three-Dimensional Motion CaptureUS20110194731 *Dec 23, 2010Aug 11, 2011Metaio GmbhMethod of determining reference features for use in an optical object initialization tracking process and object initialization tracking methodUS20140321705 *Jun 13, 2014Oct 30, 2014Metaio GmbhMethod of determining reference features for use in an optical object initialization tracking process and object initialization tracking methodUS20150042663 *Aug 9, 2013Feb 12, 2015David MandelSystem and method for creating avatars or animated sequences using human body features extracted from a still image* Cited by examinerClassifications U.S. Classification345/473, 382/128, 382/154, 345/474, 345/156International ClassificationG06T15/70Cooperative ClassificationG06K9/00281, G06T17/20, G06K9/00315European ClassificationG06T17/20, G06K9/00GLegal EventsDateCodeEventDescriptionSep 13, 2007ASAssignmentOwner name: LUCASFILM ENTERTAINMENT COMPANY LTD., CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SULLIVAN, STEVE;CALLARI, FRANCESCO;REEL/FRAME:019830/0857;SIGNING DATES FROM 20070712 TO 20070723Owner name: LUCASFILM ENTERTAINMENT COMPANY LTD., CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SULLIVAN, STEVE;CALLARI, FRANCESCO;SIGNING DATES FROM 20070712 TO 20070723;REEL/FRAME:019830/0857May 1, 2012CCCertificate of correctionAug 19, 2015FPAYFee paymentYear of fee payment: 4RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services