3D IMAGE PROCESSING METHOD AND APPARATUS

Abstract: In a method of personalising a computer game character, at least one two-dimensional image (I) of a character (1) of the game (e.g. the user of a personal computer connected to the Internet) is digitally processed at a server (S) remote from the player's computer by distorting a generic image (G) to match the user's face or head, converted to an animatable character file and downloaded onto the player's computer. Preferably two overlapping images are acquired by a digital camera and uploaded to the server where they are stereoscopically processed to provide an incomplete 3D representation (I) to which is compared the generic 3D representation (G), the latter being distorted to match the former.

Description: 3D IMAGE PROCESSING METHOD AND APPARATUS The present invention relates to a method and apparatus for processing facial images, particularly but not exclusively for use in digital animation eg computer games. Photogrammetric techniques are known for converting two or more overlapping 2D images acquired from different viewpoints into a common 3D representation and in principle such techniques can be applied to the human face to generate a 3D representation which can be animated using known digital techniques. Suitable algorithms for correlating image regions of corresponding images (eg photographs taken during airborne surveys) are already known - eg Gruen's algorithm (see Gruen, A W "Adaptive least squares correlation: a powerful image matching technique"S Afr J of Photogrammetry, remote sensing and Cartography Vol 14 No 3 (1985) and Gruen, A W and Baltsavias, E P "High precision image matching for digital terrain model generation" Int Arch photogrammetry Vol 25 No 3 (1986) p254) and particularly the "region-growing" modification thereto which is described in Otto and Chau "Region-growing algorithm for matching terrain images" Image and Vision Computing Vol 7 No 2 May 1989 p83. Essentially, Gruen's algorithm is an adaptive least squares correlation algorithm in which two image patches of typically 15 x 15 to 30 x 30 pixels are correlated (ie selected from larger left and right images in such a manner as to give the most consistent match between patches) by allowing an affine geometric distortion between coordinates in the images (ie stretching or compression in which originally parallel lines remain parallel in the transformation) and allowing an additive radiometric distortion between the grey levels of the pixels in the image patches, generating an over-constrained set of linear equations representing the discrepancies between the correlated pixels and finding a least squares solution which minimises the discrepancies. The Gruen algorithm is essentially an iterative algorithm and requires a reasonable approximation for the correlation to be fed in before it will converge to the correct solution. The Otto and Chau region-growing algorithm begins with an approximate match between a point in one image and a point in the other, utilises Gruen's algorithm to produce a more accurate match and to generate the geometric and radiometric distortion parameters, and uses the distortion parameters to predict _ approximate matches for points in the region of the neighbourhood of the initial matching point. The neighbouring points are selected by choosing the four adjacent points on a grid having a grid spacing of eg 5 or 10 pixels in order to avoid running Gruen's algorithm for every pixel. Hu et al "Matching Point Features with ordered Geometric, Rigidity and Disparity Constraints" IEEE Transactions on Pattern Analysis and Machine Intelligence Vol 16 No 10, 1994 ppl041-1049 (and references cited therein) discloses further methods for correlating features of overlapping images. Our co-pending patent applications disclose a number of improvements to the Gruen algorithm, as follows: i) the additive radiometric shift employed in the algorithm can be dispensed with; ii) if during successive iterations, a candidate matched point moves by more than a certain amount (eg 3 pixels) per iteration then it is not a valid matched point and should be rejected; iii) during the growing of a matched region it is useful to check for sufficient contrast at at least three of the four sides of the region in order to ensure that there is sufficient data for a stable convergence - in order to facilitate this it is desirable to make the algorithm configurable to enable the parameters (eg required contrast) to be optimised for different environments, and iv) in order to quantify the validity of the correspondences between respective patches of one image and points in the other image it has been found useful to re- derive the original grid point in the starting image by applying the algorithm to the matched point in the other image (ie reversing the stereo matching process) and measuring the distance between the original grid point and the new grid point found in the starting image from the reverse stereo matching. The smaller the distance the better the correspondence. However the known photogrammetric techniques still require correlations between high quality overlapping images and, in cases where there is little texture information in the subject (which is true of large regions of the human face) it is difficult or impossible to correlate all the regions, which results in holes in the 3D reconstruction. Such difficulties can be overcome by projecting an optical (particularly infra-red) pattern (particularly a speckle pattern) onto the subject but the requirement for pattern projection increases the expense of an already sophisticated apparatus. One object of the present invention is to overcome or alleviate such disadvantages. In one aspect the present invention provides a method of providing a three- dimensional representation of an object wherein two or more two-dimensional images of the object are photogrammetrically processed to generate an incomplete three-dimensional representation thereof and the incomplete three-dimensional representation is combined with a generic representation of such objects to provide the three-dimensional representation. In one embodiment the object is a human or animal body or a part thereof. In a preferred embodiment the three-dimensional representation derived from the combination with the generic representation is provided in the format of an animatable character. In a preferred embodiment the resulting three-dimensional representation is converted to the file format of a computer game character and loaded into the computer game. In another aspect the invention provides a method of personalising a computer game character wherein at least one image of a player of the game is digitally processed at a location remote from the player's computer, converted to an animatable character file and loaded onto the player's computer. For example the image can be processed on an Internet server computer and downloaded over the Internet. Further preferred features are defined in the dependent claims. In a preferred embodiment a fully automated system is provided whereby Quake, Doom, Descent and other popular games users, can be provided with a custom game character with their own face (preferably a 3D face) inserted into the character. This would enable them to use a visualisation of themselves in a game. This service could be provided via the Internet with little or no human intervention by the operator of the server. Nothing similar currently exists with this level of accessibility by the gaming public. By using a generic head during image processing, a relatively low quality of 3D surface is required in order to get an acceptable result, and the problems of holes in 3D data-sets can be eliminated. A low-resolution model is required for gaming, as the game will have to support the manipulation of the character in the gaming environment, in real time, on a variety of PCs. It is assumed that if necessary, a user would tolerate a few hours turnaround time between submitting their images and receiving a model either on a data medium such as floppy disk or by email. In one embodiment the games user would be required to take a set of images of himself/herself using a digital camera or scanned photographs, under specified guidelines. He/she will then access aWeb page hosted by the server, which will provide a form, requiring the user to enter the following information: * Name * Email address * Which game he/she wants the model for. * The images which are to be submitted. * Select a body on which he/she wants the face inserted. * Credit Card details The server will then schedule an image processing job to perform the following tasks: * Determine 3D facial geometry from the supplied image files * Modify a generic head to this geometry * Apply a texture map from the supplied images * Polygon reduce the head model * Integrate the head model with a body * Convert to complete model to the required format for the specified game. After appropriate processing, the completed character would be sent as an attachment to the specified email address, and a micro transaction performed to bill the user's credit card. A preferred embodiment of the invention is described below by way of example only with reference to Figures 1 to 3 of the accompanying drawings, wherein: Figure 1 is a schematic flow diagram of an image processing method in accordance with one aspect of the invention; Figure 2A is a schematic plan view showing one camera arrangement for acquiring the images utilised in the method of Figure 1 ; Figure 2B is a schematic plan view of another camera arrangement for acquiring the ima *goe"s utilised in the method of Figure 1; Figure 2C is a schematic plan view of yet another camera arrangement for acquiring the images utilised in the method of Figure 1, and Figure 3 is a schematic representation of an Internet-based arrangement for providing an animated games character by a method in accordance with the second aspect of the invention. Referring to Figure 1, left and right images II and 12 are acquired eg by a digital camera and processed by standard photogrammetric techniques to provide an incomplete 3D representatation 100 of the game player's head. The determination of the game player's facial geometry can involve Gruens type area matching, facial feature correlation, and facial feature recognition via a statistical model of the human face. Gruens type area matching suffers from the problem of having no projected texture, and is thus highly susceptible to the texture in the face of the subject, the ambient lighting conditions, and the difference in colour hues and intensities between images. It is also susceptible to the lack of camera model or optical geometry of the captured images. Facial feature correlation suffers from the problem that any facial feature that is incorrectly detected will cause a very poor model to be generated. Facial feature recognition via a statistical model prevents gross inaccuracies from occurring and should lead to a more robust solution. It is possible that part of the image submission process could involve the user in specifying certain key points on the images. In order to alleviate the above problems, a 3D representation of a generic head 200 is provided. Given geometric information derived from the preceding stage, the generic head model can be distorted to fit the subject's roughly calculated geometry. This head could be in one of two forms, a NURBS (Non-Uniform Rational B Spline) model, or a polygon model. The NURBS Model has the advantage of being easily deformable to the subject's geometry, but suffers from the drawback of higher processing overhead, and having to convert to polygons for subsequent processing stages. At this stage of processing (modified generic head 300) there should already be a correlation between certain points in each image, and points on the 3D model, greatly simplifying the task of texture mapping. There remains the problem of texture merging arising from the use of multiple images. A texture map is derived (400) from the 3D head (100) and attached to the representation resulting from step 300 (step 500) (ie used to render the modified generic head) and the resulting realistic character representation is then integrated with or attached to the body of the games character (step 600). If necessary the resulting model is converted to polygon form (step 700). If the modified generic head is represented in polygon form, the number of polygons may have to be reduced (step 800). There are plenty of algorithms and commercially available code for polygon reduction. The completed model may be reduced to quite a low polygon count, possibly 100 or so, in order to produce a relatively small model to transmit and use within the game. Finally the polygon-reduced representation is converted to a games file format which can be handled by the game (step 900). This last step may require liaison and co-operation with the games manufacturers, or it is conceivable that this task could be performed completely independently. The acquisition of the 2D images II and 12 will now be described with reference to Figures 2A, 2B and 2C. Each of these Figures shows different camera arrangements which could be provided as fixed stereoscopic camera arrangements in a dedicated booth provided in (say a gaming arcade) or could be set up by the games player. In each case a camera C acquires an image from one viewpoint and the same or a different camera C acquires an overlapping image from a different viewpoint. The fields of view V must overlap in the region of the face of the subject 1. In Figure 2A the cameras are diagonally disposed at right angles, in Figure 2B the cameras are parallel and in Figure 2C the cameras are orthogonal, so that one camera has a front view and the other camera has a profile view of the subject 1. The arrangement of Figure 2C is particularly preferred because the front view and profile are acquired independently. The front image and profile image can be analysed to determine the size and location of features and the resulting data can be used to select one of a range of generic heads or to adjust variable parameters of the generic head as shown, prior to step 300. By correlating a small number of points of the digitised images by means of a known algorithm (eg the Gruen algorithm) the exact camera locations and orientations can be determined and the remaining points correlated relatively easily to enable a 3D representation of the subject 1 to be generated, essentially be projecting ray lines from pairs of correlated points by virtual projectors having the same location, orientation and optical parameters as the cameras. Referring to Figure 3, the above correlation process between the generic image G and the image I of the character 1 provided by digital camera C can be performed by a server computer S on the Internet and the 2D images acquired by the camera C can either be posted by the games player (eg as photographic prints) or uploaded (eg as email attachments) onto the server from the user's computer PC via a communications link CL provided by the Internet. To this end the server computer S has stored on its hard disc HD: i) the software required to implement the processes outlined in Figure 1 including file format conversion software for the common computer games, graphics software 0 image correlation software (eg based on the Gruen algorithm or a variant thereof) and stereoscopic image processing software; ii) software to generate a WWW submission form F on the user's computer screen 5 and to process the personal information entered therein by the user, eg credit card details and the required game format of the character; iii) appropriate Internet server software including appropriate security software, and 0 iv) an appropriate operating system. Since items iii) and iv) are well known per se and items i) and ii) have already been described in sufficient detail for programmers of reasonable skill to write the i necessary code, no further description is necessary. The use's computer PC would have stored on its hard disc HD one or more games programs, Internet access software, graphics software for handling the images 0 provided by camera C and a conventional operating system, eg Windows 95® or Windows 98®. Both computer PC and computer S are provided with a standard microprocessor μP eg an Intel Pentium® processor as well as RAM and ROM and appropriate input/output circuitry I/O connected to standard modems M or other 5 comunication devices. It is expected that the WWW submission form F would be based on a Java Applet to allow the validation of the quality of the submitted images, and the selection of body Q types. It is likely that the server operator would want to test images for their size, resolution, and possibly their contrast ratio, before accepting them for processing. If this can be done by an applet before accepting any credit card transaction, then it will help to reduce bad conversions. By weeding out potential failures at an early _ stage, this will reduce wasted processing time, and will reduce customer frustration by not having to wait a few hours to find out that the images were not of sufficient quality to produce a character. Given the correct design of the web page/form, a valuable database could be constructed of games users, which may be sold or used for mail-shots for future developments. In addition to the basic information requested on the Internet submission form F, the operator can request other information of the user for his own uses, namely: * Which games he plays * Age * How he found out about us. It would also be possible to take a single frontal photograph of the subject, detect facial features, and map the image onto a generic model. As a large percentage of the brain is dedicated to the task of facial recognition, the model may be very approximate indeed to the actual geometry of the subject's face, and the texture map need only be very low resolution. This may not be acceptable for higher resolution models that may be required for games such as Tomb Raider. Although the preferred embodiment is based on an Internet server, the invention can also be implemented in a purpose-built games booth at which the images II and 12 are acquired, and the processing can be carried out either locally in the booth or remotely eg in a server computer linked to a number of such booths in a network. In a variant, more than two cameras could be used to acquire the 3D surface of the character.

Claims: Claims 1. A method of providing a three-dimensional representation of an object (1) wherein two or more two-dimensional images (II, 12) of the object are photogrammetrically processed (100) to generate an incomplete three-dimensional representation (I) thereof and the incomplete three-dimensional representation is combined (300) with a generic representation (G) of such objects to provide the three-dimensional representation. 2. A method as claimed in claim 1 wherein the object (1) is a human or animal body or a part thereof. 3. A method as claimed in claim 1 or claim 2 wherein the three-dimensional representation derived from the combination with the generic representation (G) is provided in the format of an animatable character. 4. A method as claimed in claim 3 wherein the file of the animatable character is loaded into a computer game. 5. A method of personalising a computer game character wherein at least one two- dimensional image (II, 12) of a character of the game is digitally processed at a location remote from a player's computer, converted to an animatable character file (900) and loaded onto the player's computer. 6. A method as claimed in claim 5 wherein at least two images (II, 12) of the character are stereoscopically processed (100) at said remote location to provide a three-dimensional animatable character file. 7. A method as claimed in any of claims 3 to 6 wherein the animatable character file is downloaded from a remote server (S) via a communications link (CL). 8. A method as claimed in claim 7 wherein the animatable character file is downloaded over the Internet. 9. A method as claimed in any preceding claim wherein at least one two-dimensional image (II, 12) is uploaded to a remote server (S) at which it is stereoscopically processed. 10. A method as claimed in any of claims 1 to 4 or claim 6 or any of claims 7 to 9 as dependent on claim 6 wherein a three-dimensional geometric representation (I) of the object or character (1) is combined (300) with a generic three-dimensional geometric representation (G) of such objects or characters to generate a modified generic geometric representation and a texture map is combined (500) with the generic geometric representation. 11. A method as claimed in claim 10 wherein the generic three-dimensional representation is in the form of a NURBS (Non-Uniform Rational B-Spline) model and is distorted in dependence upon detected features of the object or character (1). 12. A method as claimed in any of claims 1 to 4 or claim 6 or any of claims 7 to 9 as dependent on claim 6 wherein a two-dimensional image (II, 12) of the object or character (1) is analysed to detect features thereof and a generic representation (G) of the object or character is distorted to match its features with those of said two- dimensional image. 13. A method as claimed in claim 13 wherein two such dimensional images (II, 12) acquired from substantially orthogonal viewpoints are analysed to detect features thereof and said generic representation (G) is distorted in three dimensions to match its features with those of both two-dimensional images. 14. A method as claimed in any preceding claim wherein said two-dimensional images (11 , 12) are images of a human face or head ( 1 ). 15. A method as claimed in any preceding claim wherein a three-dimensional representation (I) of an object or character (1) is provided (700) in the form of a polygon model and the polygon model is polygon-reduced (800).

Inventors: FLOCKHART CHRISTOPHER PETER, HUGHES DUNCAN, FLOCKHART, CHRISTOPHER, PETER, HUGHES, DUNCAN
Applicants: FLOCKHART CHRISTOPHER PETER, HUGHES DUNCAN, TRICORDER TECHNOLGY PLC, FLOCKHART, CHRISTOPHER, PETER, HUGHES, DUNCAN
Application year: 1999
Main classification: G06T15
IPC codes: G06T1, G06T15, G06T17, H04N13