Abstract:
Although in my co-pending application, so-called “pyramidic panel structures” are employed to adjust the image&#39;s vanishing point(s) in accordance with the movement of the user, such structures have the disadvantage that they work primarily for so-called “corridor-like” two-dimensional images. I have recognized that, in accordance with the principles of the invention, viewpoint changes may be also dealt with by partitioning the two-dimensional image into polygon partitions, each corresponding to a surface being depicted in the image, and then coupling the vertices of the polygon partitions with the movement of the user so as to limit distortions. In this latter approach, two-dimensional images are not treated as a perspective, corridor-like image, thereby allowing a wider variety of two-dimensional images to be used with the present invention.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is related to applicant&#39;s U.S. patent application Ser. No. 09,190,743, filed on Nov. 12, 1998, which is commonly assigned and incorporated herein by reference. 
    
    
     TECHNICAL FIELD 
     This invention relates to the integration of three-dimensional computer graphics and a two-dimensional image to provide a realistic three-dimensional virtual reality experience. 
     BACKGROUND OF THE INVENTION 
     The display of a three-dimensional virtual reality world to a user requires considerable computation power, and it is typically costly to develop the necessary highly detailed models required for doing so. In order to simplify the problem, two-dimensional images, such as videos or photographs, may be used to represent or simulate portions of the three-dimensional world. A great reduction in computation power and cost can be achieved by such an arrangement. 
     A limitation of such a world occurs when a user moves within the world and views the world from a location different than the original context of the two-dimensional image which has been carefully calibrated to “fit into” the world. View changes, such as from a location different than the image&#39;s ideal viewing point also known as the “center of projection,” result in the image not aligning or fitting well with surrounding objects of the three-dimensional world. Display techniques, however, to deal with such viewpoint changes are described in applicant&#39;s co-pending application which discloses manipulating the two-dimensional image through the use of so-called “pyramidic panel structures.” More specifically, the two-dimensional image is manipulated so as to adjust the image&#39;s vanishing point(s) in accordance with the movement of the user. In this manner, as the user moves away from the ideal viewing point, warping acts to limit the discontinuities between the two-dimensional image and the surroundings of the world. However, these latter novel display techniques have the disadvantage that they work primarily for so-called “corridor-like” two-dimensional images, such as roads, or streets, wherein parallel features appear to converge at a so-called “vanishing point.” 
     SUMMARY OF THE INVENTION 
     Although in my co-pending application, so-called “pyramidic panel structures” are employed to adjust the image&#39;s vanishing point(s) in accordance with the movement of the user, such structures have the disadvantage that they work primarily for “corridor-like” two-dimensional images. I have recognized that, in accordance with the principles of the invention, viewpoint changes may also be dealt with by partitioning the two-dimensional image into polygon partitions, each corresponding to a surface being depicted in the image, and then coupling the vertices of the polygon partitions with the movement of the user so as to limit distortions. More specifically, image points located farther away from the view plane oft he user move more closely with the movement oft he user&#39;s current view point than those image points located closer. In this manner, as the viewer moves away from the image&#39;s ideal viewing point, IVP, the resulting warping in the image acts to counteract and limit distortions. Importantly, in this latter approach, two-dimensional images are not treated as so-called “corridor-like” images, thereby allowing a wider variety of two-dimensional images to be used with the present invention. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     In the drawings: 
     FIG. 1 shows an example of that which a user sees when a user views the world from the ideal viewing point for a two-dimensional image representing a portion of the world; 
     FIG. 2 shows an example of that which a user sees when a user moves within the world of FIG.  1  and views the two-dimensional image from a location different than the image&#39;s ideal viewing point, without the use of the present invention; 
     FIG. 3 shows an exemplary process, in accordance with the principles of the invention, for manipulating the two-dimensional image in accordance with the movement of the user; 
     FIGS. 4-6 depict partitioning the two-dimensional image into polygon partitions and mapping the vertices thereof in accordance with the movement of the user; and 
     FIGS. 7 and 8 depict examples of that which a user sees when a user views the world from a location left of the image&#39;s ideal viewing point, without and with the use of the present invention, respectively. 
    
    
     DETAILED DESCRIPTION 
     To better understand the invention, FIGS. 1-2 show examples of that which a user sees when the user moves within a three-dimensional virtual reality world (x,y,z) and views a two-dimensional image (x,y) from a location at the image&#39;s ideal viewing point (IVP), and then from a different location, i.e., a location different than the original context of the image. For simplification of terminology purposes, we shall use the term two-dimensional image to denote either a video clip or a photograph. 
     FIG. 1 shows an exemplary three-dimensional reality world  105  which is divided into two portions, a view plane  110  on which is shown or displayed a two-dimensional image  115 , such as a still photograph, picture or a current frame of a video clip; and the remainder of the world  120 . In this case, two-dimensional image  115  is a picture of Bretton Hall located at the intersection of 86 th  street and Broadway in New York City, used to simulate a building  125  within virtual reality world  105  defined by the rectangular coordinates (x,y,z). The remainder of the world  120  will be represented using computer graphics techniques, and is thus referred to herein as computer graphics (CG Part)  120 . Although not shown, within CG Part  120  there may be various synthetic, three-dimensional objects modeled in, for example, the Virtual Reality Modeling Language (VRML). 
     Note that although three-dimensional world  105  cannot be actually rendered in a two-dimensional plane (x,y), it can be projected to and displayed on a view plane so as to appear to have three dimensions (x,y,z). Accordingly, the techniques of the present invention are preferably employed with computers and software, which are sufficiently sophisticated to display images on a two-dimensional plane as having three dimensions. Note that to make the world look realistic, computer graphics display techniques use the z component of objects to scale accordingly the x and y components as a function of its distance (z-axis) to the user&#39;s viewpoint. 
     Two-dimensional image  115  is carefully placed, cropped and sized in the surrounding environment of CG Part  120  as a function of the dimensions of the surroundings within the world. However, when viewing two-dimensional image  115 , there exists only one viewpoint from which the image&#39;s content can be viewed properly. This unique location is called the image&#39;s ideal viewing point (IVP), also known as the “center of projection.” In FIG. 1, two-dimensional image  115  is seen from its ideal viewing point. Users, however, rarely view image  115  only from its idea viewing point. As users move within world  105 , such as to the left or right of building  125 , or move closer to or farther away from the image, they view image  115  from positions other than its ideal viewing point. Absent the use of the present invention, such viewpoint changes would cause features within image  115  to distort as shown in FIG.  2 . 
     Unlike applicant&#39;s above referenced co-pending application, two-dimensional images are not treated as so-called “corridor-like” image, thereby allowing a wider variety of two-dimensional images to be used with the present invention. Similarly, however, as the user moves away from the ideal viewing point, two-dimensional image  115  is manipulated to account for the movement of the user, as discussed herein below. In accordance with the principles of the invention, two-dimensional image  115  is partitioned into separate polygon partitions, each corresponding to a desired surface of objects depicted in the image. Each vertex of the polygon partition is then coupled to the movement of the user with respect to the image&#39;s ideal viewing point, with the coupling strength dependent on the vertex&#39;s distance from view plane  110 , as discussed more fully herein below. The result of this coupling is that objects in two-dimensional image  115  appear to behave as textured-mapped solid models. 
     FIG. 3 shows an exemplary process in accordance with the principles of the invention for manipulating two-dimensional image  115  in accordance with the viewer&#39;s position. The process is entered at step  130  whenever it is determined that the viewer&#39;s position has changed. 
     At step  135 , a vector, {overscore (C)} is projected from the image&#39;s ideal viewing point, IVP, normal to view plane  110  which contains image  115 . At step  140 , objects of interest within two-dimensional image  115  are then partitioned into separate polygon partitions, each corresponding to a desired surface of the object, as depicted in FIG.  4 . Here, and for illustration purposes only, building  125  has been segmented into only two polygon partitions  145  and  150 , corresponding to the left and right sides of the building, respectively. Note that the vertices of the polygon partitions are labeled Q 1 -Q 6 , with their intersection located along line segment {overscore (Q 3 Q 4 )}. 
     Thereafter in step  155 , the current viewpoint of the user, V, is determined, and a vector {overscore (T)} projected from the ideal viewing point, IVP, to the viewer&#39;s current location, V. Next at step  160 , a coupling ratio α i  is calculated for each vertex Q i  according to the following equation:                  α   i     =         D   i     -        C            D   i         ,           Eq   .                (   1   )                                  
     where D i  is the orthogonal distance from the image&#39;s ideal viewing point (IVP) to the image plane in the three dimensional space (x,y,z) of the virtual world containing the vertex Q i . As the viewer moves, each vertex Q i  is then mapped into a new vertex Q i  in accordance with the following equation at step  165 : 
     
       
           Q   i   ′=Q   i   +α   i   {overscore (T)}   (Eq.2) 
       
     
     thereby forming new polygon partitions  145 ′ and  150 ′ within the world coordinates (x,y,z). This is illustrated in FIG.  5 . Also for illustration purposes, shown in FIG. 6 is the orthogonal distance D 3  for vertex Q 3 . Note that α i  varies from 1 for those vertex points located at infinity (D=∞) to 0 for those vertex points located at the viewing image plane D=|{overscore (C)}|, thereby providing stronger coupling for objects farther away. At step  165 , the corresponding images in original polygon partitions  145  and  150  are then texture-mapped onto new polygon partitions  145 ′ and  150 ′, respectively. Similarly, the above described procedure is repeated for each polygon partition (step  170 ). 
     It should be understood that the above polygons partitions have been manipulated in the three-dimensional space (x,y,z) of the virtual world at step  165  such that image points located farther away from view plane  110  move more closely with the movement of the user&#39;s current view point than those points located farther closer to the view plane. In this manner, as the viewer moves away from the image&#39;s ideal viewing point, IVP, the resulting warping in the image acts to limit distortions. 
     FIGS. 7 and 8 compare the display of two-dimensional image  115  for the display of the same image, with and without the use of the polygon partitions of the present invention. More specifically, FIG. 7 depicts viewing two-dimensional image  115  at a location from the left of the image&#39;s ideal viewing point, IVP, without the use of the present invention. In this latter figure, note the distortions in the image. FIG. 8 depicts the same two-dimensional image manipulated in accordance with the principles of the invention. Note, however, that in this latter figure, the distortions has been substantially eliminated. 
     It should be clearly understood that objects within the two-dimensional image may be partitioned into more than two polygon partitions, if. For example, instead of treating the sides of the building as two surfaces, multiple polygon partitions may be employed, each representing a different portion of the building in the three-dimensional space of world  105 , such as doors, walls and windows. In such a case, each polygon partition is manipulated likewise in the same manner as described herein above. Partitioning objects in the two-dimensional image into a greater number of smaller polygon partitions further reduces distortions. It is contemplated that the number of polygon partitions that each object is divided into may be chosen empirically based on image content as well as the user&#39;s range of movement within the virtual world, and the processing power of the computer system used. 
     The foregoing merely illustrates the principles of the invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangement which, although not explicitly describe or shown herein, embody the principles of the invention and are included within its spirit and scope.