Abstract:
A method for increasing multimedia data accessibility is presented and more specifically increasing the interactivity with images shown on a display device. A series of images is displayed according to a user&#39;s position relative to a display screen by displaying a first image, receiving information regarding a change in the user&#39;s position relative to the display screen, and displaying a second image in response to the change in the user&#39;s position. The series of images are packaged, compressed and transmitted in a manner that makes it feasible to simultaneously receive, display and interact with the series of images. The scale of a portion of the images is manipulated based on interaction therewith.

Description:
FIELD OF THE INVENTION  
         [0001]    The present invention relates to a method for increasing multimedia data accessibility and more specifically to increasing the interactivity with images shown on a display device.  
         BACKGROUND OF THE INVENTION  
         [0002]    While there are a number of computer-based media types that provide interactive 3D representations of objects or scenes, none of these media types provides intuitive interaction while being inexpensively produced and lending themselves to Internet content delivery and play back. The Internet&#39;s range of media types is driven largely by transmission bandwidth limitations together with the need for new and more compelling content that make use of new media types.  
           [0003]    Further, the method by which these media types are viewed and interacted with restricts the users to able-bodied people. When viewing an image, visually impaired people can only view the image when using magnification software supplied by a third party. The available magnification software is cumbersome to use when trying to take advantage of multimedia data, especially images. Often it is desired to magnify only the image; however, third party software does not readily offer this feature. There are two basic types of magnification software: entire display magnification which hides from immediate view a large portion of the screen, and mouse centered magnification which provides a fixed screen showing an enlarged version of the area around the mouse. These systems provide only a partial solution as the available magnification software does not provide quick access to the program nor is it flexible enough to use alternative input devices.  
           [0004]    The interactive 3D media types available that are compatible with Internet resource limitations (i.e. QuicktimeVR by Apple Corp. and 360 by IPIX Corp.) require labour intensive production or special capture equipment. This increases the cost of producing these interactive 3D media types. Further, user navigation of these 3D media types uses a mouse, which is not an intuitive navigation tool, especially when viewing images of a scene. Additionally, these media types require media transmission to be completed before viewing or interacting with the media. In cases where the size of the media is large, this creates a long time delay before the images can be viewed.  
           [0005]    Existing immersive virtual reality technologies provide for 3D interactivity but require special head gear or viewing apparatus, or special cameras. In addition, these methods do not provide for low bandwidth transmission and low latency response to user input.  
           [0006]    U.S. Pat. No. 5,574,836, titled “Interactive Display Apparatus and method with Viewer Position Compensation”, discloses a system that provides an intuitive interactive environment for users. However, this system is based on changing the position of an object to be displayed according to the position of the user. As this system is concerned with changing the position of an object, it does not provide a method for intuitive interaction in a 3D environment or viewing a 3D object from a plurality of viewing angles.  
         SUMMARY OF INVENTION  
         [0007]    Accordingly, it is an object of the invention to provide a method for intuitively viewing an image or series of images (either multiple views of an object or different images of unrelated objects).  
           [0008]    It is a further object of the invention to provide a method which provides intuitive navigation within a 3D environment.  
           [0009]    It is another object of the invention to provide a method which provides intuitive navigation of a 3D image without requiring the position of the viewed object to change.  
           [0010]    It is an additional object of the invention to provide a method for transmitting and viewing a series of related images that conforms to Internet resource limitations.  
           [0011]    It is a further object of the invention to provide a method for viewing and interacting with images from a series of images prior to receiving every image.  
           [0012]    It is a further object of the invention to provide an easily accessible, integrated screen display magnification tool.  
           [0013]    In accordance one object of the present invention there is provided a method of displaying a series of images according to a user&#39;s position relative to a display screen comprising the steps of: displaying a first image from the series of images, receiving information regarding a change in the user&#39;s position relative to the display screen, and displaying a second image from the series of images in response to the change in the user&#39;s position.  
           [0014]    In accordance with another object of the invention there is provided a method of simultaneously receiving, displaying and interacting with a series of images in response to movement of an interactive device comprising the steps of: (a) receiving for display a first image from the series of images, (b) receiving for display subsequent images from the series of images, and (c) permitting viewing of and interacting with the first image while performing step (b) where interaction with the first image is in response to signals from the interactive device.  
           [0015]    In accordance with a further object of the invention there is provided a method of transmitting a series of images to increase the fidelity of transmission comprising the steps of: compressing a selected first image from the series of images separately compressing each image from the series of images, and progressively transmitting each compressed image from the series of images in a manner that first transmits the selected first compressed image and then transmits each image from the series of images such that the further each image is from the selected first compressed image the later it is transmitted.  
           [0016]    In accordance with another object of the invention there is provided a method of increasing the scale of a portion of a displayed object comprising the steps of: providing a perpetual foreground icon for accessing a magnification tool, accessing the magnification tool, tracking movement of an interactive device, displaying a specified area as an enlarged area, and changing an enlargement power in response to the interactive device while tracking movement and displaying the specified area.  
           [0017]    In accordance with yet another object of the invention there is provided a computer readable medium having stored thereon computer-executable instructions for displaying a series of images according to a user&#39;s position relative to a display screen comprising the steps of: displaying a first image from the series of images, receiving information regarding a change in the user&#39;s position relative to the display screen, and displaying a second image from the series of images in response to the change in the user&#39;s position.  
           [0018]    In accordance with a further object of the present invention there is provided a computer readable medium having stored thereon computer-executable instructions for simultaneously receiving, displaying and interacting with a series of images in response to movement of an interactive device comprising the steps of: (a) receiving for display a first image from the series of images. (b) receiving for display subsequent images from the series of images, and (c) permitting viewing of and interacting with the first image while performing step (b) where interaction with the first image is in response to signals from the interactive device.  
           [0019]    In accordance with another object of the invention there is provided a computer readable medium having stored thereon computerexecutable instructions for transmitting a series of images to increase the fidelity of transmission comprising the steps of: compressing a selected first image from the series of images, separately compressing each image from the series of images, and progressively transmitting each compressed image from the series of images in a manner that first transmits the selected first compressed image and then transmits each image from the series of images such that the further each image is from the selected first compressed image the later it is transmitted.  
           [0020]    In accordance with another object of the present invention there is provided a computer readable medium having stored thereon computer-executable instructions for increasing the scale of a portion of a displayed object comprising the steps of: providing a perpetual foreground icon for accessing a magnification tool, accessing the magnification tool, tracking movement of an interactive device, displaying a specified area as an enlarged area, and changing an enlargement power in response to the interactive device while tracking movement and displaying the specified area. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0021]    The present invention will be described in greater detail below, including specific examples thereof, with regard to the figures, in which:  
         [0022]    [0022]FIG. 1A is an illustration of a prior art system used to capture images at various view angles to comprise the image set used during playback,  
         [0023]    [0023]FIG. 1B is an illustration of an embodiment of the present invention in which the capture system of FIG. 1A is shown in connection with various display computers,  
         [0024]    [0024]FIG. 2 is a prior art illustration of resulting images from the corresponding image capture view angles,  
         [0025]    [0025]FIG. 3 is an illustration of a recording system used for automated image capture according to an embodiment of the present invention,  
         [0026]    [0026]FIG. 4 is an illustration of compression and transmission sequence used during transmission of images according to an embodiment of the present invention,  
         [0027]    [0027]FIG. 5 is an illustration of a computer system that is used to play back the images according to an embodiment of the present invention,  
         [0028]    [0028]FIG. 6 is a top view illustration of the various head angles that result in display of corresponding images from FIG. 2 according to an embodiment of the present invention, and  
         [0029]    [0029]FIG. 7 is a screen view of a screen magnification tool. 
     
    
     DETAILED DESCRIPTION OF INVENTION  
       [0030]    [0030]FIG. 1A is an image capture system  11  according to the prior art wherein 2-Dimensional images are used to create a simulated, interactive 3-Dimensional environment. The image capture system  11  includes a camera  12  and an object  10  showing various image capture rotation angles  1 ,  2 ,  3 ,  4 ,  5 ,  6 ,  7  that comprise an image set used during playback and a computer  14  used for image formatting. Individual capture rotation angles are shown: a reference angle  1 , rightward rotation angles  3 ,  5 , and  7  and leftward rotation angles  2 ,  4 , and  6 .  
         [0031]    A system  13  according to an embodiment of the present invention shown in FIG. 1B is composed of three segments: image capture  11 , image transmission, and image display. The computer  14  functions for packaging the series of images from the image capture system  11  for efficient image transmission. Transmission links  16  to the computer  14  provide electrical communication to other computers  18  that are equipped for image display.  
         [0032]    [0032]FIG. 2 is a sample illustration of resulting individual images  20  from the corresponding image capture rotation angles  1 ,  2 ,  3 ,  4 ,  5 ,  6 , and  7  shown in the image capture system  11  of FIG. 1A. A reference image  21  corresponds to the reference angle  1  shown in FIG. 1A. Leftward rotation images  23 ,  25 , and  27  correspond to leftward rotation angles  3 ,  5 , and  7  respectively. Similarly, rightward rotation images  22 ,  24 , and  26  correspond to rightward rotation angles  2 ,  4 , and  6  respectively. Although only left and right rotation image capture sequences are shown, this method may be extended to account for camera/object rotation angles that displace in the up and down axes as well, generating a mosaic of images.  
         [0033]    The camera  12  shown in FIGS. 1A and 1B can be moved about the object  10  or the object  10  can be rotated before the camera  12  generating various views  1  to  7  and thus, the corresponding images  20  are captured. The captured images  20  are stored in computer  14 .  
         [0034]    The image series can also be a dynamic series in which movement occurs. The camera  12 / object  10  spatial relationship may be changed manually or by machine control. FIG. 3 is an illustration of a recording system  37  used for automated image capture. A camera  33  is mounted to a servo-controlled pan and tilt gimble  34 . The servo-controlled pan and tilt gimble  34  is controlled by a computer  36  through an electrical communication media  35  using motion object tracking algorithms which are known in the art. By processing a video signal from the camera  33  delivered to the computer  36  through the electrical communication media  35 , the computer  36 -controls the servo-controlled pan and tilt gimble  34  again through the electrical communication  35  such that the camera  33  maintains line-of-sight  31  with a moving object  30  having a trajectory  32  In this manner, a sequence of images is captured such as those shown in FIG. 2.  
         [0035]    There are several methods available as well that use conventional video cameras, or digital still cameras. For example, a hand-held video camera may be pointed at the subject or scene that is to be captured and processed into a 3D computer image. Once recorded, the images may be edited on a computer as required and then formatted for “playback” by the holographic image user. These images can be edited in the computer  14  (of FIG. 1B) to form a continuous series of images that represent a complete field of view.  
         [0036]    Image Transmission  
         [0037]    A series of images (i.e. as shown in FIGS. 2 and 4) is first compressed and then each image is transmitted according to its positional relationship with a reference image (i.e. image  21  in FIG. 2 and image  41  in FIG. 4). According to an embodiment of the present invention, the computer  14  connected to the capture camera  12  may be used for packing the series of images for efficient transmission. As shown in FIG. 1B, the transmission links  16  may optionally provide electrical communication to other computers  18  that are equipped for image display.  
         [0038]    As shown in FIG. 4, a sequence of images  40  is captured either manually or automatically. After editing, the images  40  are formatted and compressed. The preferred method of compression is carried out such that one image is first selected for compression—typically the reference image  41  in the sequence of images  40 . Because of the strong content correlation between the reference image  41  and outlying rotation image  48  and  49 , motion video compression algorithms as are known in the art can be employed on the outlying rotation images  48  and  49  to reduce the data size for improved transmission efficiency. Therefore, a motion compression algorithm is applied separately on outlying leftward rotation image  49  and outlying rightward rotation image  48  of sequence  40  lying on either sided of the reference image  41  where the reference image is the first image to be displayed as data is received for presentation.  
         [0039]    A greater amount of data must be transmitted to an image display (i.e. the computers  18  as shown in FIG. 1 B) for a 3D presentation of a sequence of images  40  than that of a single image as the 3D presentation uses several views of an object. Increased data transmission potentially delays the point in time at which the user may view the holographic or 3D nature of the image data. In order to overcome this, formatting and compression are done such that a progressive approach is used in which a portion of the data may be viewed and interacted with before the entire set of data is received. In order to reduce delay, to provide the user with useful but not necessarily complete information sooner, the following order is used:  
         [0040]    1) Given that the uncompressed and unformatted representation (raw data) of the holographic image of an object is a series of images  40  of that object, each successive image is captured at a subsequently larger view angle than the first capture image. For purposes of applying known interframe motion compression techniques and for initial viewing of the holographic object image, the reference image  41  (or center image) becomes the reference for the outlying images. This is the first image transmitted.  
         [0041]    2) Given that the sequence of images  48  and  49  lying on either side of the reference image  41  exhibit very strong frame-to-frame correlation, interframe motion compression is applied separately to each of the outlying leftward rotation image  49  and outlying rightward image  48  to achieve superior compression ratios.  
         [0042]    3) The resulting compressed images from the outlying rightward rotation images  48  and the outlying leftward rotation images  49  are transmitted progressively in an order that allows compressed images further away from the reference image  41  to be transmitted later. Therefore, the compressed images are ordered alternating left and right side of the reference image  41  as follows: image  43 , image  42 , image  45 , image  44 , image  47 , and image  46 .  
         [0043]    The further away an image is from the reference image  41 , the later it is transmitted. In this manner, early presentation of an object image is viewable during the presentation phase while media data is still being transmitted; therefore, 3D image fidelity increases over time. In general, data that increases pixel or spatial resolution is transmitted progressively later.  
         [0044]    Image Display  
         [0045]    After the media is captured, edited, formatted, compressed, and transmitted, an image series display method of the system of the present invention is applied. Referring now to an embodiment in FIG. 5, which is an illustration of a computer system  57  that is used to play back the images. The computer system  57  includes a computer display  50 , a computer camera  51 , a keyboard  54  and a mouse  55  (or other pointing device), a joystick  58 , and a computer  53  that are electrically connected to each other using standard and known interconnection protocols. As shown in FIG. 6, the computer camera  51  is mounted on or near the computer display  50  such that the user&#39;s head  60  is within the field of view of the computer camera  51  while a user&#39;s head  60  is positioned to have the computer display  50  within convenient viewing range.  
         [0046]    The computer camera  51  is connected to and operated by the computer  53  such that the video signal from the camera  51  is received by the computer  53  for processing. The computer  53  processes the video signal to implement real-time head tracking such that the user&#39;s head  60  is actively tracked to determine it&#39;s relative spatial relationship to the computer display  50 . A method of head tracking is discussed in detail in Applicant&#39;s U.S. Pat. No. 5,574,836 titled “Interactive Display Apparatus and Method with Viewer Position Compensation” herein incorporated by reference.  
         [0047]    Optionally, if the images to be presented are received from a remote location as in the case of a web server (not shown), then a transmission link  56  (such as a modem, Digital Subscriber Line (DSL), cable connection) is connected to the computer  53  such that the compressed and formatted images may be received for presentation.  
         [0048]    Next, the presentation method according to an embodiment of the present invention will be detailed. A formatted and compressed image sequence  20  such as those shown in FIG. 2 is received via transmission link  56  and now resident in part or in entirety on the computer  53 . The first image to be received is the reference image  21  as shown in FIG. 2. The reference image  21  may be the only image transmitted or it may be the first image of a series of images. If there are no other images available for display, then this image will be selected for display in a view window  52  (see FIG. 5). If only the reference image  21  is sent then the with the image sequence  20  is initiated. The view window  52  is that area of the computer display  50  in which one of the images in the image sequence  20  is being displayed.  
         [0049]    There are several degrees of movement under which the user&#39;s head  60  can be followed. Relative to the computer display  50 , these include translation from left to right, translation in and out, head roll from left to right, and head tilt up and down. For the purpose of illustration, translation from left to right is now presented. This is not intended to represent the preferred degree of freedom as all degrees contribute to the modeling of a 3D physical representation.  
         [0050]    As the user&#39;s head  60  moves from side to side changing rotation angles  61  through  67  as denoted in FIG. 6, a corresponding image relating to the current rotation angle is shown in the view window  52 . Assuming now that the user&#39;s head  60  is at rotation angle  61 , the reference image  21  is shown in view window  52 . Accordingly, when the user&#39;s head  60  position is moved to rotation angle  63 , image  23  is displayed. Likewise, for rotation angle  65  results in image  25  displayed, rotation angle  67  results in image  27  displayed, rotation angle  62  results in image  22  displayed, rotation angle  63  results in image  24  displayed, and rotation angle  64  results in image  26  displayed. The actual number of images and the rotation angles used in this example are not intended to be limiting as these values may vary as required for a specific implementation.  
         [0051]    In the event that image data transmission is in progress while image data is displayed, then the method presented here will allow interaction with the image sequence  20  prior to receiving individual images. If the preferred image for display is not available then a substitute will be provided until that image is available. For example, if the user&#39;s head  60  is now at rotation angle  67  but the corresponding image  27  is not yet transmitted, then the nearest neighbor image will be displayed in the display window  52  in the following order of preference, image  25 , image  23 , and image  21 . Accordingly, this method applies to right side rotation angle images  22 ,  24 , and  26 . This method may be extended to other degrees of freedom and view angles as is reflected in a physical modeling of the capture process.  
         [0052]    Although the aforementioned embodiment uses head motion to interact with the images, this is not the only possible method of interaction; other body parts may also be tracked to navigate through an image series. Alternatively, more traditional devices such as the mouse  55  or the joystick  58  may be used (see FIG. 5).  
         [0053]    Additional attributes that depart from real world simulated viewing are also added to aid in viewing. Examples of these are continuous object rotation, and amplified scaling of an image.  
         [0054]    For the case in which only a single image is available for presentation, the distance between the user&#39;s head  60  and computer&#39;s display  50  as determined by the head tracking software is used to scale the image size. Other head movements such as tilt, roll, and displacement which are normally used to simulate change in view angles are not used in their normal function. These head movements are used to change the position of the image being viewed.  
         [0055]    Screen Magnification  
         [0056]    [0056]FIG. 7 shows a computer display  50  wherein a screen magnification tool  70  is provided according to an embodiment of the present invention. The screen magnification tool  70  can be quickly accessed through highlighting of an icon  71  on the screen which remains in the foreground of the display  50 . Accessing the magnification tool  70  is made simpler by requiring only a single highlight of the icon in distinction from the double “mouse click” that has become the industry standard for executing a program.  
         [0057]    To allow for more intuitive interaction with the magnification tool  70 , magnification power can be revised in the same manner as the tool  70  is used. For example, magnification power of the tool  70  can be increased or decreased through the use of a scroll wheel on a standard mouse. Alternatively, head tracking (as disclosed above) may be used to move the position of magnification tool  70  and also change the magnification power through an action such as forward and backward translation, for example.  
         [0058]    In this manner, the magnification tool  70  acts identically to a physical magnification glass. As a result, users will find interaction with the magnification tool  70  simplified as they may interact with the tool  70  just as they would the well-known physical device.  
         [0059]    Media presentation consists of receiving the image data, decompressing it, loading it into memory and displaying one of the images that compose media, the selection of that image which is made by the user&#39;s head position with respect to the display device. The selection process is done in such a way that the user&#39;s view angle with respect to the display is represented by the camera&#39;s view angle with respect to the object. This selection process results in enhancing the user&#39;s perception of a 3D interactive environment or of viewing a 3D object as opposed to a 2D image of a 3D object.  
         [0060]    For example, as the user moves his head to the left, a image showing more of left side of the object image is shown. If the user looks over the top or up, a image representing the angle of the object is shown (provided multiple left-to-right image sequences are captured at media generation time). If the user moves closer to the display, the image is scaled up representing a closer look of the 3D object. If images providing additional resolution are transmitted, then these are presented rather than scaling the image.