System, apparatus and method of displaying information for foveal vision and peripheral vision

A system, apparatus and method of displaying information for foveal vision and peripheral vision are provided. The system, apparatus and method generate a composite display image fully viewable with foveal vision and peripheral vision. The system, apparatus and method detect, either through gesture or speech recognition, an area on the composite display image a user is focusing on with foveal vision and display the area in a resolution higher than the resolution in which the rest of the composite display image is displayed.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention is directed to information display. More specifically, the present invention is directed to a system, apparatus and method of displaying information for foveal vision and peripheral vision.

2. Description of Related Art

In live group meetings, background information may be placed on walls of a conference room to allow participants to scan the information as needed. After a participant has scanned a piece of background information, however, the participant's focus is ordinarily returned to information relating to a topic in discussion. When a participant's focus is directed to a particular piece of information, that piece of information may be said to be in the participant's foveal vision. All other pieces of information may then be said to be in the participant's peripheral vision.

Foveal vision is high resolution or acuity vision whereas peripheral vision is wide field of vision. Unlike foveal vision, peripheral vision is not of high resolution but nonetheless provides situation awareness and/or information detection. Put in another way, foveal vision may be likened to a magnifier whereas peripheral vision to data compression. It (peripheral vision) allows a biological computer (e.g., a human being), which includes a foveal system and a peripheral system, to determine where to find relevant information. From this, the biological computer calculates in milliseconds all necessary muscle commands for bringing the foveal system into position. This may happen three to four times per second, each time, the compressed data of the peripheral system is automatically recalculated to compensate for eye movements. Thus, using peripheral vision a participant may quickly determine which of the different pieces of information that are placed on the wall of the conference room to focus on when needed.

In e-meetings, information is displayed on a computer display screen. Participants, therefore, are constrained to only one piece of information at a time. Consequently, to consult a piece of background information that is not being presently displayed, a search may have to be undertaken on the computer system to find and bring forth the information. This may consume time detracting the participant from the topic in discussion.

Hence, what is needed is a system, apparatus and method of displaying information in an e-meeting for foveal vision and peripheral vision.

SUMMARY OF THE INVENTION

The present invention provides a system, apparatus and method of displaying information for foveal vision and peripheral vision. The system, apparatus and method generate a composite display image fully viewable with foveal vision and peripheral vision. The system, apparatus and method detect, either through gesture or speech recognition, an area on the composite display image a user is focusing on with foveal vision and display the area in a resolution higher than the resolution in which the rest of the composite display image is displayed. The composite display image may be generated by multiple computer systems. Particularly, one computer system may be used to generate the area of the composite display image in the higher resolution while another computer system may be used to generate the rest of the composite image display in the lower resolution.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention is implemented using Everywhere Display (ED) technology. ED is a product of International Business Machines (IBM) Corp. and allows every surface in a space to be transformed into a projected “touch screen” display. Particularly, ED projects an image onto a surface and uses gesture recognition to determine where on the image a user is pointing. ED is disclosed in a variety of references of which the most applicable ones follow: STEERABLE INTERFACES FOR PERVASIVE COMPUTING SPACES by Pingali et al., IBM T.J. Watson Research Center, March 2003; EMBEDDING INTERACTIONS IN A RETAIL STORE ENVIRONMENT: THE DESIGN AND LESSONS LEARNED by Sukaviriya et al., IBM T.J. Watson Research Center; TRANSFORMING SURFACES INTO TOUCH-SCREENS by Pinhanez et al., IBM T.J. Watson Research Center, December 2001; DYNAMICALLY RECONFIGURABLE VISION-BASED USER INTERFACES by Kjeldsen et al., IBM T.J. Watson Research Center; and INTERACTING WITH STEERABLE PROJECTED DISPLAYS by Kjeldsen et al., IBM T.J. Watson Research Center, May 2002. The references are provided in an Information Disclosure Statement filed concurrently with the present invention. The disclosure of each one of the references is herein incorporated by reference.

FIG. 1depicts an exemplary ED system. The system includes a computer102, a liquid crystal display (LCD) projector120, a pan/tilt mirror130and a video camera with a pan/tilt head140. The computer102has a first controller board (controller board1104) to control the pan/tilt head of the video camera140, a second controller board (controller board2110) to control the LCD projector120and a third controller board (controller board3112) to control the pan/tilt mirror130. The computer102further contains a video capture board106connected to the video camera140and a graphics board108connected to the LCD projector120.

The pan/tilt mirror130is used to deflect images from the LCD projector120to surfaces (e.g., walls, floor, etc.) of a room. The projected image may be processed to compensate for perspective distortions. The pan/tilt video camera140is used to detect hand/body activity on the projected area. This allows a user to interact with the system by simply touching the projected image.

As stated in the references, ED obviates the need to wire surfaces to use with sensing devices, use monitors or other sensing devices as well as to have users wear graphics goggles or special input devices. Further the ED projector can be easily steered to a new surface by a simple software command, thus avoiding fixed setups where projectors or cameras can only perform one specific task. The pan/tilt mechanisms provide dynamic adaptation to changes in the surface being used and to different patterns of user activity.

The present invention uses two interconnected ED systems as shown inFIG. 2. One ED system (e.g., system200) may be placed above or behind a user to project one high resolution image on a surface directly in front of the user and within easy gesture reach. The other ED system (e.g., system250) may be placed at a location to project a composite of multiple images in moderate resolution on surfaces near the user. The projected image from ED system200may be used to provide information about the topic in discussion and the images from ED system250may provide background information. Thus, ED system200may project an image that may be likened to foveal vision and ED system250may project images akin to peripheral vision. Consequently, the two interconnected ED systems will provide images comparable to a wall of charts that is visible by both foveal and peripheral visions.

FIG. 3depicts an exemplary wall of charts display using the present invention. Each area (i.e., areas1–10) in the wall of charts may display background information in moderate resolution for peripheral vision whereas area11may display information in high resolution for forveal vision. Thus, areas1–10may be displayed using background (or peripheral) projector310(i.e., the projector of system250) and area11may be displayed using forveal projector320(i.e., the projector of system200).

When the user desires to have a closer inspection of one of the background images (i.e., one of the images from system250), the user may select the image using gesture recognition or speech recognition. When that occurs, an exchange of images will occur. That is, the selected background image will then be projected by system200in high resolution in area11and the original image from system200will be displayed by system250in moderate resolution in the area where the background image was displayed. Thus, any image displayed by system250in moderate resolution may be displayed by system200in high resolution when selected as mentioned above. Note that to select an image by speech recognition, the user may have to indicate the position of the image (e.g., third image from the left, right, top, bottom etc.) or its title.

Any one or all the images being projected by either or both systems200and250may come from a remote location. Particularly, since e-meetings involve participants who are geographically separated from each other, any number of the images may be from a remote participant or participants. Further, the images may be from a remote ED system or a remote video-conferencing system. In addition, any image that is being displayed in high resolution by a remote ED system or a remote video-conferencing system may be displayed by a local ED system of the present invention.

FIG. 4depicts a pictorial representation of a network of data processing systems that may be used to provide the images to the present invention. Network data processing system400contains a network402, which is the medium used to provide communications links between various devices and computers connected together within network data processing system400. Network402may include connections, such as wire, wireless communication links, or fiber optic cables.

In the depicted example, server404is connected to network402along with storage unit406. In addition, clients408,410, and412are connected to network402. These clients408,410, and412may be, for example, personal computers or network computers. In the depicted example, server404provides data, such as boot files, operating system images, and applications to clients408,410and412. Clients408,410and412are clients to server404. Network data processing system400may include additional servers, clients, and other devices not shown. In the depicted example, network data processing system400is the Internet with network402representing a worldwide collection of networks and gateways that use the TCP/IP suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, government, educational and other computer systems that route data and messages. Of course, network data processing system400also may be implemented as a number of different types of networks, such as for example, an intranet, a local area network (LAN), or a wide area network (WAN).FIG. 4is intended as an example, and not as an architectural limitation for the present invention.

Referring toFIG. 5, a block diagram of a data processing system that may be implemented as a server, such as server404inFIG. 4, is depicted in accordance with a preferred embodiment of the present invention. Data processing system500may be a symmetric multiprocessor (SMP) system including a plurality of processors502and504connected to system bus506. Alternatively, a single processor system may be employed. Also connected to system bus506is memory controller/cache508, which provides an interface to local memory509. I/O bus bridge510is connected to system bus506and provides an interface to I/O bus512. Memory controller/cache508and I/O bus bridge510may be integrated as depicted.

Peripheral component interconnect (PCI) bus bridge514connected to I/O bus512provides an interface to PCI local bus516. A number of modems may be connected to PCI local bus516. Typical PCI bus implementations will support four PCI expansion slots or add-in connectors. Communications links to network computers408,410and412inFIG. 4may be provided through modem518and network adapter520connected to PCI local bus516through add-in boards.

Additional PCI bus bridges522and524provide interfaces for additional PCI local buses526and528, from which additional modems or network adapters may be supported. In this manner, data processing system500allows connections to multiple network computers. A memory-mapped graphics adapter530and hard disk532may also be connected to I/O bus512as depicted, either directly or indirectly.

With reference now toFIG. 6, a block diagram illustrating a data processing system is depicted in which the present invention may be implemented. Data processing system600is an example of a client computer. Data processing system600employs a peripheral component interconnect (PCI) local bus architecture. Although the depicted example employs a PCI bus, other bus architectures such as Accelerated Graphics Port (AGP) and Industry Standard Architecture (ISA) may be used. Processor602and main memory604are connected to PCI local bus606through PCI bridge608. PCI bridge608also may include an integrated memory controller and cache memory for processor602. Additional connections to PCI local bus606may be made through direct component interconnection or through add-in boards. In the depicted example, local area network (LAN) adapter610, SCSI host bus adapter612, and expansion bus interface614are connected to PCI local bus606by direct component connection. In contrast, audio adapter616, graphics adapter618, and audio/video adapter619are connected to PCI local bus606by add-in boards inserted into expansion slots. Expansion bus interface614provides a connection for a keyboard and mouse adapter620, modem622, and additional memory624. Small computer system interface (SCSI) host bus adapter612provides a connection for hard disk drive626, tape drive628, and CD-ROM/DVD drive630. Typical PCI local bus implementations will support three or four PCI expansion slots or add-in connectors.

An operating system runs on processor602and is used to coordinate and provide control of various components within data processing system600inFIG. 6. The operating system may be a commercially available operating system, such as Windows XP™, which is available from Microsoft Corporation. An object oriented programming system such as Java may run in conjunction with the operating system and provide calls to the operating system from Java programs or applications executing on data processing system600. “Java” is a trademark of Sun Microsystems, Inc. Instructions for the operating system, the object-oriented operating system, and applications or programs are located on storage devices, such as hard disk drive626, and may be loaded into main memory604for execution by processor602.

As another example, data processing system600may be a stand-alone system configured to be bootable without relying on some type of network communication interface, whether or not data processing system600comprises some type of network communication interface. As a further example, data processing system600may be a Personal Digital Assistant (PDA) device, which is configured with ROM and/or flash ROM in order to provide non-volatile memory for storing operating system files and/or user-generated data.

The depicted example inFIG. 6and above-described examples are not meant to imply architectural limitations. For example, data processing system600may also be a notebook computer or hand held computer in addition to taking the form of a PDA. Data processing system600also may be a kiosk or a Web appliance.

In any case, any one of clients408,410and412may represent the ED system ofFIG. 2and the present invention may reside on any data storage medium (i.e., floppy disk, compact disk, hard disk, ROM, RAM, etc.) used by a computer system.

FIG. 7depicts a flowchart of a process that may be used by the present invention. The process starts when a user instantiates the ED system ofFIG. 2(step700). Then a check is continuously being made to determine whether the user has selected a moderately displayed image. If so, the image is displayed in high resolution. Note that this may occur by having the image displayed by system200or by having the image be displayed by system250in high resolution (steps710and712). The process ends when the user terminates the session.

The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. For example, only one ED system may be used. In that case, when the user selects a background image for inspection, the ED system may display the image at a higher resolution. Thus, many modifications and variations will be apparent to those of ordinary skill in the art. Thus, the embodiment was chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.