A display system has a display device arranged to generate a first view in a first direction relative to the display device and to generate a second view in a second direction relative to the display device, the second direction being different from the first direction, and a device that detects the hand position of at least one user or of a pointing device held by the hand of such user, the position relative to the display device, the user-input being arranged to control the first view and the second view on basis of the detection and an observation mechanism for observation of the, at least one user and arranged to detect whether the user-input is provided to control the first view or the second view, on basis of the observation.

This application claims the priority of European Patent Application 03100596.0, filed on Mar. 10, 2003, in the European Patent Office.

The invention relates to a display system, comprising:a display device being arranged to generate a first view in a first direction relative to the display device and to generate a second view in a second direction relative to the display device, the second direction being different from the first direction; anduser-input means for accepting user-input by means of detecting a position of a hand of an user or of a pointing device held by the hand, the position relative to the display device, the user-input means being arranged to control the first view and the second view on basis of the detection.

The invention further relates to an apparatus comprising such a display system.

The invention further relates to a vehicle comprising such a display system.

An embodiment of the unit of the kind described in the opening paragraph is known from the article “3D Touchless Display Interaction”, by C. van Berkel, in SID Proceedings International Symposium, vol. 33, number 2, pp 1410-1413, 2002. In this article a display system is disclosed comprising a display device and means for detecting hand and finger movements in the neighborhood of the display device. The detection of the hand and finger movements is based on measuring changes of the strength of an electric field which is generated by means of a number of electrodes. An application, which runs on the display system, is controlled on basis of the detected finger and/or hand positions. The application is arranged to generate a sequence of images which are provided to the display device which is designed to create a view on basis of the sequence of images.

The known display system might comprise a 2D display device, but it might also comprise a 3D display device. A 3D display device is arranged to create multiple views, each of which has its own direction relative to the display device. In general, a first one of the views is to be observed by the left eye of an user and a second one of the views is to be observed by the right eye of the user, giving the user an enhanced depth impression.

In the case of an appropriate design, i.e. correct dimensions and disposal of optical elements and light generating elements, it is possible to generate multiple views with one display device, whereby the angle between a first direction of a first view and a second direction of a second view is relatively large. In patent application EP 02078417.9 of 19 Aug. 2002 this is disclosed. Typically the views are not related to the same content then. That means e.g. that the first view corresponds with an image being generated by a first application and that the second view corresponds with another image being generated by a second application. For example the first application corresponds to a movie and the second application corresponds to a car navigation program. Both applications might be running for one single user which is alternatingly watching the first view and the second view. However, it also possible that there are multiple users, each of them watching one of the views being generated by the display device.

If there are multiple applications running and each of them has to be controlled by means of user-input which is accepted and processed by the means for detecting hand and finger movements then a problem occurs: for which of the applications is the user input provided?

It is an object of the invention to provide a display system of the kind described in the opening paragraph which is arranged to accept user-inputs for multiple applications corresponding to the first and second view, respectively.

This object of the invention is achieved in that the display system in further comprising observation means for observation of the user and for detecting whether the userinput is provided to control the first view or the second view, on basis of the observation and the first direction. By observation of the user the display system according to the invention is arranged to determine which of the generated views are visible by that user. It is assumed that the user wants to control the application corresponding to the view which is visible by that user. Optionally the display system according to the invention is arranged to observe multiple users. E.g. on basis of detection of the presence of the hand of the user being located left from the display device the display system is arranged to control the view in the left direction. Alternatively on basis of detection of the presence of the hand of the user being located right from the display device the display system is arranged to control the view in the right direction.

In an embodiment of the display system according to the invention, the observation means are arranged to determine an orientation of the hand of the user relative to the display device and to determine whether a first angle between the orientation of the hand and the first direction is less than a first predetermined threshold. The orientation of the hand is in general related to the position of the user relative to the display device and thus related to the direction of one of the views of the display device. Determination of the orientation of the hand relative to the display device is relatively easy and robust. See for example the article “Driving a 3D Articulated Hand Model in Real-Time using video Input”, by J. Segen and S. Kumar, in proceedings of Tenth Image and Multidimensional Digital Signal Processing Workshop, Alpbach, Austria, July 1998.

Optionally the first predetermined threshold is equal to another angle which is determined between the orientation of the hand and the second direction. In other words, the orientation of the hand is compared with the first direction corresponding to the first view and the second direction corresponding to the second view. The minimum difference corresponds with the best matching view.

Preferably the observation means are also arranged to determine an orientation of the wrist and/or lower arm of the user relative to the display device and to determine whether a second angle between the orientation of the wrist and/or lower arm and the first direction is less than a second predetermined threshold. By taking into account these parts of the arm, a relatively robust method of estimating the position of the user is achieved. It should be noted that in typical situations a first user will be located left from the display device and a second user will be located right from the display device. If a lower arm is detected left from the display device then most probably this arm belongs to the first user and if a lower arm is detected right from the display device then most probably this arm belongs to the right user. In the former case, control of the application corresponding to the left view will be performed and in the latter case, control of the application corresponding to the right view will be performed.

Optionally the second predetermined threshold is equal to another angle which is determined between the orientation of the wrist and/or lower arm of the user relative to the display device and the second direction. In other words, the orientation of the wrist and/or lower arm of the user relative to the display device is compared with the first direction corresponding to the first view and the second direction corresponding to the second view. The minimum difference corresponds with the best matching view.

In an embodiment of the display system according to the invention, the observation means comprise computing means to compute a trajectory of the hand on basis of a number of detected positions of the hand and to determine whether a third angle between a vector which represents the trajectory of the hand, and the first direction is less than a third predetermined threshold. By tracking the position of the hand a good estimation of the position of the user is achieved. This is based on the assumption that an user moves his hand in the direction to the display device in order to control the display system.

Optionally the third predetermined threshold is equal to another angle which is determined between the vector which represents the trajectory of the hand and the second direction. In other words, the vector which represents the trajectory of the hand is compared with the first direction corresponding to the first view and the second direction corresponding to the second view. The minimum difference corresponds with the best matching view.

In an embodiment of the display system according to the invention, the observation means are arranged to determine the position of the eyes of the user relative to the display device and to determine whether a fourth angle between a direction of a vector from the display device to a first one of the eyes and the first direction is less than a fourth predetermined threshold. This embodiment according to the invention is particularly interesting for appliance in a hand-held device. E.g. by imaging the environment of the handheld device, the position of the user relative to the display device is determined. On basis of this relative position, the direction of the vector from the display device to the first one of the eyes is computed.

Optionally the fourth predetermined threshold is equal to another angle which is determined between the vector from the display device to a first one of the eyes and the second direction. In other words, the vector from the display device to a first one of the eyes is compared with the first direction corresponding to the first view and the second direction corresponding to the second view. The minimum difference corresponds with the best matching view.

By detecting both the position of the eyes of the user and of parts of his arm a relative accurate and robust distinction between user-inputs is achieved.

Optionally some of the above mentioned predetermined thresholds are mutually equal.

In an embodiment of the display system according to the invention, the observation means comprise optical means for imaging the user. Optical means might be one or more cameras and computing means for image analysis.

In an embodiment of the display system according to the invention, the observation means are arranged to observe the user on basis of acoustic, ultraviolet or radio frequency waves. Preferably these type of observation means are located adjacent to the display device. By means of sending and receiving e.g. ultra-sound the presence of a hand or lower arm in the neighborhood of the display device can relatively easily be detected. With other types of waves, e.g. electromagnetic or ultra-violet waves a similar type of detection can be performed. An advantage of these type of observation means is that they are relatively cheap. Another advantage of these type of observation means is that they are relatively robust.

In an embodiment of the display system according to the invention, the observation means are arranged to observe the user on basis of measuring temperature differences. By means of infra-red sensors the presence of a hand in the neighborhood of the display device can relatively easily be detected. Especially, when multiple infra-red sensors are applied, movement of a hand relative to these sensors can easily be determined, since the local temperature will most probably change because of the movement of the hand in the vicinity of the sensors.

In an embodiment of the display system according to the invention the userinput means comprise the observation means. Alternatively the observation means comprise the user-input means. This means that two functions are performed by the same means. The first function is the detection of the position of the hand relative to the display device in order to control the first view or the second view. The second function is the observation of the user to determine whether the user-input is provided to control the first view or the second view. An advantage of this embodiment according to the invention is a reduction of costs.

It is a further object of the invention to provide an apparatus of the kind described in the opening paragraph, comprising a display system which is arranged to accept user-inputs for multiple applications corresponding to the first and second view, respectively.

This object of the invention is achieved in that the display system further comprises observation means for observation of the user and for detecting whether the userinput is provided to control the first view or the second view, on basis of the observation and the first direction.

The apparatus might be a PDA (personal digital assistant) providing functions to the user like a calendar, calculator, text processor, games, address-book, etcetera. The apparatus might also be a mobile phone, electronic book or laptop.

It is a further object of the invention to provide a vehicle of the kind described in the opening paragraph, comprising a display system which is arranged to accept userinputs for multiple applications corresponding to the first and second view, respectively.

This object of the invention is achieved in that the display system further comprises observation means for observation of the user and for detecting whether the userinput is provided to control the first view or the second view, on basis of the observation and the first direction.

The vehicle might be a car, bus, truck, airplane, train or boat. Typically, two or even more passengers (including the driver or pilot) are seated adjacent to each other in these vehicles. In general, the amount of space for positioning display systems is limited in these vehicles. That means that sharing a display system by multiple passengers is advantageous in these cases. In order to provide each of the passengers with its own image content, a display systems which can generate multiple views is a good solution. The appliance of touchscreens or similar devices as user-input means is advantages in vehicles because of the limited space. Hence, the display system according to the invention is advantageous in vehicles.

Modifications of the display system and variations thereof may correspond to modifications and variations thereof of the apparatus and vehicle described.

Same reference numerals are used to denote similar parts throughout the figures.

FIG. 1schematically shows an embodiment of the display system100according to the invention and two users116and118. The display system100comprises:a display device102being arranged to generate a first view112in a first direction120relative to the display device120and to generate a second view114in a second direction122relative to the display device102, the second direction122being different from the first direction120;user-input means104for accepting user-input by means of detecting a position of a hand110of an user or of a pointing device108held by the band, the position relative to the display device102. The user-input means104are arranged to control the first view112and the second view114on basis of the detection of the position of the hand110of or of the pointing device108; andobservation means106for observation of the users116and118. The observation means106are arranged to detect whether the user-input is provided to control the first112view or the second view114.

The display device102comprises one or more luminance modulation units to visualize respective sequences of images being provided by means of multiple image sources. In the case of a single luminance modulation unit temporal or spatial multiplexing is applied to render the images of the respective sequences. Typical image source are a DVD player, a receiver for receiving broadcast images, a set-top box, a satellite-tuner, a VCR player or a computer being arranged to render graphical images. The luminance modulation units can be based on know display technologies like CRT (Cathode Ray Tube), LCD (Liquid Crystal Display) or PDF (Plasma display panel). The display device102further comprises optical means to direct a first sequence of images into a first direction120resulting in a first view and to direct a second sequence of images into a second direction122. The first of the views can be seen by the first user116and the second view can be seen by the second user.

The user input-means104might be a so-called touch-screen. Typically a touch-screen is based on measuring local changes in electrical capacitance or resistance. It comprises two electrodes being disposed parallel with the display device102. By pushing on one of the electrodes the distance between the electrodes is decreased locally, resulting in a change of the local electrical capacitance or resistance.

The user input-means104might comprise sensing electrodes being arranged in an active matrix or around the edge of the display device. See the description in connection withFIG. 5for a brief explanation of this technology.

The user input-means104might comprise other types of detection means, e.g. pairs of ultra-violet or infra-red or ultra-sound transmitters and receivers, being disposed at opposite edges of the display device102.

The observation means106is arranged to inspect the users116and118by means of imaging the environment in front of the display device102. The working of these observation means is as follows. It is assumed that a first user116is located left from the display device102and a second user118is located right from the display device102. The first user116is able to watch the first view112and the second user118is able to watch the second view114. At a certain moment in time the first user wants to control the application which corresponds to the first view112. So, the first user moves his hand110towards the display device102to put his finger at or close to the position of the display device102which represents an active user-interface control gadget. In the meantime the hand110of the user is imaged. The orientation123of the hand110is determined by means of image analysis. The determined orientation123of the hand is compared with the first direction120and the second direction122of the display system. If the angle between the determined orientation123of the hand and the first direction120is less small than the angle between the determined orientation123of the hand and second direction122, then it is assumed that the user-input is provided to control the first112view and consequently the appropriate application is controlled. If the angle between the determined orientation123of the hand and the first direction120is bigger than the angle between the determined orientation123of the hand and second direction122, then it is assumed that the user-input is provided to control the second view114and consequently the other application is controlled.

Optionally, the observation means106are also arranged to determine the orientation of the lower-arms of the users116and118.

Optionally, the users116and118make use of pointing devices, e.g.108to provide user-input to the display system100.

FIG. 2schematically shows that by rotating the display device102the user116can select one of the views. In connection withFIG. 1it is described that a display system100is shared by multiple users, e.g. two users. In the case ofFIG. 2, a display system is used by a single user which is alternatingly watching the views112-114being generated by the display device102. The user can “switch” from a first view112which is generated in a first direction120to a second view113which is generated in a second direction121by pivoting the display system. This method of selecting one of the views112-114is particular interesting in the case of a mobile system like a PDA. A first view112might correspond to a calculator, a second view113might correspond to a map and a third view114might correspond to a calendar manager.

FIG. 3Aschematically shows that the position of the user116a-116crelative to the display device102is determined by means of imaging the environment. Suppose there is one user which is using the display device as described in connection withFIG. 2. In order to determine to which of the views112-114the user is watching at a particular moment the display system comprises a camera106which is arranged to image the user in its environment. The camera106comprises a wide-angle lens (or fish-eye lens) to image the scene comprising the user, with an angle of approximately 180°. Determining to which of the views112-114the user is watching means that the angle of a vector220-222from the display device102to one of the eyes116a-116chas to be estimated. Suppose that the user116ais watching view112then the display device is pivoted such that the vector220from the display device102to one of the eyes116asubstantially corresponds to the direction120of the first view. If an image is captured by the camera106at that moment it will look like a first image304. The first image304comprises a representation of the user306and of his hand308. The relative position of the representation of the user306in the first image304is related to the position of the user relative to the display device.

If an image is captured at the moment that the user is watching the second view113it will look like a second image302and if an image is captured at the moment that the user is watching the third view114it will look like a third image300.

To determine the relative position of the representation of the user306in the first image the image display system comprises image analysis means. For detection of faces in images several algorithms exist. Many of these algorithms are based on skin-tone detection. Others are based on detection of particular shapes, e.g. circles and ellipses. Besides that, it is possible to make use of local differences in sharpness in the acquired images. The user116will be closer to the camera than other objects in his environment. Hence, the representation of the user306in the first image will be relatively sharp compared with other objects in the first image.

FIG. 3Bschematically shows a mobile phone320comprising a display system according to the invention as described in connection withFIG. 3A. The mobile phone320comprises a display device102being arranged to generate multiple views and a camera106for imaging the user.

FIG. 4Aschematically shows a display system400according to the invention comprising a number of sensing means402-410being disposed adjacent to the display device. The sensing of the sensing means402-410might be based on acoustic, ultra-violet or radio frequency waves. The sensing means are arranged to detect the presence of an object in their vicinity. If e.g. one of the sensors406which is disposed at the left side of the display device102detects the presence of an object, then the application which controls the view in the corresponding direction is provided with user-input.

Sensing means based on light are commercially available, e.g. at IVO GmbH & Co. Typically these optical sensing means are arranged to measure a distance. These sensing means are so-called proximity detectors. The appliance of these sensing means in the display system according to the invention is as follows. If e.g. a hand is put in the neighborhood of the display device102in order to provide the display system with user input then one or multiple of the sensors detect that there is an object close to the one or multiple sensors. So, each of the sensors measures the distance of objects in their respective neighborhoods. If one of the measured distances is below a predetermined threshold and user-input is accepted by means of the user-input means at the same time then it is easily detected for which of the applications the user input is required.

Sensing means based on ultra-sound are also commercially available.FIG. 4Bschematically shows an apparatus420comprising the display system400according to the invention, comprising a number of sensing means406-410being disposed adjacent to the display device102. The display system400further comprises a (not depicted) touch-sensitive screen being placed in front of the display device102. The apparatus420further comprises a keyboard as user-input means. The apparatus420further comprises a processor on which multiple applications can run simultaneously. A particular application which can be executed by the processor is a game for two players. When playing that game it is relevant that each of the players has its own private view. That means that the first player is not allowed to see the view of the second player and vice versa. An example of such game is “stratego”. The first player is located right from the display device102and the second player is located left from the display device102. The players have to provide userinput by means of touching the touch-sensitive screen. At the moment that the touch-sensitive screen registers user-input both the first array of ultra-sound sensors406which is located right from the display device102and the second array of ultra-sound sensors410which is located left from the display device102perform a distance measurement. If one of the elements of the first array of ultra-sound sensors406measures a first distance between the one of the elements and an object in its measurement field, which is less than other distances being measured by the elements of the second array of ultra-sound sensors410then the userinput is assumed to be provided by the first player and the application is controlled consequently. Similarly, if one of the elements of the second array of ultra-sound sensors410measures a second distance between the one of the elements and an object in its measurement field, which is less than other distances being measured by the elements of the first array of ultra-sound sensors406then the user-input is assumed to be provided by the second player and the application is controlled consequently.

FIG. 4Cschematically shows the inside of a car, with the display system400according to the invention being located in the dash-board of the car.FIG. 4Cshows the view424on the road in front of the car, the steering-wheel422of the driver and a mirror426. The mirror426is optional. Alternatively the display system400provides a first view to the driver of the car which corresponds to images being captured by a camera which is located such that the scene behind the car can be imaged. The display system400comprises sensing means402-410as described in connection withFIGS. 4A and 4B. Besides that, the display system400comprises cameras106a-106bbeing located in the ceiling and/or wall of the car. These cameras106a-106bare designed to observe the environment in front of the display device102. If one of the cameras106a-106bimages an arm412or416the orientation414,418is determined. The determined orientation414or418is compared with the directions of the views of the display device102, as described in connection withFIG. 1. On basis of the measurements of the sensing means402-410and the image analysis of the images captured by the cameras106a-106bthe display system400is arranged to determine which of the persons in the car, i.e. the driver or the passenger being seated next to him, has provided userinput.

FIG. 4Dschematically shows an apparatus comprising the display system400according to the invention which has to be controlled by means of pointing devices108-109.

Instead of using a finger-tip to touch a touch-sensitive screen, in many cases it is advantageous to use a device with a sharper top, i.e. a pointing device108-109. Detection of the presence of a hand110-111which holds such a pointing device108-109can be achieved similar to what has been described above. That means that the display systems according to the invention can be applied in the case that the users have a pointing device in their hand to provide user-input and in the case that they do not make use of such a pointing device.

FIGS. 5A and 5Bschematically show the concept of cross capacitance sensing.FIG. 5Ashows a simple combination of two electrodes502-504forming a capacitor, with a capacitive current flowing between them.FIG. 5Bshows the same two electrodes502-504whereby a hand110is placed near the electrodes502-504. Some of the field lines506-510are terminated on the hand and the current is decreased relative to the current which flows in the case without the hand110being placed near the electrodes, like inFIG. 5A. A measurable effect can be obtained in which the spatial range is roughly equivalent to the separation between the electrodes502-504. Arrays of multiple electrodes offer rich possibilities for object and gesture recognition. For instance, n transmitters and n receivers arranged around the edge of a display contain n2 electrode pairs at many different separations. See for more details the article “3D Touchless Display Interaction”, by C. van Berkel, in SID Proceedings International Symposium, vol. 33, number 2, pp 1410-1413, 2002.

FIG. 6schematically shows an embodiment according to the invention which is arranged to determine a trajectory600of a hand of an user. By localizing the positions110a-110cof a hand as function of time a trajectory600of movement can be computed. The localization of the different positions110a-1110cof the hand might be based on images being captured as described in connection withFIG. 1, distances being measured as described in connection withFIGS. 4A and 4B, or based on differences in electrical fields as described in connection withFIGS. 5A and 5B. The trajectory600of the hand is compared with the first120and second122view of the display device102. The best matching direction corresponds to the view which is controlled as a result of user-input being accepted in a predetermined time interval after the computation of the trajectory600has finished.

FIG. 7schematically shows a display system700according to the invention, comprising:a display device102being arranged to generate a first view in a first direction relative to the display device and to generate a second view in a second direction relative to the display device;user-input means104for accepting user-input by means of detecting a position of a hand of an user or of a pointing device held by the hand, relative to the display device. The user-input means104are arranged to control a first application706corresponding to the first view and to control a second application708corresponding to the second view on basis of the detected position;observation means106for observation of the user and arranged to detect whether the user-input is provided to control the first application706or the second application708, on basis of the observation; andcomputing means for running the first706and the second708application.

The observation means106and the user-input means104are combined. Or in other words, the user-input means104and observation means106share a number of components. That means that the display system700according to the invention, has means20704which are arranged to determine to which part of the display device102is pointed to, at a particular moment and to determine what the relative position is of the person who is pointing to the display device102at that particular time. These means704are e.g. arranged to determine the position of the finger-tip and to estimate the orientation of the hand. This can for example be based on a number of images taken from different directions. This is disclosed in the article “Driving a 3D Articulated Hand Model in Real-Time using video Input”, by J. Segen and S. Kumar, in proceedings of Tenth Image and Multidimensional Digital Signal Processing Workshop, Alpbach, Austria, July 1998. Alternatively, both parameters, i.e. location of the finger-tip and orientation of the hand are determined by means of electromagnetic fields as disclosed in the article “3D Touchless Display Interaction”, by C. van Berkel, in SID Proceedings International Symposium, vol. 33, number 2, pp 1410-1413, 2002. By applying a number of well disposed ultrasound sensors the same result can be achieved. Notice that in various medical equipment, transducers are applied which provide data to reconstruct three dimensional image data sets.