Source: https://patents.google.com/patent/US6075557?oq=6760745
Timestamp: 2018-04-24 15:42:03
Document Index: 411152638

Matched Legal Cases: ['application No. 08', 'Application No. 98302989', 'Application No. 98302989', 'application No. 08', 'application No. 08', 'application No. 08']

US6075557A - Image tracking system and method and observer tracking autostereoscopic display - Google Patents
Image tracking system and method and observer tracking autostereoscopic display Download PDF
US6075557A
US6075557A US09061690 US6169098A US6075557A US 6075557 A US6075557 A US 6075557A US 09061690 US09061690 US 09061690 US 6169098 A US6169098 A US 6169098A US 6075557 A US6075557 A US 6075557A
US09061690
Nicholas Steven Holliman
H04N13/0479—Picture reproducers using observer tracking for tracking rotational head movements in a plane parallel to the screen
H04N13/0443—Picture reproducers using a half transparent mirror or prism
An image tracking system which is configured to present a sequence of images, determine the position of a target image in a previously presented image, determine movement of the target image between the previously presented image and a subsequently presented image, and indicate the position of the target image in the subsequently presented image as the aforementioned determined position modified by the determined movement.
The present invention relates to an image tracking system and method. The present invention also relates to an observer tracking autostereoscopic display in which such a system and method may be used. The system and method may also be used in other applications, such as security surveillance, video and image compression, video conferencing, computer games, driver monitoring, graphical user interfaces, camera auto-focus systems and multimedia.
Autostereoscopic displays are well known and examples are disclosed in EP 0 602 934, EP 0 656 555, EP 0 708 351, EP 0 726 482 and GB 9619097.0. FIG. 1 of the accompanying drawings illustrates schematically the basic components of a typical autostereoscopic display. The display comprises a display system 1 and a tracking system 2. The tracking system 2 comprises a tracking sensor 3 which supplies a sensor signal to a tracking processor 4. The tracking processor derives from the sensor signal an observer position data signal which is supplied to a display control processor 5 of the display system 1. The processor 5 converts the position data signal into a window steering signal and supplies this to a steering mechanism 6 which cooperates with a display 7 such that an observer 8 can view the display autostereoscopically throughout an extended range of observer positions.
According to a first aspect of the invention, there is provided an image tracking system comprising first means for presenting a sequence of images and second means for determining the position of a target image in a previously presented image from the first means, characterised by third means for determining movement of the target image between the previously presented image and a subsequently presented image from the first means, and fourth means for indicating the position of the target image in the subsequently presented image as the position determined by the second means modified by the movement determined by the third means.
FIG. 1 is a schematic diagram of a known type of observer tracking autostereoscopic display;
FIG. 16 shows an observer tracking autostereoscopic display constituting an embodiment of the invention and including a video image tracking system also constituting an embodiment of the invention. The tracking system 2 shown in FIG. 16 differs from that shown in FIG. 1 in that the tracking sensor 3 comprises a Sony XC999 NTSC camera operating at a 60 Hz field rate and the tracking processor 4 is provided with a mouse 60 and comprises a Silicon Graphics entry level machine of the Indy series equipped with an R4400 processor operating at 150 MHz and a video digitiser and frame store having a resolution of 640×240 picture elements (pixels) for each field captured by the camera 3. The camera 3 is disposed on top of the 3D display 7 and points towards the observer 8 who sits in front of the display. The normal distance between the observer 8 and the camera 3 is about 0.85 meters, at which distance the observer has a freedom of movement in the lateral or X direction of about 500 millimeters. The distance between two pixels in the image formed by the camera 3 corresponds to about 0.7 and 1.4 millimeters in the X and Y directions, respectively, the Y resolution being halved because each interlaced field is individually used. The template described hereinafter is selected to have 150×50 pixels, corresponding to a region of about 105×70 millimeters. The mouse 60 is used during template capture as described hereinafter. The camera 3 captures and presents to the processor 4 a continuous sequence of images of the user under ambient lighting.
f.sub.1 (x,y)=f.sub.2 (x+Δx,y+Δy)
Δf=f.sub.1 (x)-f.sub.2 (x)
These approximations suffer from the inevitable noise present in the image fields. One way of reducing this effect is to fit a polynomial surface over a small window. For instance, over a 3×3 element window, the image may be fitted with the following surface:
f(x,y)=ax.sup.2 +by.sup.2 +cxy+dx+ey+g
The parameters {a,b,c,d,e,g} are determined by minimising the following cost function: ##EQU7## where the summation is over the 3×3 window centred at the current pixel. The minimisation is achieved when the partial derivative with each parameter is zero. This provides a system of equations which can easily be solved. The partial derivatives of f(x,y) are then calculated as: ##EQU8## for a 3×3 window, the final expressions may be represented by the following filters: ##EQU9## which are the conventional Prewitt edge detectors.
1. An image tracking system which determines a position of a target image as the position of the target image varies, the image tracking system comprising:
first means for presenting a sequence of images,
second means for determining the position of the target image in a previously presented image from the first means,
third means for determining movement of the target image between the previously presented image and a subsequently presented image from the first means, and
fourth means for producing an output indicating the position of the target image in the subsequently presented image as the position determined by the second means modified by the movement determined by the third means,
wherein the third means is arranged to determine movement at a plurality of image elements in a second image portion which is wholly within and smaller than the target image,
the third means is arranged to determine translational movement of the target image, and
the third means is arranged to solve a set of equations:
f.sub.1 (x.sub.i,y.sub.i)=f.sub.2 (X.sub.i,y.sub.i)+Δxδf.sub.2 (x.sub.i,y.sub.i)/δx+Δyδf.sub.2 (x.sub.i,y.sub.i)/δy
where xi and yi are Cartesian coordinates of an ith image element, i is each integer such that 1≦i≦j and j is an integer greater than one, f1 and f2 are functions representing the previously and subsequently presented images and Δx and Δy are the Cartesian components of the movement.
2. A system as claimed in claim 1, wherein the subsequently presented image is consecutive with the previously presented image.
15. An image tracking system which determines a position of a target image as the position of the target image varies, the image tracking system comprising:
wherein the second means is arranged to search for the target image in a first image portion which is smaller than the images of the sequence and which includes the position indicated by the fourth means, and
the second means is arranged to search for the target image in the whole of the previously presented image if the search in the first image portion is unsuccessful.
16. An image tracking system which determines a position of a target image as the position of the target image varies, the image tracking system comprising:
the second means is arranged to search for the target image in the whole of an initial previously presented image.
17. A System as claimed in claim 13, wherein the second means is arranged to perform template matching of the target image at a plurality of first positions in the first image portion to find the best match.
18. An image tracking system which determines a position of a target image as the position of the target image varies, the image tracking system comprising:
wherein the second means is arranged to search for the target image in a first image portion which is smaller than the images of the sequence and which includes the position indicated by the fourth means,
the second means is arranged to perform template matching of the target image at a plurality of first positions in the first image portion to find the best match, and
the second means is arranged to perform template matching of the target image at a plurality of second positions which are more finely spaced than the first positions and which are disposed adjacent a position corresponding to the best match.
19. An image tracking system which determines a position of a target image as the position of the target image varies, the image tracking system comprising:
the second means is arranged to perform a correlation between the target image and a region corresponding to the first position and a correlation between the target image and a region corresponding to the second position and to select the highest correlation.
20. A system as claimed in claim 19, wherein the second means is arranged to compare the highest correlation with a threshold for acceptability.
28. An image tracking method for sequentially presented images, comprising determining the position of a target image in a previously presented image, determining movement of the target image between the previously presented image and a subsequently presented image, and indicating the position of the target image in the subsequently presented image as the position in the previously presented image modified by the determined movement
wherein the step of determining movement comprises the steps of determining movement at a plurality of image elements in a second image portion which is wholly within and smaller than the target image, determining translational movement of the target image, and solving a set of equations:
f.sub.1 (x.sub.i, y.sub.i)=f.sub.2 (x.sub.i,y.sub.i)+Δxδf.sub.2 (x.sub.i,y.sub.i)/δx+Δyδf.sub.2 (x.sub.i,y.sub.i)/δy
US09061690 1997-04-17 1998-04-16 Image tracking system and method and observer tracking autostereoscopic display Expired - Fee Related US6075557A (en)
GB9707782 1997-04-17
GB9707782A GB9707782D0 (en) 1997-04-17 1997-04-17 Image tracking system and method and observer tracking autostereoscopic display
US6075557A true US6075557A (en) 2000-06-13
ID=10810934
US09061690 Expired - Fee Related US6075557A (en) 1997-04-17 1998-04-16 Image tracking system and method and observer tracking autostereoscopic display
US (1) US6075557A (en)
EP (1) EP0877274A3 (en)
JP (1) JP3565707B2 (en)
GB (1) GB9707782D0 (en)
US6301387B1 (en) * 1998-12-18 2001-10-09 University Of Washington Template matching using correlative auto-predictive search
US20020006213A1 (en) * 2000-05-12 2002-01-17 Sergey Doudnikov Apparatus and method for displaying three-dimensional image
US6411326B1 (en) * 1997-05-21 2002-06-25 Olympus Optical Co., Ltd. Stereo image display unit
US20020110272A1 (en) * 2001-02-12 2002-08-15 Tomas Brodsky Method and apparatus for improving object boundaries extracted from stereoscopic images
US6459446B1 (en) * 1997-11-21 2002-10-01 Dynamic Digital Depth Research Pty. Ltd. Eye tracking apparatus
US20030128175A1 (en) * 2002-01-09 2003-07-10 International Business Machines Corporation Stereoscopic display system and method
WO2003098922A1 (en) * 2002-05-15 2003-11-27 The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations An imaging system and method for tracking the motion of an object
US6707933B1 (en) * 1999-11-03 2004-03-16 Kent Ridge Digital Labs Face direction estimation using a single gray-level image
US20040164939A1 (en) * 2001-05-15 2004-08-26 Johnson Mark Thomas Display device comprising a plurality of leds
US20040189804A1 (en) * 2000-02-16 2004-09-30 Borden George R. Method of selecting targets and generating feedback in object tracking systems
US20050286125A1 (en) * 2004-06-24 2005-12-29 Henrik Sundstrom Proximity assisted 3D rendering
US20070014489A1 (en) * 2005-07-13 2007-01-18 Ying Sun Nonrigid registration of cardiac perfusion MR images using adaptive local template matching
US7212674B1 (en) * 1999-11-15 2007-05-01 Fujifilm Corporation Method, apparatus and recording medium for face extraction
US20070098218A1 (en) * 2005-11-02 2007-05-03 Microsoft Corporation Robust online face tracking
US7221780B1 (en) * 2000-06-02 2007-05-22 Sony Corporation System and method for human face detection in color graphics images
US20070165037A1 (en) * 2005-12-19 2007-07-19 Samsung Electronics Co., Ltd. Apparatus and method for displaying three-dimensional image according to position of user
US20090103621A1 (en) * 2007-10-22 2009-04-23 Sony Corporation Image processing apparatus and image processing method
US20090278936A1 (en) * 2004-12-11 2009-11-12 Siegmund Pastoor Method for autostereoscopically producing three-dimensional image information from scanned sub-pixel extracts and device for carrying out said method
US20110235866A1 (en) * 2010-03-23 2011-09-29 Fujifilm Corporation Motion detection apparatus and method
US20110310003A1 (en) * 2010-05-21 2011-12-22 Fraunhofer-Gesellschaft Zur Forderung Der Angewandten Forschung E.V. Image display device and method of displaying images
US20130136302A1 (en) * 2011-11-25 2013-05-30 Samsung Electronics Co., Ltd. Apparatus and method for calculating three dimensional (3d) positions of feature points
WO2013173776A1 (en) * 2012-05-18 2013-11-21 Reald Inc. Control system for a directional light source
US20140028812A1 (en) * 2012-07-27 2014-01-30 Kabushiki Kaisha Toshiba Three-dimensional video display apparatus
US20140092472A1 (en) * 2012-10-02 2014-04-03 ReaID Inc. Stepped waveguide autostereoscopic display apparatus with a reflective directional element
US20140362194A1 (en) * 2013-06-11 2014-12-11 Kabushiki Kaisha Toshiba Image processing device, image processing method, and stereoscopic image display device
US8917441B2 (en) 2012-07-23 2014-12-23 Reald Inc. Observe tracking autostereoscopic display
US9237337B2 (en) 2011-08-24 2016-01-12 Reald Inc. Autostereoscopic display with a passive cycloidal diffractive waveplate
US9482874B2 (en) 2010-11-19 2016-11-01 Reald Inc. Energy efficient directional flat illuminators
US9898080B2 (en) 2015-09-07 2018-02-20 Samsung Electronics Co., Ltd. Method and apparatus for eye tracking
KR100516638B1 (en) 2001-09-26 2005-09-22 엘지전자 주식회사 Video telecommunication system
WO2004088348A1 (en) * 2003-03-31 2004-10-14 Seeing Machines Pty Ltd Eye tracking system and method
GB0328741D0 (en) 2003-12-11 2004-01-14 Sony Uk Ltd Object detection
EP1621153B1 (en) * 2004-07-28 2007-08-15 BrainLAB AG Stereoscopic visualisation apparatus for the combination of scanned and video images
US7359105B2 (en) 2006-02-07 2008-04-15 Sharp Kabushiki Kaisha Spatial light modulator and a display device
CN101506844B (en) 2006-08-11 2012-02-08 汤姆逊许可公司 For accurately depict motion displays and digital projectors
US20110181587A1 (en) * 2010-01-22 2011-07-28 Sony Corporation Image display device having imaging device
GB201202691D0 (en) * 2012-02-16 2012-04-04 Dimenco B V Autostereoscopic display device and drive method
US3828122A (en) * 1970-09-29 1974-08-06 Bendix Corp Tv area correlation tracker
GB1605201A (en) * 1978-03-02 1983-05-25 Elliott Brothers London Ltd Target tracking arrangements
EP0579319A2 (en) * 1992-07-16 1994-01-19 Philips Electronics Uk Limited Tracking moving objects
US5349379A (en) * 1992-09-09 1994-09-20 Dimension Technologies Inc. Autostereoscopic display illumination system allowing viewing zones to follow the observer's head
WO1996018925A1 (en) * 1994-12-13 1996-06-20 Xenotech Research Pty. Ltd. Tracking system for stereoscopic display systems
EP0769881A2 (en) * 1995-10-18 1997-04-23 Sharp Kabushiki Kaisha Method of calibrating a display which tracks an observer
GB2317291A (en) * 1996-09-12 1998-03-18 Sharp Kk Observer tracking directional display
US5771121A (en) * 1995-01-07 1998-06-23 Hentschke; Siegbert Observer-adaptive autostereoscopic shutter monitor
US5726800A (en) * 1992-12-17 1998-03-10 Sharp Kabushiki Kaisha Autostereoscopic directional display apparatus
A. Azarbayejani et al.; MIT Media Laboratory Perceptual Computing Section Technical Report No. 374, 1996; "Real-Time 3-D Tracking of the Human Body".
A. Azarbayejani et al.; MIT Media Laboratory Perceptual Computing Section Technical Report No. 374, 1996; Real Time 3 D Tracking of the Human Body . *
A. P. Pentland; Scientific American; vol. 274, No. 4, pp. 68 76, 1996, Smart Rooms . *
A. P. Pentland; Scientific American; vol. 274, No. 4, pp. 68-76, 1996, "Smart Rooms".
A. Suwa et al.; IEEE Workshop on Visual Signal Processing and Communications; 1993; "A Video Quality Improvement Technique for Videophone and Videoconference Terminal".
A. Suwa et al.; IEEE Workshop on Visual Signal Processing and Communications; 1993; A Video Quality Improvement Technique for Videophone and Videoconference Terminal . *
D. Ezra et al.; U.S. application No. 08/546,510; Filed on Oct. 20, 1995. *
European Search Report for Application No. 98302989.3 1238 dated Nov. 17, 1999. *
European Search Report for Application No. 98302989.3-1238 dated Nov. 17, 1999.
G. J. Woodgate et al.; U.S. application No. 08/592,563; Filed on Jan. 26, 1996. *
G. J. Woodgate et al.; U.S. application No. 08/863,086; Filed on May 23, 1997. *
HHI; The HHI on the Digital Media World; 1996. *
ISCAN; Eye Point of Regard Analysis Lab. *
L. S. Shapiro et al.; U.S. application No. 08/733,623; Filed on Oct. 17, 1996. *
N. Oliver et al.; MIT Media Laboratory Perceptual Computing Section Technical Report No. 396, 1997; "LAFTER: Lips and Face Real Time Tracker".
N. Oliver et al.; MIT Media Laboratory Perceptual Computing Section Technical Report No. 396, 1997; LAFTER: Lips and Face Real Time Tracker . *
Origin Instruments; The DynaSight Sensor Developer Manual. *
Origin Instruments; The DynaSight Sensor--Developer Manual.
Origin Instruments; Tthe DynaSight Sensor User Manual ; 1993. *
Origin Instruments; Tthe DynaSight Sensor--User Manual-; 1993.
R. Brunelli et al.; IEEE Trans on Pattern Analysis and Machine Intelligence; vol. 15; No. 10; 1993; "Face Recognition: Features Versus Templates".
R. Brunelli et al.; IEEE Trans on Pattern Analysis and Machine Intelligence; vol. 15; No. 10; 1993; Face Recognition: Features Versus Templates . *
Search Report for Application No. GB 9707782.0; Dated Jul. 9, 1997. *
T. S. Jebara et al.; MIT Media Laboratory, Perceptual Computing Technical Report #401; 1996; "Parametrized Structure from Motion of 3D Adaptive Feedback Tracking of Faces".
T. S. Jebara et al.; MIT Media Laboratory, Perceptual Computing Technical Report 401; 1996; Parametrized Structure from Motion of 3D Adaptive Feedback Tracking of Faces . *
US20160023100A1 (en) * 1998-08-10 2016-01-28 Cybernet Systems Corporation Real-time head tracking system for computer games and other applications
US6584224B2 (en) * 1998-12-18 2003-06-24 University Of Washington Template matching using correlative auto-predicative search
US7308111B2 (en) * 2000-05-12 2007-12-11 Lg Electronics Inc. Apparatus and method for displaying three-dimensional image
US6751345B2 (en) * 2001-02-12 2004-06-15 Koninklijke Philips Electronics N.V. Method and apparatus for improving object boundaries extracted from stereoscopic images
US7177446B2 (en) 2001-08-01 2007-02-13 Canon Kabushiki Kaisha Video feature tracking with loss-of-track detection
US20100245589A1 (en) * 2002-05-15 2010-09-30 The Board Of Governors For Higher Education State Of Rhode Island And Providence Plantations Camera control system to follow moving objects
US20050226464A1 (en) * 2002-05-15 2005-10-13 Ying Sun Camera control system to follow moving objects
JP2007503956A (en) * 2003-09-05 2007-03-01 バリアン・メディカル・システムズ・テクノロジーズ・インコーポレイテッドＶａｒｉａｎ Ｍｅｄｉｃａｌ Ｓｙｓｔｅｍｓ Ｔｅｃｈｎｏｌｏｇｉｅｓ， Ｉｎｃｏｒｐｏｒａｔｅｄ Track objects and devices and methods for monitoring the position moving
WO2005064944A1 (en) * 2003-12-22 2005-07-14 Objectvideo, Inc. Master-slave automated video-based surveillance system
US8300043B2 (en) * 2004-06-24 2012-10-30 Sony Ericsson Mobile Communications AG Proximity assisted 3D rendering
US8130272B2 (en) * 2004-12-11 2012-03-06 Fraunhofer-Gesellschaft Zur Forderung Der Angewandten Forschung E.V. Method for autostereoscopically producing three-dimensional image information from scanned sub-pixel extracts and device for carrying out said method
US9805566B2 (en) 2005-09-08 2017-10-31 Avigilon Fortress Corporation Scanning camera-based video surveillance system
US9363487B2 (en) 2005-09-08 2016-06-07 Avigilon Fortress Corporation Scanning camera-based video surveillance system
US8537208B2 (en) * 2005-12-19 2013-09-17 Samsung Electronics Co., Ltd. Apparatus and method for displaying three-dimensional image according to position of user
CN101420613B (en) 2007-10-22 2012-10-10 索尼株式会社 Image processing device and image processing method
US8199805B2 (en) * 2007-10-22 2012-06-12 Sony Corporation Image processing apparatus and image processing method
US9600714B2 (en) * 2011-11-25 2017-03-21 Samsung Electronics Co., Ltd. Apparatus and method for calculating three dimensional (3D) positions of feature points
US9420266B2 (en) * 2012-10-02 2016-08-16 Reald Inc. Stepped waveguide autostereoscopic display apparatus with a reflective directional element
EP0877274A2 (en) 1998-11-11 application
EP0877274A3 (en) 1999-12-29 application
GB9707782D0 (en) 1997-06-04 grant
GB2324428A (en) 1998-10-21 application
JPH113429A (en) 1999-01-06 application
JP3565707B2 (en) 2004-09-15 grant
US5231674A (en) 1993-07-27 Eye tracking method and apparatus
US20020075384A1 (en) 2002-06-20 Eye tracking apparatus
US20030142068A1 (en) 2003-07-31 Selective real image obstruction in a virtual reality display apparatus and method
US5912721A (en) 1999-06-15 Gaze detection apparatus and its method as well as information display apparatus
US20010030715A1 (en) 2001-10-18 Stereo image display apparatus
Morimoto et al. 2000 Pupil detection and tracking using multiple light sources
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HOLLIMAN, NICOLAS STEVEN;HONG, QI HE;EZRA, DAVID;AND OTHERS;REEL/FRAME:009336/0675