Patent Publication Number: US-2010110209-A1

Title: Fast motion measurement device for gaming

Description:
FIELD OF THE INVENTION 
     The present invention generally relates to a dedicated motion sensing measurement system based on an image sensor that is optimized for measurement of fast motions. 
     BACKGROUND OF THE INVENTION 
     Current motion measurement systems for interactive visual systems such as gaming or automatic driving systems rely on gyro-sensing of movement or analysis of video images. Gaming systems such as the Nintendo Wii™ device use a gyro-device to sense motion of the hand held interface. However, this device senses rotational movements and as such does not sense movement which occurs with a constant linear velocity. Automotive movement sensors for collision avoidance rely on analysis of video images as captured at standard video rates such as 30 frames/sec. In many interactive systems, the motion occurs with sections of linear velocity with rapid changes in direction. As a result, a need exists for a motion sensing measurement system that can measure rapid motion of all types with rapid changes in direction. 
     In U.S. Pat. No. 7,194,126, motion analysis is performed on video images from a stereo image capture system. However, the system is relatively slow due to low sensitivity of the imaging system and high data content in each image resulting in image processing limitations. As a result, the system operates at only 8 frames/sec. 
     In United States Patent Publication No. 2003/0235327, a method for surveillance is described which is based on object tracking with edge maps to define the shape and location of objects. A number of techniques for generating edge maps are described. In addition, for consumer applications such as interactive gaming and automotive applications, it is desirable to have a motion measurement system that is low in cost and requires low computational power. 
     United States Patent Publication No. 2006/0146161 discloses a CMOS sensor which has two detectors in each pixel. By operating the pixels so that the two detectors have different exposure times, motion in the image can be detected at each pixel by subtracting the signals from the two detectors. 
     To provide a motion measurement system that is capable of measuring rapid motion, the present invention discovered that a sensitive imaging system should be combined with image processing that reduces the amount of data that needs to be processed to obtain motion information. 
     SUMMARY OF THE INVENTION 
     It is an object of the present invention to provide a motion measurement system that is capable of measuring motion data at 500 frames/sec or faster. The present invention achieves this rapid processing by using a sensitive imaging system that is combined with an image processing method that substantially reduces the amount of data that must be analyzed in making the motion measurements. The data reduction is provided by converting the captured images into edge maps with 1 bit depth. 
     A motion measurement system is described with one lens and one imaging sensor for generating x-y motion information. 
     Another motion measurement system is described with two lenses and two image sensors for generating x-y-z motion information. 
     A system embodiment is described which includes a sensitive imaging system for use with the motion measurement method of the present invention. 
     These and other objects, features, and advantages of the present invention will become apparent to those skilled in the art upon a reading of the following detailed description when taken in conjunction with the drawings wherein there is shown and described an illustrative embodiment of the invention. 
     ADVANTAGEOUS EFFECTS OF THE INVENTION  
     The present invention has the advantage of faster image capture, along with reduced data in the images so that the data processing is reduced when obtaining motion measurements. Embodiments are disclosed for obtaining x-y motion measurements and x-y-z motion measurements. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       While the specification concludes with claims particularly pointing out and distinctly claiming the subject matter of the present invention, it is believed that the invention will be better understood from the following description when taken in conjunction with the accompanying drawings, wherein: 
         FIG. 1  is a block diagram of an image capture system of the present invention; 
         FIG. 2  is a schematic diagram of an image capture device of the present invention with 1 lens and 1 image sensor; 
         FIG. 3  is a flowchart for an embodiment of the method of the present invention in which an image capture device such as that shown in  FIG. 2  is used to generate an x-y motion map; 
         FIG. 3A  is an illustration of an image as captured from a first image capture device of the present invention; 
         FIG. 3B  is an illustration of an image as captured with an object in x-direction translation from first image by the first image capture device; 
         FIG. 3C  is an edge map of the image from  FIG. 3A ; 
         FIG. 3D  is an edge map of the image from  FIG. 3B ; 
         FIG. 4  is a schematic diagram of an image capture device of the present invention with 2 lenses and 2 image sensors; 
         FIG. 5  is a flow chart for another embodiment of the method of the present invention in which an image capture device such as that shown in  FIG. 4  is used to generate an x-y motion map along with a z motion map; 
         FIG. 5A  is an illustration of an image as captured from a first image capture device of the present invention; 
         FIG. 5B  is an edge map of the image from  FIG. 5A ; 
         FIG. 5C  is an illustration of an image as captured with object in z-direction translation from first image by the first image capture device; 
         FIG. 5D  is an edge map of the image from  FIG. 5C ; 
         FIG. 5E  is an illustration of an image as captured from a second image capture device of the present invention; 
         FIG. 5F  is an edge map of the image from  FIG. 5E ; 
         FIG. 5G  is an illustration of an image as captured with object in z-direction translation from first image by the second image capture device of the present invention; 
         FIG. 5H  is an edge map of the image from  FIG. 5G ; and 
         FIG. 6  is an illustration of a buffer as can be used to form edge maps. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The present invention provides a motion measurement system that is capable of rapid motion measurement which can measure all types of motion including rotational motion, linear motion and motion occurs in the directions of x-y and x-y-z. In the motion measurement system of the present invention, motion is measured with a high speed image capture and analysis system that captures images of the scene and identifies motion in the scene with a reduced bit rate approach to enable fast data processing. 
       FIG. 1  shows a block diagram of an image capture device  100  of the present invention, such as a digital camera. The lens assembly  110  gathers light from the scene being imaged along an optical axis to produce an image on the image sensor  120 . The image sensor  120  has an array of pixels that converts the scene to charge packets and eventually a plurality of signal levels referred to herein as image data. The image data is transmitted to the image processor  130  where the image data is analyzed and improved to form a final image. The final image is sent to storage and/or to a display  140 . Alternately, the final image can be sent to the data transmitter  160  for transmission to another device. Based on input from the user interface  150 , the image processor  130  can analyze the image differently to determine what capture control signals to send to the lens assembly  110  and the image sensor  120 , and how the final image should be handled. Input from the user interface  150  can also determine the method that the image data is sent for storage, display or transmission.  FIG. 2  shows a cross sectional schematic diagram of the image capture device for an embodiment of the present invention which includes 1 lens assembly and 1 image sensor. The image capture device  100  includes a lens assembly  110  which gathers light from a scene and presents the light along a common optical axis  290  to an image sensor  120 . The lens assembly  110  is mounted inside a barrel  220  (or formed as a stacked device such as in wafer level manufactured lenses) which is mounted in a case  280  along with the image sensor  120 , the image processor  130 , the storage and display unit  140 , and the data transmitter  160 . The lens assembly  110  can include fixed or moveable lens elements, a shutter, an iris or an image stabilizer. The image sensor  120  includes an array of pixels. The user interface  150  is shown as a button but the user interface can include a variety of buttons, knobs and touch sensitive devices that the operator can operate. In this embodiment, motion is measured in the directions that are perpendicular to the optical axis  290  of the lens, as in the x, y directions. 
       FIG. 3  shows a flow chart for an embodiment of the present invention where x-y motion is measured. In Step  310 , a first image is captured by the image sensor  120  wherein the image is designated image  1 . An example of image  1  is shown in  FIG. 3A . In Step  320 , the code value for each pixel in image  1  is subtracted from one or more adjacent pixels in the image processor  130 . (The method for subtraction is described herein below.) In Step  330  an edge map EM 1  is created in the image processor  130  based on the subtracted values for each pixel, wherein if the subtracted value is less than a threshold value, the pixel is determined to be in a relatively homogeneous region in the image and as such not at an edge, so the value for the pixel in EM 1  is set to 0. However, if the subtracted value is greater than a threshold value, the pixel is determined to be located at an edge and the pixel value in EM 1  is set to 1. As previously stated, United States Patent Publication No. 2003/0235327 describes a number of methods for generating edge maps. As part of the present invention, the image is converted to an edge map with a bit depth of 1 bit per pixel. By prudently selecting the edge threshold value, much of the data in the image that is attributed to small scale detail such as texture is eliminated so that only larger objects are identified in the edge map and as such the imaging data connected to the small scale detail does not have to be considered for motion measurements. By converting the image to a 1 bit edge map, the amount of data is reduced from a typical 8 or 10 bit depth per pixel to 1 bit which equates to a reduction in the number of bits that must be processed to 1/256 or less. This reduction in the number of bits greatly increases the rate that the images can be processed. An example of EM 1  is shown in  FIG. 3C . In Step  340 , a second image is captured sequentially by the image sensor  120  which is designated image  2 . An example of image  2  is shown in  FIG. 3B . In Step  350 , the code value for each pixel in image  2  is subtracted from one or more adjacent pixels by the image processor  130 . In Step  360  an edge map EM 2  is created in the image processor  130  based on the subtracted values for each pixel, wherein if the subtracted value is less than a threshold value, the pixel is determined to be in a relatively homogeneous region in the image and as such not at an edge, so the value for the pixel in EM 2  is set to 0. However, if the subtracted value is greater than a threshold value, the pixel is determined to be located at an edge and the pixel value in EM 2  is set to 1. An example of EM 2  is shown in  FIG. 3D . 
     In Step  370 , EM 1  is correlated to EM 2  in the image processor  130  to determine the location and degree of differences between EM 1  and EM 2 . In Step  380 , the locations and degrees of difference between EM 1  and EM 2  are used to create an x-y motion map. The x-y motion is then stored, analyzed for appropriate action or transmitted to another device where the x-y motion is analyzed for appropriate action such as changing the action in an interactive game or causing the brakes to be applied in an automobile. The x-y motion can be converted to x-y velocity by combining the x-y motion and the time between captures of images  1  and  2 . 
     Referring to  FIG. 6 , the present invention subtracts by storing three consecutive lines from the image into a buffer  600 . It is noted that the number of storage units lengthwise of the buffer  600  equals the number of pixels in a row of the image (The number of pixels excludes any non-imaging pixels included on the image sensor, commonly referred to as black pixels.) It is also noted that the locations are identified with the row as the first value of the location and the columns as the second value of the location. For simplicity of illustration, location  2 , 2  (the kernel location) is used in this example. The image processor  130  retrieves the value at location  2 , 2  and the locations  2 , 1 ;  1 , 2 ;  2 , 3 ; and  3 , 2  and does four subtractions. Location  2 , 2  is subtracted from location  2 , 1  and is temporarily stored; location  2 , 2  is subtracted from location  1 , 2  and is temporarily stored; location  2 , 2  is subtracted from location  2 , 3  and is temporarily stored; and location  2 , 2  is subtracted from location  3 , 2  and is temporarily stored. All the values in temporary storage are added together and compared to a threshold. If the compared value is greater than the threshold, a value of one is stored in a fourth line of the buffer (edge map readout row) at location  4 , 2  (In other words, in the 4 th  row of the buffer and at the column location of the kernel location.) If the compared value is less than the threshold, a value of zero is stored in a fourth line of the buffer at location  4 , 2  (In other words, in the 4 th  row of the buffer and at the column location of the kernel location.) This process is repeated for every location until the end of the line is reached and then the fourth row is read out. All the buffer locations are shifted down one row ( 2 , 2  goes to location  3 , 2 ). The subtraction process is then repeated for every location except the edge locations which are automatically set to zero. Those skilled in the art may readily recognize that other variations of the subtractive process are possible. A subtractive approach is preferred due to its low computational requirements which allows the process to run quickly in the image processor. 
     Within the scope of the present invention, the image sensor  120  and the image processor  130  can be separate devices or they can be combined on one chip to reduce the cost and size of the image capture device  100 . Alternately, aspects of the image processor  130  such as those associated with converting images to edge maps may be included with the image sensor  120  in a single working module which provides edge maps or motion information as along with other information as described below in the invention while other aspects of image processing are done in a separate image processor  130 . 
       FIG. 4  shows a further embodiment of the present invention in which an image capture device  400  is comprised of 2 lens assemblies and 2 image sensors. Wherein, in addition to the elements shown in common with  FIG. 2 , a second lens assembly  410  mounted in a barrel  420  gathers light from a scene and presents the light along a second optical axis to the second image sensor  430 . The image capture device  400  is mounted in an enclosure  480 . The optical axis  290  of lens assembly  110  is separated by a distance (such as 5 to 100 mm) from the optical axis  490  of lens assembly  410  so that images captured by image sensors  120  and  430  have different perspectives of the scene being imaged. 
       FIG. 5  shows a flow chart for another method which is an embodiment of the present invention in which a motion map with x-y-z motion is created. In Step  510 , a first image is captured by image sensor  120  wherein the image is designated image  1 A. An example of image  1 A is shown in  FIG. 5A . In Step  520 , the code value for each pixel in image  1 A is subtracted from one or more adjacent pixels in image processor  130 . In Step  530  an edge map EM 1 A is created in the image processor  130  based on the subtracted values for each pixel, wherein if the subtracted value is less than a threshold value, the pixel is determined to be in a relatively homogeneous region in the image and as such not at an edge, so the value for the pixel in EM 1 A is set to 0. However, if the subtracted value is greater than a threshold value, the pixel is determined to be located at an edge and the pixel value in EM 1 A is set to 1. An example of image EM 1 A is shown in  FIG. 5B . 
     In Step  540 , a second image is captured sequentially by the image sensor  120  which is designated image  2 A. An example of image  2 A is shown in  FIG. 5C . In Step  550 , the code value for each pixel in image  2 A is subtracted from one or more adjacent pixels by image processor  130 . In Step  560  an edge map EM 2 A is created in the image processor  130  based on the subtracted values for each pixel, wherein if the subtracted value is less than a threshold value, the pixel is determined to be in a relatively homogeneous region in the image and as such not at an edge, so the value for the pixel in EM 2 A is set to 0. However, if the subtracted value is greater than a threshold value, the pixel is determined to be located at an edge and the pixel value in EM 2 A is set to 1. An example of image EM 2 A is shown in  FIG. 5D . 
     In Step  570 , EM 1 A is correlated to EM 2 A in the image processor  130  to determine the location and degree of differences between EM 1 A and EM 2 A. In Step  580 , the locations and degrees of difference between EM 1 A and EM 2 A are used to create an x-y motion map. By combining the x-y motion map with the time between captures of images  1 A and  2 A, the x-y velocities for objects in the scene can be calculated. 
     In Step  512 , a first image is captured by image sensor  430  wherein the image is designated image  1 B. An example of image  1 B is shown in  FIG. 5E . Preferably Step  512  is performed substantially simultaneously with Step  510  so that the motion and lighting present during the capture of image  1 A and image  1 B is substantially the same. In addition, the photographic conditions such as exposure time and iris setting should be substantially the same for image  1 A and image  1 B so the images look substantially the same. In Step  522 , the code value for each pixel in image  1 B is subtracted from one or more adjacent pixels in the image processor  130 . In Step  532  an edge map EM 1 B is created in the image processor  130  based on the subtracted values for each pixel, wherein if the subtracted value is less than a threshold value, the pixel is determined to be in a relatively homogeneous region in the image and as such not at an edge, so the value for the pixel in EM 1 B is set to 0. However, if the subtracted value is greater than a threshold value, the pixel is determined to be located at an edge and the pixel value in EM 1 B is set to 1. An example of image EM 1 B is shown in  FIG. 5F . 
     In Step  585 , EM 1 A is correlated to EM 1 B to create a disparity map DM 1  which shows the differences in locations of objects in EM 1 A and EM 1 B due to their respective different perspectives. Wherein the disparity map shows the pixel shifts needed to get the edges of like objects in EM 1 A and EM 1 B to align. By calibrating the image capture device for measured disparity value vs z location, the disparity values in the disparity map can be used to calculate z locations by triangulation along with the separation between the optical axes of lenses  110  and  410 . A discussion of methods to produce rangemaps and disparity maps for the purpose of autofocus, improved image processing and object location mapping can be found in U.S. patent application Ser. No. 11/684,036. 
     In Step  542 , a second image is captured sequentially by image sensor  430  which is designated image  2 B. An example of image  2 B is shown in  FIG. 5G . Step  542  is preferably performed at substantially the same time as Step  540  so that image  2 A and image  2 B have substantially the same motion and lighting present during the capture of the images. In addition, the photographic settings such as exposure time and iris setting should be the same so that image  2   a  and  2 B look substantially the same. In Step  552 , the code value for each pixel in image  2 B is subtracted from one or more adjacent pixels by image processor  130 . In Step  562  an edge map EM 2 B is created in image processor  130  based on the subtracted values for each pixel, wherein if the subtracted value is less than a threshold value, the pixel is determined to be in a relatively homogeneous region in the image and as such not at an edge, so the value for the pixel in EM 2 B is set to 0. However, if the subtracted value is greater than a threshold value, the pixel is determined to be located at an edge and the pixel value in EM 2 B is set to 1. An example of image EM 2 B is shown in  FIG. 5H . 
     In Step  590 , EM 2 A is correlated to EM 2 B to create a disparity map DM 2  which shows the differences in locations of objects in EM 2 A and EM 2 B due to their respective different perspectives. 
     In Step  595 , DM 1  is correlated to DM 2  to identify differences between the disparity maps based on z motion and produce a z motion map. The z motion information is then stored, analyzed for appropriate action or transmitted to another device where the z motion is analyzed for appropriate action. The benefit provided by reduced image processing needs is similar to that given for the previous embodiment since the data is again reduced from 8 or 10 bit depth to 1 bit depth. 
     By combining the change in disparity for each pixel with the time between image captures for images  1 A and  2 A (which should be substantially the same time as the time between captures for images  1 B and  2 B), the z velocities for objects in the images can be calculated. 
     While the invention is preferentially described as converting images to a 1 bit edge maps for motion measurement, those skilled in the art will recognize that the goal of the invention is to greatly reduce the number of bits associated with each image so that faster motion measurement is possible, however there may be conditions where a 2 bit or more edge map will enable a better interpretation of the edge map while still providing a substantial reduction in the number of bits associated with the images. 
     In a system embodiment of the present invention, the above method is used within a system that includes a sensitive imaging system, wherein the lens assemblies  110  and  410  have been improved to increase the amount of light that is gathered from the scene by using a lens that has an f# of 3.0 or less. The lens can have a fixed focal length to reduce costs or it can have a variable focal length to provide more flexible imaging capabilities. The lens can have a focus setting that is fixed to further reduce costs and to improve the accuracy of the calibration for triangulation and eliminate the need to autofocus or the lens can have an autofocus system. Similarly, the image sensors  120  and  430  can be improved to increase the efficiency of converting the image from the lens to image data. An image sensor with panchromatic pixels which gather light from substantially the full visible spectrum can be used. In addition, the image sensor can be operated without an infrared filter to extend the spectrum that light is gathered into the near infrared. Image sensors with larger pixels provide a larger area for gathering light. The image sensors can be backside illuminated to increase the active area of the pixels for gathering light and increase the quantum efficiency of the pixels in the image sensors. Finally for operation where the moving objects to be measured are within approximately 15 feet of the image capture device, an infrared light can be used to supply illumination to the scene. 
     As an example, compared to an image capture device with an f# 3.5 lens and a standard front side illuminated Bayer image sensor with 2.0 micron pixels and an infrared filter and no illumination from the image capture device, a preferred embodiment of the invention would include: an f#2.5 (or less) lens; a fixed focal length which has a field of view that is just wide enough to image the desired scene; a fixed focus setting in the middle of the depth of the desired scene (in absence of a prior knowledge of the depth of the scene, the lens should be focused at the hyperfocal length which is the focus setting which provides the largest depth of field); an image sensor with panchromatic pixels; no infrared filter; 4.0 micron pixels; the image sensor is back side illuminated and an infrared light is provided with the image capture device (note that in portable applications, the power consumption of the infrared light may be too large to support with batteries). The benefits of these changes for the preferred embodiment are as follows:
         a) An f#2.5 lens gathers 2× the light compared to an f# 3.5 lens.   b) A fixed focal length lens with a non-autofocus focus system (fixed or manually adjustable focus) is much lower cost and time required to autofocus is eliminated.   c) Panchromatic pixels provide 3× the light gathering compared to Bayer pixels.   d) Eliminating the infrared filter provides a 20% increase in available light.   e) 4.0 micron pixels have 4× the light gathering area compared to 2.0 micron pixels.   f) A backside illuminated pixel has 3× the active area and 2× the quantum efficiency of a front side illuminated pixel.   g) An infrared light can provide illumination in an environment that is perceived to be dark by the user.
 
Thus, by combining the method of the present invention for measuring x-y motion and x-y-z motion with a sensitive imaging system, a very fast motion measuring system is provided which can operate at over 500 frames/sec.
       

     In a further embodiment of the invention, the image sensor(s) includes panchromatic pixels and color pixels distributed across the image sensor. The panchromatic pixels are readout separately from the color pixels to form panchromatic images and color images. The panchromatic images are converted to edge maps that are subsequently used to measure motion. The color images are stored as color images without being reduced in bit depth. The frame rate of the panchromatic images is faster than the frame rate for the color images, for example, the frame rate of the panchromatic images can be 10× the frame rate of the color images so that fast motion measurements can be obtained at the same time that low noise color images can be obtained. 
     The present invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention. 
     PARTS LIST 
     
         
           100  Image capture device 
           110  Lens assembly block 
           120  Image sensor block 
           130  Image processor block 
           140  Storage and display block 
           150  User interface block 
           160  Data transmitter block 
           220  Barrel 
           280  Enclosure 
           290  optical axis 
           310  Step 
           320  Step 
           330  Step 
           340  Step 
           350  Step 
           360  Step 
           370  Step 
           380  Step 
           400  Image capture device with 2 lenses and 2 image sensors 
           410  Lens assembly 
           420  Barrel 
           430  Image sensor 
           480  Enclosure 
           510  Step 
           512  Step 
           520  Step 
           522  Step 
           530  Step 
           532  Step 
           540  Step 
           542  Step 
           550  Step 
           552  Step 
           560  Step 
           562  Step 
           570  Step 
           580  Step 
           585  Step 
           590  Step 
           595  Step 
           600  Buffer