Patent Publication Number: US-11650320-B2

Title: System and method for refining coordinate-based three-dimensional images obtained from a three-dimensional measurement system

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation application of U.S. application Ser. No. 16/517,754, which was filed on Jul. 22, 2019, now granted as U.S. Pat. No. 10,955,554; which is a continuation application of U.S. application Ser. No. 15/942,371, which was filed on Mar. 30, 2018, abandoned; which is a continuation application of U.S. application Ser. No. 15/162,454, which was filed on May 23, 2016, now granted as U.S. Pat. No. 9,933,522; which is a continuation application of U.S. application Ser. No. 13/841,304, which was filed on Mar. 15, 2013; which in turn claims priority to U.S. Provisional Application No. 61/693,969, which was filed on Aug. 28, 2012. Each of the foregoing applications is incorporated herein by reference as if reproduced below in its entirety. 
    
    
     FIELD OF THE INVENTION 
     The invention is generally related to combining lidar (i.e., laser radar) measurements and video images to generate three dimensional images of targets, and more particularly, to refining the three dimensional images of the targets. 
     BACKGROUND OF THE INVENTION 
     Various conventional systems attempt to merge lidar measurements and video images to obtain three dimensional images of targets. Typically, these conventional systems require some pre-specified initial model of the target in order to combine lidar measurements with the video image to form a three dimensional image. 
     Unfortunately, the pre-specified initial model places significant restraints on the ability of the systems to combine lidar measurements with the video image and as a result, the three dimensional image is seldom sufficient for purposes of identifying the target. 
     Furthermore, these conventional systems typically are unable to adequately account for motion of the target and hence often require that the target remain substantially motionless during an image capture period. 
     What is needed is an improved system and method for capturing three dimensional images using lidar and video measurements and refining the three dimensional images to account for stochastic errors. 
     SUMMARY OF THE INVENTION 
     Various implementations of the invention combine measurements generated by a a lidar system with images generated by a video system to resolve a six degrees of freedom trajectory that describes motion of a target. Once this trajectory is resolved, an accurate three-dimensional image of the target may be generated. In some implementations of the invention, the system may refine the 3D image by reducing the stochastic components in the transformation parameters (i.e., v x , v y , v z , ω x , ω y  and ω z ) between video frame times. 
     These implementations, their features and other aspects of the invention are described in further detail below. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    illustrates a combined lidar and video camera system according to various implementations of the invention. 
         FIG.  2    illustrates a lidar (i.e., laser radar) according to various implementations of the invention. 
         FIG.  3    illustrates a scan pattern for a lidar subsystem that employs two lidar beams according to various implementations of the invention. 
         FIG.  4    illustrates a scan pattern for a lidar subsystem that employs four lidar beams according to various implementations of the invention. 
         FIG.  5    illustrates a relationship between points acquired from the lidar subsystem from separate beams at substantially the same instance of time that may be used to estimate an x-component of angular velocity of a target according to various implementations of the invention. 
         FIG.  6    illustrates a relationship between points acquired from the lidar subsystem from separate beams at substantially the same instance of time that may be used to estimate a y-component of angular velocity of a target according to various implementations of the invention. 
         FIG.  7    illustrates a relationship between points acquired from the video subsystem that may be used to estimate a two-dimensional (e.g. x and y components) translational velocity and a z-component of angular velocity of a target according to various implementations of the invention. 
         FIG.  8    illustrates a scan pattern of a lidar beam according to various implementations of the invention. 
         FIG.  9    illustrates a timing diagram which may be useful for describing various timing aspects associated with measurements from the lidar subsystem according to various implementations of the invention. 
         FIG.  10    illustrates a timing diagram which may be useful for describing various timing aspects associated with measurements from the lidar subsystem in relation to measurements from the video subsystem according to various implementations of the invention. 
         FIG.  11    illustrates a block diagram useful for processing lidar measurements and video images according to various implementations of the invention. 
         FIG.  12    illustrates a block diagram useful for processing lidar measurements and video images according to various implementations of the invention. 
         FIG.  13    illustrates a block diagram useful for processing lidar measurements and video images according to various implementations of the invention. 
         FIG.  14    illustrates a block diagram useful for processing lidar measurements and video images according to various implementations of the invention. 
         FIG.  15    illustrates a block diagram useful for processing lidar measurements and video images according to various implementations of the invention. 
         FIG.  16    illustrates a flowchart depicting example operations performed by a system for refining a 3D image, according to various aspects of the invention. 
         FIG.  17    illustrates a flowchart depicting example operations performed by a system for refining a 3D image, according to various aspects of the invention. 
     
    
    
     DETAILED DESCRIPTION 
       FIG.  1    illustrates a combined lidar and video camera system  100  according to various implementations of the invention. Various implementations of the invention utilize synergies between lidar measurements and video images to resolve six degrees of freedom for motion of a target to a degree not otherwise possible with either a lidar or video camera alone. 
     Combined system  100  includes a lidar subsystem  130 , a video subsystem  150 , and a processing system  160 . As illustrated, lidar subsystem  130  includes two or more lidar beam outputs  112  (illustrated as a beam  112 A, a beam  112 B, a beam  112 ( n− 1), and a beam  112   n ); two or more reflected beam inputs  114  each corresponding to one of beams  112  (illustrated as a reflected beam  114 A, a reflected beam  114 B, a reflected beam  114 ( n −1), and a reflected beam  114   n ); two or more lidar outputs  116  each associated with a pair of beam  112 /reflected beam  114  (illustrated as a lidar output  116 A associated with beam  112 A/reflected beam  114 A, a lidar output  116 B associated with beam  112 B/reflected beam  114 B, a lidar output  116 ( n −1) associated with beam  112 ( n −1)/reflected beam  114 ( n −1), and a lidar output  116   n  associated with beam  112   n /reflected beam  114   n ). 
     In some implementations of the invention, beam steering mechanism  140  may be employed to steer one or more beams  112  toward target  190 . In some implementations of the invention, beam steering mechanism  140  may include individual steering mechanisms, such as a steering mechanism  140 A, a steering mechanism  140 B, a steering mechanism  140 C, and a steering mechanism  140 D, each of which independently steers a beam  112  toward target  190 . In some implementations of the invention, one beam steering mechanism  140  may independently steer pairs or groups of beams  112  toward target  190 . 
     In some implementations of the invention, beam steering mechanism  140  may include one or more mirrors, each of which may or may not be separately controlled, each mirror steering one or more beams  112  toward target  190 . In some implementations of the invention, beam steering mechanism  140  may directly steer an optical fiber of beam  112  without use of a mirror. In some implementations of the invention, beam steering mechanism  140  may be controlled to steer beams  112  in azimuth and/or elevation. Various techniques may be used by beam steering mechanism  140  to steer beam(s)  112  toward target  190  as would be appreciated. 
     In some implementations of the invention, beam steering mechanism  140  may be used to control both an azimuth angle and an elevation angle of two beams  112  toward the target. By controlling both the azimuth angle and the elevation angle, the two beams  112  may be used to scan a volume for potential targets or track particular targets such as target  190 . Other scanning mechanisms may be employed as would be apparent. In some implementations of the invention, the two beams  112  may be offset from one another. In some implementations of the invention, the two beams  112  may be offset vertically (e.g., in elevation) or horizontally (e.g., in azimuth) from one another by a predetermined offset and/or a predetermined angle, either of which may be adjustable or controlled. 
     In some implementations of the invention, beam steering mechanism  140  may be used to control both an azimuth angle and an elevation angle of four beams  112  toward the target. In some implementations, the four beams  112  may be arranged with horizontal and vertical separations. In some implementations, the four beams may be arranged so as to form at least two orthogonal separations. In some implementations, the four beams may be arranged in a rectangular pattern, with pairs of beams  112  offset from one another vertically and horizontally. In some implementations, the four beams may be arranged in other patterns, with pairs of beams  112  offset from one another. The separations of the four beams  112  may be predetermined offsets and/or predetermined angles, which may be fixed, adjustable and/or controlled. 
     A certain portion of each beam  112  may be reflected back from target  190  to lidar subsystem  130  as reflected beam  114 . In some implementations of the invention and as illustrated in  FIG.  1   , reflected beam  114  follows the same optical path (though in reverse) as beam  112 . In some implementations of the invention, a separate optical path may be provided in lidar subsystem  130  or in combined system  100  to accommodate reflected beam  114 . 
     In some implementations of the invention, lidar subsystem  130  receives a reflected beam  114  corresponding to each beam  112 , processes reflected beam  114 , and outputs lidar output  116  to processing system  160 . 
     Combined system  100  also includes video subsystem  150 . Video subsystem  150  may include a video camera for capturing two dimensional images  155  of target  190 . Various video cameras may be used as would be apparent. In some implementations of the invention, the video camera may output images  155  as pixels at a particular resolution and at a particular image or frame rate. Video images  155  captured by video subsystem  150  are forwarded to processing system  160 . In some implementations of the invention, lidar subsystem  130  and video subsystem  150  are offset from one another in terms of position and orientation. In particular, lidar measurements typically correspond to three dimensions (e.g., x, y, and z) whereas video images typically correspond to two dimensions (e.g., x and y). Various implementations of invention calibrate lidar subsystem  130  with video subsystem  150  to ensure that data provided by each system refers to the same location in a given coordinate system as would be apparent. 
     Combined system  110  may include one or more optional video subsystems (not otherwise illustrated) for capturing additional two-dimensional images  155  of target  190  from different positions, perspectives or angles as would be apparent. 
     In some implementations of the invention, processing system  160  receives lidar outputs  116  from lidar subsystem  130  and images  155  from video subsystem  150  and stores them in a memory or other storage device  165  for subsequent processing. Processing system  160  processes lidar outputs  116  and images  155  to generate a three-dimensional image of target  190 . In some implementations of the invention, processing system  160  determines a trajectory of target  190  from a combination of lidar outputs  116  and images  155  and uses the trajectory to generate a motion stabilized three-dimensional image of target  190 . 
     In some implementations of the invention, lidar subsystem  130  may include, for each of beams  112 , a dual frequency, chirped coherent laser radar system capable of unambiguously and simultaneously measuring both range and Doppler velocity of a point on target  190 . Such a laser radar system is described in co-pending U.S. application Ser. No. 11/353,123, entitled “Chirped Coherent Laser Radar System and Method,” (the “Chirped Lidar Specification”), which is incorporated herein by reference in its entirety. For purposes of clarity, a “beam” referenced in the Chirped Lidar Specification is not the same as a “beam” referred to in this description. More particularly, in the Chirped Lidar Specification, two beams are described as output from the laser radar system, namely a first beam having a first frequency (chirped or otherwise) and a second beam having a second frequency (chirped or otherwise) that are simultaneously coincident on a point on a target to provide simultaneous measurements of both range and Doppler velocity of the point on the target. For purposes of simplicity and clarity, a singular “beam” as discussed herein may refer to the combined first and second beams output from the laser radar system described in the Chirped Lidar Specification. The individual beams discussed in the Chirped Lidar Specification are referred to herein henceforth as “signals.” Nonetheless, various implementations of the invention may employ beams other than those described in the Chirped Lidar Specification provided these beams provide simultaneous range and Doppler velocity measurements at points on the target. 
       FIG.  2    illustrates a lidar  210  that may be used to generate and process beam  112  and reflected beam  114  to provide lidar output  116  according to various implementations of the invention. Each lidar  210  unambiguously determines a range and Doppler velocity of a point on target  190  relative to lidar  210 . Lidar  210  includes a first frequency lidar subsection  274  and a second frequency lidar subsection  276 . First frequency lidar subsection  274  emits a first frequency target signal  212  toward target  190  and second frequency lidar subsection  276  emits a second frequency target signal  214  toward target  190 . The frequencies of first target signal  212  and second target signal  214  may be chirped to create a dual chirp system. 
     First frequency lidar subsection  274  may include a laser source controller  236 , a first laser source  218 , a first optical coupler  222 , a first signal delay  244 , a first local oscillator optical coupler  230 , and/or other components. Second frequency lidar subsection  276  may include a laser source controller  238 , a second laser source  220 , a second optical coupler  224 , a second signal delay  250 , a second local oscillator optical coupler  232  and/or other components. 
     First frequency lidar subsection  274  generates first target signal  212  and a first reference signal  242 . First target signal  212  and first reference signal  242  may be generated by first laser source  218  at a first frequency that may be modulated at a first chirp rate. First target signal  212  may be directed toward a measurement point on target  190  either independently or combined with second target signal  214 . First frequency lidar subsection  274  may combine target signal  256  that was reflected from target  190  with first reference signal  242 , which is directed over a path with a known or otherwise fixed path length, to result in a combined first target signal  262 . 
     Second frequency lidar subsection  276  may be collocated and fixed with respect to first frequency lidar subsection  274  (i.e., within lidar  210 ). More particularly, the relevant optical components for transmitting and receiving the respective laser signals may be collocated and fixed. Second frequency lidar subsection  276  may generate second target signal  214  and a second reference signal  248 . Second target signal  214  and second reference signal  248  may be generated by second laser source  220  at a second frequency that may be modulated at a second chirp rate. In some implementations of the invention, the second chirp rate is different from the first chirp rate. 
     Second target signal  214  may be directed toward the same measurement point on target  190  as first target beam  212 . Second frequency lidar subsection  276  may combine one portion of target signal  256  that was reflected from target  190  with second reference signal  248 , which is directed over a path with a known or otherwise fixed path length, to result in a combined second target signal  264 . 
     Processor  234  receives combined first target signal  262  and combined second target signal  264  and measures a beat frequency caused by a difference in path length between each of the reflected target signals and its corresponding reference signal, and by any Doppler frequency created by target motion relative to lidar  210 . The beat frequencies may then be combined linearly to generate unambiguous determinations of range and Doppler velocity of target  190  as set forth in the Chirped Lidar Specification. In some implementations, processor  234  provides the range and Doppler velocity measurements to processing system  160 . In some implementations, processor  234  is combined with processing system  160 ; in such implementations, processing system  160  receives combined first target signal  262  and combined second target signal  264  and uses them to determine range and Doppler velocity. 
     As described, each beam  112  provides simultaneous measurements of range and Doppler velocity of a point on target  190  relative to lidar  210 . According to various implementations of the invention, various numbers of beams  112  may be used to provide these measurements of target  190 . In some implementations of the invention, two or more beams  112  may be used. In some implementations of the invention, three or more beams  112  may be used. In some implementations of the invention four or more beams  112  may be used. In some implementations of the invention, five or more beams  112  may be used. 
     In various implementations of the invention, beams  112  may be used to gather measurements for different purposes. For example, in some implementations of the invention, a particular beam  112  may be used for purposes of scanning a volume including target  190 . In some implementations of the invention, multiple beams  112  may be used to accomplish such scanning. In some implementations of the invention, a particular beam  112  may be used to monitor a particular feature or position on target  190 . In some implementations of the invention, multiple beams  112  may be used to independently monitor one or more features and/or positions on target  190 . In some implementations of the invention, one or more beams  112  may be used to scan target  190  while one or more other beams  112  may be used to monitor one or more features and/or positions on target  190 . 
     In some implementations of the invention, one or more beams  112  may scan target  190  to obtain a three dimensional image of target  190  while one or more other beams  112  may be monitoring one or more features and/or positions on target  190 . In some implementations of the invention, after a three dimensional image of target  190  is obtained, one or more beams  112  may continue scanning target  190  to monitor and/or update the motion aspects of target  190  while one or more other beams  112  may monitor one or more features and/or positions on target  110 . 
     In some implementations of the invention, measurements obtained via one or more beams  112  used to monitor and/or update the motion aspects of target  190  may be used to compensate measurements obtained via the one or more other beams  112  used to monitor one or more features and/or positions on target  190 . In these implementations of the invention, the gross motion of target  190  may be removed from the measurements associated with various features and/or positions on target  190  to obtain fine motion of particular points or regions on target  190 . In various implementations of the invention, fine motion of target  190  may include various vibrations, oscillations, or motion of certain positions on the surface of target  190  relative to, for example, a center of mass, a center of rotation, another position on the surface of target  190  or other position. In various implementations of the invention, fine motion of target  190  may include, for example, relative motion of various features such as eyes, eyelids, lips, mouth corners, facial muscles or nerves, nostrils, neck surfaces, etc. or other features of target  190 . 
     In some implementations of the invention, based on the gross motion and/or the fine motion of target  190 , one or more physiological functions and/or physical activities of target  190  may be monitored. For example, co-pending U.S. patent application Ser. No. 11/230,546, entitled “System and Method for Remotely Monitoring Physiological Functions” describes various systems and methods for monitoring physiological functions and/or physical activities of an individual and is incorporated herein by reference in its entirety. 
     In some implementations of the invention, one or more beams  112  may be used to monitor one or more locations on an eyeball of target  190  and measure various position and motion aspects of the eyeball at the each of these locations. Co-pending U.S. patent application Ser. No. 11/610,867, entitled “System and Method for Tracking Eyeball Motion” describes various systems and methods for tracking the movement of an eyeball and is incorporated herein by reference in its entirety. 
     In some implementations of the invention, one or more beams  112  may be used to focus on various features or locations on a face of target  190  and measure various aspects of the face with respect to the features or locations on the face of target  190 . For example, certain facial features or facial expressions may be monitored over a period of time to infer a mental state of target  190 , to infer an intent of target  190 , to infer a deception level of target  190  or to predict an event associated with target  190  (e.g., certain facial muscles may twitch just prior to a change in expression or prior to speech). 
     In some implementations of the invention, one or more beams  112  may be used to monitor one or more locations on a neck of target  190 . The measured motion aspects of the neck of target  190  may be used to determine throat movement patterns, vocal cord vibrations, pulse rate, and/or respiration rate. In some implementations of the invention, one or more beams  112  may be used to monitor one or more locations on an upper lip of target  190  to detect and measure vibrations associated with speech of target  190 . These vibrations may be used to substantially reproduce the speech of target  190 . 
     In some implementations of the invention, one or more beams  112  may serve one purpose during a first period or mode of operation of combined system  100  and may switch to serve a different purpose during a second period or mode of operation of combined system  100 . For example, in some implementations of the invention, multiple beams  112  may be used to measure various motion aspects of target  190  so that processing system  160  may determine or acquire a trajectory of target  190 . Once the trajectory of target  190  is acquired, some of the multiple beams  112  may switch to monitoring certain other aspects or features of target  190  while other ones of the multiple beams  112  measure motion aspects of target  190  so that its trajectory can be maintained. 
     In some implementations of the invention, five beams  112  scan target  190  to obtain a three dimensional image of target  190 . In these implementations, four of these beams  112  each scan a portion of target  190  (using various scanning patterns as described in further detail below) while a fifth beam  112  performs an “overscan” of target  190 . The overscan may be a circular, oval, elliptical or similar round scan pattern or a rectangular, square, diamond or similar scan pattern or other scan pattern useful for capturing multiple measurements of various points on target  190  (or at least points within close proximity to one another) within relatively short time intervals. These multiple measurements may correspond to other measurements made by the fifth beam  112  (i.e., multiple visits to the same point by the fifth beam  112 ) or to measurements made by one or more of the other four beams  112  (i.e., visits to the same point by the fifth beam and one or more of the other four beams  112 ). In some implementations, the pattern of the overscan may be selected to provide additional vertical and/or horizontal spread between measurements of target  190 . Both the multiple measurements and additional spread may be used to improve estimates of the motion of target  190 . Use of the fifth beam  112  to overscan target  190  may occur during each of the different modes of operation referred to above. 
     In some implementations of the invention, once the trajectory of target  190  is satisfactorily acquired, one or more beams  112  may provide measurements useful for maintaining the trajectory of target  190  as well as monitor other aspects of features of target  190 . In such implementations, other beams  112  may be used to scan for other targets in the scanning volume. 
     As illustrated in  FIG.  1   , a target coordinate frame  180  may be used to express various measurements associated with target  190 . Various coordinate frames may be used as would be appreciated. In some implementations of the invention, various ones of the subsystems  130 ,  150  may express aspects of target  190  in coordinate frames other than target coordinate frame  180  as would be appreciated. For example, in some implementations of the invention, a spherical coordinate frame (e.g., azimuth, elevation, range) may be used to express measurements obtained via lidar subsystem  130 . Also for example, in some implementations of the invention, a two dimensional pixel-based coordinate frame may be used to express images  155  obtained via video subsystem  150 . Various implementations of the invention may use one or more of these coordinate frames, or other coordinate frames, at various stages of processing as will be appreciated. 
     As would be appreciated, in some implementations of the invention, various coordinate transformations may be required to transform measurements from lidar subsystem  130 , which may be expressed in a spherical coordinates with reference to lidar subsystem  130  (sometimes referred to as a lidar measurement space), to the motion aspects of target  190 , which may be expressed in Cartesian coordinates with reference to target  190  (sometimes referred to as target space). Likewise, various coordinate transformations may be required to transform measurements from video subsystem  150 , which may be expressed in Cartesian or pixel coordinates with reference to video subsystem  150  (sometimes referred to as video measurement space), to the motion aspects of target  190 . In addition, measurements from combined system  100  may be transformed into coordinate frames associated with external measurement systems such as auxiliary video, infrared, hyperspectral, multispectral or other auxiliary imaging systems. Coordinate transformations are generally well known. 
     As would be appreciated, in some implementations of the invention, various coordinate transformations may be required to transform measurements from lidar subsystem  130  and/or video subsystem  150  to account for differences in position and/or orientation of each such subsystem  130 ,  150  as would be apparent. 
       FIG.  3    illustrates a scan pattern  300  which may be used to scan a volume for targets  190  according to various implementations of the invention. Scan pattern  300  includes a first scan pattern section  310  and a second scan pattern section  320 . First scan pattern section  310  may correspond to a scan pattern of a first beam  112  (e.g., beam  112 A) that may be used to scan the volume (or portion thereof). Second scan pattern section  320  may correspond to a scan pattern of a second beam  112  (e.g., beam  112 B) that may be used to scan the volume (or portion thereof). 
     As illustrated in  FIG.  3   , the first beam  112  scans an upper region of scan pattern  300  whereas the second beam  112  scans a lower region of scan pattern  300 . In some implementations of the invention, the scan pattern sections  310 ,  320  may include an overlap region  330 . Overlap region  330  may be used to align or “stitch together” first scan pattern section  310  with second scan pattern section  320 . In some implementations of the invention, scan patterns  310 ,  320  do not overlap to form overlap region  330  (not otherwise illustrated). 
     In implementations of the invention where lidar subsystem  130  employs a vertically displaced scan pattern  300  (such as that illustrated in  FIG.  3   ), first beam  112  is displaced vertically (i.e., by some vertical distance, angle of elevation, or other vertical displacement) from a second beam  112 . In this way, the pair of beams  112  may be scanned with a known or otherwise determinable vertical displacement. 
     While scan pattern  300  is illustrated as having vertically displaced scan pattern sections  310 ,  320  in  FIG.  3   , in some implementations of the invention, scan pattern may have horizontally displaced scan sections. In implementations of the invention where lidar subsystem  130  employs a horizontally displaced scan pattern (not otherwise illustrated), first beam  112  is displaced horizontally (i.e., by some horizontal distance, angle of azimuth, or other horizontal displacement) from second beam  112 . In this way, the pair of beams  112  may be scanned with a known or otherwise determinable horizontal displacement. 
     While  FIG.  3    illustrates a scan pattern  300  with two vertically displaced scan pattern sections  310 ,  320 , various numbers of beams may be stacked to create a corresponding number of scan pattern sections as would be appreciated. For example, three beams may be configured with either vertical displacements or horizontal displacements to provide three scan pattern sections. Other numbers of beams may be used either horizontally or vertically as would be appreciated. 
       FIG.  4    illustrates a scan pattern  400  for lidar subsystem  130  that employs four beams  112  according to various implementations of the invention. As illustrated in  FIG.  4   , lidar subsystem  130  includes four beams  112  arranged to scan a scan pattern  400 . Scan pattern  400  may be achieved by having a first pair of beams  112  displaced horizontally from one another and a second pair of beams  112  displaced horizontally from one another and vertically from the first pair of beam  112 , thereby forming a rectangular scanning arrangement. Other scanning geometries may be used as would be apparent. Scan pattern  400  may be achieved by controlling the beams independently from one another, as pairs (either horizontally or vertically), or collectively, via beam scanning mechanism(s)  140 . 
     Scan pattern  400  includes a first scan pattern section  410 , a second scan pattern section  420 , a third scan pattern section  430 , and a fourth scan pattern section  440 . In some implementations of the invention, each of the respective scan pattern sections  410 ,  420 ,  430 ,  440  may overlap an adjacent scan pattern portion by some amount (illustrated collectively in  FIG.  4    as overlap regions  450 ). For example, in some implementations of the invention, scan pattern  400  includes an overlap region  450  between first scan pattern section  410  and third scan pattern section  430 . Likewise, an overlap region  450  exists between a first scan pattern section  410  and a second scan section  420 . In some implementations of the invention, various ones of these overlap regions  450  may not occur or otherwise be utilized. In some implementations of the invention, for example, only vertical overlap regions  450  may occur or be utilized. In some implementations of the invention, only horizontal overlap regions  450  may occur or be utilized. In some implementations of the invention, no overlap regions  450  may occur or be utilized. In some implementations of the invention, other combinations of overlap regions  450  may be used. 
     As illustrated in  FIG.  3    and  FIG.  4   , the use by lidar subsystem  130  of multiple beams  112  may increase a rate at which a particular volume (or specific targets within the volume) may be scanned. For example, a given volume may be scanned twice as fast using two beams  112  as opposed to scanning the same volume with one beam  112 . Similarly, a given volume may be scanned twice as fast using four beams  112  as opposed to scanning the same volume with two beams  112 , and four times as fast as scanning the same volume with one beam  112 . In addition, multiple beams  112  may be used to measure or estimate various parameters associated with the motion of target  190  as will be discussed in more detail below. 
     According to various implementations of the invention, particular scan patterns (and their corresponding beam configurations) may be used to provide measurements and/or estimates of motion aspects of target  190 . As described above, each beam  112  may be used to simultaneously provide a range measurement and a Doppler velocity measurement at each point scanned. 
     In some implementations of the invention, for each beam  112 , a point scanned by that beam  112  may be described by an azimuth angle, an elevation angle, and a time. Each beam  112  provides a range measurement and a Doppler velocity measurement at that point and time. In some implementations of the invention, each point scanned by beam  112  may be expressed as an azimuth angle, an elevation angle, a range measurement, a Doppler velocity measurement, and a time. In some implementations of the invention, each point scanned by beam  112  may be expressed in Cartesian coordinates as a position (x, y, z), a Doppler velocity and a time. 
     According to various implementations of the invention, measurements from lidar subsystem  130  (i.e., lidar outputs  116 ) and measurements from video subsystem  150  (frames  155 ) may be used to measure and/or estimate various orientation and/or motion aspects of target  190 . These orientation and/or motion aspects of target  190  may include position, velocity, acceleration, angular position, angular velocity, angular acceleration, etc. As these orientation and/or motion aspects are measured and/or estimated, a trajectory of target  190  may be determined or otherwise approximated. In some implementations of the invention, target  190  may be considered a rigid body over a given time interval and its motion may be expressed as translational velocity components expressed in three dimensions as v x   trans , v y   trans , and v z   trans , and angular velocity components expressed in three dimensions as ω x , ω y , and ω z  over the given time interval. Collectively, these translational velocities and angular velocities correspond to six degrees of freedom of motion for target  190  over the particular time interval. In some implementations of the invention, measurements and/or estimates of these six components may be used to express a trajectory for target  190 . In some implementations of the invention, measurements and/or estimates of these six components may be used to merge the three-dimensional image of target  190  obtained from lidar subsystem  130  with the two-dimensional images of target  190  obtained from video subsystem  150  to generate three-dimensional video images of target  190 . 
     In some implementations of the invention, the instantaneous velocity component v z (t) of a point on target  190  may be calculated based on the range measurement, the Doppler velocity measurement, the azimuth angle and the elevation angle from lidar subsystem  130  as would be apparent. 
     Lidar subsystem  130  may be used to measure and/or estimate translational velocity v z   trans  and two angular velocities of target  190 , namely ω x  and ω y . For example,  FIG.  5    illustrates an exemplary relationship between points with corresponding measurements from two beams  112  that may be used to estimate x and y components of angular velocity of target  190  according to various implementations of the invention. More particularly, and generally speaking, as illustrated in  FIG.  5   , in implementations where beams  112  are displaced from one another along the y-axis, a local velocity along the z-axis of point P A  determined via a first beam  112 , a velocity of point P B  determined via a second beam  112 , and a distance between P A  and P B  may be used to estimate an angular velocity of these points about the x-axis (referred to herein as ω x ) as would be appreciated. In some implementations of the invention, these measurements may be used to provide an initial estimate of ω x . 
       FIG.  6    illustrates another exemplary relationship between points with corresponding measurements from two beams  112  that may be used to estimate an angular velocity according to various implementations of the invention. More particularly, as illustrated in  FIG.  6   , in implementations were beams  112  are displaced from one another on target  190  along the x-axis, a velocity of point P A  determined via a first beam  112 , a velocity of point P B  determined by a second beam  112 , and a distance between P A  and P B  on target  190  along the x-axis may be used to estimate an angular velocity of these points about the y-axis (referred to herein as ω y ). In some implementations of the invention, these measurements may be used to provide an initial estimate of ω y . 
       FIG.  5    and  FIG.  6    illustrate implementations of the invention where two beams  112  are disposed from one another along a vertical axis or a horizontal axis, respectively, and the corresponding range (which may be expressed in three-dimensional coordinates x, y, and z) and Doppler velocity at each point are measured at substantially the same time. In implementations of the invention that employ beams  112  along a single axis (not otherwise illustrated), an angular velocity may be estimated based on Doppler velocities measured at different points at different times along the single axis. As would be appreciated, better estimates of angular velocity may obtained using: 1) measurements at points at the extents of target  190  (i.e., at larger distances from one another), and 2) measurements taken within the smallest time interval (so as to minimize any effects due to acceleration). 
       FIG.  5    and  FIG.  6    illustrate conceptual estimation of the angular velocities about different axes, namely the x-axis and the y-axis. In general terms, where a first beam  112  is displaced on target  190  along a first axis from a second beam  112 , an angular velocity about a second axis orthogonal to the first axis may be determined from the velocities along a third axis orthogonal to both the first and second axes at each of the respective points. 
     In some implementations of the invention, where two beams are displaced along the y-axis from one another (i.e., displaced vertically) and scanned horizontally with vertical separation between scans, estimates of both ω x  and ω y  may be made. While simultaneous measurements along the x-axis are not available, they should be sufficiently close in time in various implementations to neglect acceleration effects. In some implementations of the invention where two beams  112  are displaced along the x-axis from one another and at least a third beam  112  is displaced along the y-axis from the pair of beams  112 , estimates of ω x , ω y  and v z   trans  may be made. In some implementations of the invention, estimates of both ω x , ω y  and v z   trans  may be made using four beams  112  arranged in a rectangular fashion. In such implementations, the measurements obtained from the four beams  112  include more information than necessary to estimate ω x , ω y  and v z   trans . This so-called “overdetermined system” may be used to improve the estimates of ω x , ω y  and V z   trans  as would be appreciated. 
     As has been described, range and Doppler velocity measurements taken at various azimuth and elevation angles and at various points in time by lidar subsystem  130  may be used to estimate translational velocity v z   trans  and estimate two angular velocities, namely, ω x  and ω y , for the rigid body undergoing ballistic motion. 
     In some implementations of the invention, ω x , ω y  and v z   trans  may be determined at each measurement time from the measurements obtained at various points as would be appreciated. In some implementations of the invention, ω x , ω y  and v z   trans  may be assumed to be constant over an particular interval of time. In some implementations of the invention, ω x , ω y  and v z   trans  may be determined at various measurement times and subsequently averaged over a particular interval of time to provide estimates of ω x , ω y  and v z   trans  for that particular interval of time as would be appreciated. In some implementations of the invention, the particular time interval may be fixed or variable depending, for example, on the motion aspects of target  190 . In some implementations of the invention, a least squares estimator may be used to provide estimates of ω x , ω y  and v z   trans  over a particular interval of time as would be appreciated. Estimates of ω x , ω y  and v z   trans  may be obtained in other manners as would be appreciated. 
     In some implementations of the invention, images from video subsystem  150  may be used to estimate three other motion aspects of target  190 , namely translational velocity components v x   trans  and v y   trans  and angular velocity component ω z  over a given interval of time. In some implementations of the invention, frames  155  captured by video subsystem  150  may be used to estimate x and y components of velocity for points on target  190  as it moves between frames  155 .  FIG.  7    illustrates a change in position of a particular point or feature I A  between a frame  155  at time T and a frame  155  at subsequent time T+Δt. 
     In some implementations of the invention, this change of position is determined for each of at least two particular points or features in frame  155  (not otherwise illustrated). In some implementations of the invention, the change of position is determined for each of many points or features. In some implementations of the invention, translational velocity components v x   trans  and v y   trans , and angular velocity component ω z  of target  190  may be estimated based on a difference in position of a feature I A (T) and I A (T+Δt) and a difference in time, Δt, between the frames  155 . These differences in position and time may be used to determine certain velocities of the feature, namely, v x   feat  and v y   feat  that may in turn be used to estimate the translational velocity components v x   trans  and v y   trans , and angular velocity component ω z  of target  190 . Such estimations of velocity and angular velocity of features between image frames are generally understood as would be appreciated. 
     In some implementations of the invention, many features of target  190  are extracted from consecutive frames  155 . The velocities v x   feat  and v y   feat  of these features over the time interval between consecutive frames  155  may be determined based on changes in position of each respective feature between the consecutive frames  155 . A least squares estimator may be used to estimate the translational velocities v x   trans  and v y   trans , and the angular velocity ω z  from the position changes of each the extracted features. 
     In some implementations of the invention, a least squares estimator may use measurements from lidar subsystem  130  and the changes in position of the features in frames  155  from video subsystem  150  to estimate the translational velocities v x   trans , v y   trans  and v z   trans  and the angular velocities ω x , ω y , and ω z  of target  190 . 
     As has been described above, lidar subsystem  130  and video subsystem  150  may be used to estimate six components that may be used describe the motion of target  190 . These components of motion may be collected over time to calculate a trajectory of target  190 . This trajectory may then be used to compensate for motion of target  190  to obtain a motion stabilized three dimensional image of target  190 . In various implementations of the invention, the trajectory of target  190  may be assumed to represent ballistic motion over various intervals of time. The more accurately trajectories of target  190  may be determined, the more accurately combined system  100  may adjust the measurements of target  190  to, for example, represent three dimensional images, or other aspects, of target  190 . 
     In various implementations of the invention, a rate at which measurements are taken by lidar subsystem  130  is different from a rate at which frames  155  are captured by video subsystem  150 . In some implementations of the invention, a rate at which measurements are taken by lidar subsystem  130  is substantially higher than a rate at which frames  155  are captured by video subsystem  150 . In addition, because beams  112  are scanned through a scan volume by lidar subsystem  130 , measurements at different points in the scan volume may be taken at different times from one another; whereas pixels in a given frame  155  are captured substantially simultaneously (within the context of video imaging). In some implementations of the invention, these time differences are resolved in order to provide a more accurate trajectory of target  190 . 
     As illustrated in  FIG.  8   , in some implementations of the invention, a scan pattern  840  may be used to scan a volume for targets. For purposes of explanation, scan pattern  840  represents a pattern of measurements taken by a single beam. In some implementations multiple beams may be used, each with their corresponding scan pattern as would be apparent. As illustrated, scan pattern  840  includes individual points  810  measured left to right in azimuth at a first elevation  831 , right to left in azimuth at a second elevation  832 , left to right in azimuth at a third elevation  833 , etc., until a particular scan volume is scanned. In some implementations, scan pattern  840  may be divided into intervals corresponding to various timing aspects associated with combined system  100 . For example, in some implementations of the invention, scan pattern  840  may be divided into time intervals associated with a frame rate of video subsystem  150 . In some implementations of the invention, scan pattern  840  may be divided into time intervals associated with scanning a particular elevation (i.e., an entire left-to-right or right-to-left scan). In some implementations of the invention, scan pattern  840  may be divided into time intervals associated with a roundtrip scan  820  (illustrated in  FIG.  8    as a roundtrip scan  820 A, a roundtrip scan  820 B, and a roundtrip scan  820 C) at one or more elevations (i.e., a left-to-right and a return right-to-left scan at either the same or different elevations). Similar timing aspects may be used in implementations that scan vertically in elevation (as opposed to horizontally in azimuth). Other timing aspects may be used as well. 
     As illustrated in  FIG.  8    and again for purposes of explanation, each interval may include N points  810  which may in turn correspond to the number of points  810  in a single scan (e.g.,  831 ,  832 ,  833 , etc.) or in a roundtrip scan  820 . A collection of points  810  for a particular interval is referred to herein as a sub-point cloud and a collection of points  810  for a complete scan pattern  840  is referred to herein as a point cloud. In some implementations of the invention, each point  810  corresponds to the lidar measurements of range and Doppler velocity at a particular azimuth, elevation, and a time at which the measurement was taken. In some implementations of the invention, each point  810  corresponds to the lidar measurements of range (expressed x, y, z coordinates) and Doppler velocity and a time at which the measurement was taken. 
       FIG.  9    illustrates a timing diagram  900  useful for describing various timing aspects associated with measurements from lidar subsystem  130  according to various implementations of the invention. Timing diagram  900  includes points  810  scanned by beam  112 , sub-point clouds  920  formed from a plurality of points  810  collected over an interval corresponding to a respective sub-point cloud  920 , and a point cloud  930  formed from a plurality of sub-point clouds  920  collected over the scan pattern. Timing diagram  900  may be extended to encompass points  810  scanned by multiple beams  112  as would be appreciated. 
     Each point  810  is scanned by a beam  112  and measurements associated with each point  810  are determined by lidar subsystem  130 . In some implementations of the invention, points  810  are scanned via a scan pattern (or scan pattern section). The interval during which lidar subsystem  130  collects measurements for a particular sub-point cloud  920  may have a time duration referred to as T SPC . In some implementations of the invention, the differences in timing of the measurements associated with individual points  810  in sub-point cloud  920  may be accommodated by using the motion aspects (e.g., translational velocities and angular velocities) for each point to adjust that point to a particular reference time for sub-point cloud  920  (e.g., t RSPC ). This process may be referred to as stabilizing the individual points  810  for the motion aspects of target  190 . 
     In some implementations of the invention, the velocities may be assumed to be constant over the time interval (i.e., during the time duration T SPC ). In some implementations of the invention, the velocities may not be assumed to be constant during the period of the scan pattern and acceleration effects may need to be considered to adjust the measurements of points  810  to the reference time as would be appreciated. In some implementations of the invention, adjustments due to subdivision of the time interval may also need to be accommodated. As illustrated in  FIG.  9   , the reference time for each sub-point cloud  920  may be selected at the midpoint of the interval, although other reference times may be used. 
     In some implementations of the invention, similar adjustments may be made when combining sub-point clouds  920  into point clouds  930 . More particularly, in some implementations of the invention, the differences in timing of the measurements associated with sub-point clouds  920  in point cloud  930  may be accommodated by using the motion aspects associated with the measurements. 
     In some implementations of the invention, the measurements associated with each sub-point cloud  920  that is merged into point cloud  930  are individually adjusted to a reference time associated with point cloud  930 . In some implementations of the invention, the reference time corresponds to a frame time (e.g., time associated with a frame  155 ). In other implementations of the invention, the reference time correspond to an earliest of the measurement times of points  1110  in point cloud  930 , a latest of the measurement times of points  1110  in point cloud  930 , an average or midpoint of the measurement times of points  1110  in point cloud  930 , or other reference time associated with point cloud  930 . 
     Although not otherwise illustrated, in some implementations of the invention, similar adjustments may be made to combine point clouds  930  from individual beams  112  into aggregate point clouds at a particular reference time. In some implementations of the invention, this may be accomplished at the individual point level, the sub-point cloud level or the point cloud level as would be appreciated. For purposes of the remainder of this description, sub-point clouds  920  and point clouds  930  refer to the collection of points  810  at their respective reference times from each of beams  112  employed by lidar subsystem  130  to scan target  190 . 
     In some implementations of the invention, motion aspects of target  190  may be assumed to be constant over various time intervals. For example, motion aspects of target  190  may be assumed to be constant over T SPC  or other time duration. In some implementations of the invention, motion aspects of target  190  may be assumed to be constant over a given T SPC , but not necessarily constant over T PC . In some implementations of the invention, motion aspects of target  190  may be assumed to be constant over incremental portions of T SPC , but not necessarily over the entire T SPC . As a result, in some implementations of the invention, a trajectory of target  190  may be expressed as a piece-wise function of time, with each “piece” corresponding to the motion aspects of target  190  over each individual time interval. 
     In some implementations, timing adjustments to compensate for motion may be expressed as a transformation that accounts for the motion of a point from a first time to a second time. This transformation, when applied to measurements from, for example, lidar subsystem  130 , may perform the timing adjustment from the measurement time associated with a particular point (or sub-point cloud or point cloud, etc.) to the desired reference time. Furthermore, when the measurements are expressed as vectors, this transformation may be expressed as a transformation matrix. Such transformation matrices and their properties are generally well known. 
     As would be appreciated, the transformation matrices may be readily used to place a position and orientation vector for a point at any time to a corresponding position and orientation vector for that point at any other time, either forwards or backwards in time, based on the motion of target  190 . The transformation matrices may be applied to sub-point clouds, multiple sub-point clouds and point clouds as well. In some implementations, a transformation matrix may be determined for each interval (or subinterval) such that it may be used to adjust a point cloud expressed in one interval to a point cloud expressed in the next sequential interval. In these implementations, each interval has a transformation matrix associated therewith for adjusting the point clouds for the trajectory of target  190  to the next interval. In some implementations, a transformation matrix may be determined for each interval (or subinterval) such that it may be used to adjust a point cloud expressed in one interval to a point cloud expressed in the prior sequential interval. Using the transformation matrices for various intervals, a point cloud can be referenced to any time, either forward or backward. 
       FIG.  10    illustrates a timing diagram  1000  useful for describing various timing aspects associated with measurements from lidar subsystem  130  in relation to measurements from video subsystem  150  according to various implementations of the invention. In some implementations of the invention, point cloud  930  may be referenced to the midpoint of a time interval between frames  155  or other time between frames  155 . In some implementations of the invention, point cloud  930  may be referenced to a frame time corresponding to a particular frame  155 . Point cloud  930  may be referenced in other manners relative to a particular frame  155  as would be appreciated. 
     As illustrated in  FIG.  10   , PC m−1  is the expression of point cloud  930  referenced at the frame time of frame I n−1 ; PC m  is the expression of point cloud  930  referenced at the frame time of frame I n ; and PC m+1  is the expression of point cloud  930  referenced at the frame time of frame I n+1 ; and PC m+2  is the expression of point cloud  930  referenced at the frame time of frame I n+2 . In some implementations, point cloud  930  may be referenced at other times in relation to the frames and frames times as would be apparent. 
     As described above, a transformation matrix T i,i+1  may be determined to transform an expression of point cloud  930  at the i th  frame time to an expression of point cloud  930  at the (i+1) th  frame time. In reference to  FIG.  10   , a transformation matrix T m−1,m  may be used to transform PC m−1  to PC m ; a transformation matrix T m,m+1  may be used to transform PC m  to PC m+1 ; and a transformation matrix T m+1,m+2  may be used to transform PC m+1  to PC m+2 . In this way, transformation matrices may be used to express point clouds  930  at different times corresponding to frames  155 . 
     According to various implementations of the invention, the transformation matrices which are applied to point cloud  930  to express point cloud  930  from a first time to a second time are determined in different processing stages. Generally speaking, transformation matrices are directly related with six degree of motion parameters ω x , ω y , ω z , v x   trans , v y   trans , and v z   trans  that may be calculated in two steps: first ω x , ω y , and v z   trans  from lidar subsystem and second v x   trans , v y   trans , and ω z , from video subsystem. 
       FIG.  11    illustrates a block diagram of a configuration of processing system  160  that may be used during a first phase of the first processing stage to estimate a trajectory of target  190  according to various implementations of the invention. In some implementations of the invention, during the first phase of the first stage, a series of initial transformation matrices (referred to herein as T i,i+1   (0) ) are determined from various estimates of the motion aspects of target  190 . As illustrated, lidar subsystem  130  provides range, Doppler velocity, azimuth, elevation and time for at each point as input to a least squares estimator  1110  that is configured to estimate angular velocities ω x  and ω y  and translational velocity v z   trans  over each of a series of time intervals. In some implementations of the invention, angular velocities ω x  and ω y  and translational velocity v z   trans  are iteratively estimated by varying the size of the time intervals (or breaking the time intervals into subintervals) as discussed above until any residual errors from least squares estimator  1110  for each particular time interval reach acceptable levels as would be apparent. This process may be repeated for each successive time interval during the time measurements of target  190  are taken by lidar subsystem  130 . 
     Assuming that target  190  can be represented over a given time interval as a rigid body (i.e., points on the surface of target  190  remain fixed with respect to one another) undergoing ballistic motion (i.e., constant velocity with no acceleration), an instantaneous velocity of any given point  810  on target  190  can be expressed as:
 
 v=v   trans +[ω×( R−R   c   −v   trans   *Δt )]  Eq. (1)
 
where
         v is the instantaneous velocity vector of the given point;   v trans  is the translational velocity vector of the rigid body;   ω is the rotational velocity vector of the rigid body;   R is the position of the given point on the target;   R c  is the center of rotation for the target; and   Δt is the time difference of each measurement time from a given reference time.       

     Given the measurements available from lidar subsystem  130 , the z-component of the instantaneous velocity may be expressed as:
 
 v   z   =v   z   trans +[ω×( R−R   c   −v   trans   *Δt )] z   Eq. (2)
 
where
         v z  is the z-component of the instantaneous velocity vector;   v z   trans  is the z-component of the translational velocity vector; and   [ω×(R−R c −V trans *Δt)] z  is the z-component of the cross product.       

     In some implementations of the invention, frame-to-frame measurements corresponding to various features from images  155  may be made. These measurements may correspond to a position (e.g., x feat , y feat )) and a velocity (e.g., v x   feat , v y   feat ) for each of the features and for each frame-to-frame time interval. In implementations where a z-coordinate of position is not available from video subsystem  150 , an initial estimate of z may be made using, for example, an average z component from the points from lidar subsystem  130 . Least squares estimator  1120  estimates angular velocities ω x , ω y , and ω z  and translational velocities v x   trans , v y   trans , and v z   trans  which may be expressed as a transformation matrix T i,i+1   (0)  for each of the relevant time intervals. In some implementations of the invention, a cumulative transformation matrix corresponding to the arbitrary frame to frame time interval may be determined. 
       FIG.  12    illustrates a block diagram of a configuration of processing system  160  that may be used during a second phase of the first processing stage to estimate a trajectory of target  190  according to various implementations of the invention. In some implementations of the invention, during the second phase of the first stage, new transformation matrices (referred to herein as T i,i+1   (1) ) are determined from various estimates of the motion aspects of target  190 . As illustrated, measurements from lidar subsystem  130  of range, Doppler velocity, azimuth, elevation and time for at each of the N points are input to a least squares estimator  1110  of processing system  160  along with the transformation matrices T i,i+1   (0)  to estimate angular velocities ω x  and ω y  and translational velocity v z   trans  over each of a series of time intervals in a manner similar to that described above during the first phase. 
     The primary difference between the second phase and the first phase is that least squares estimator  1120  uses the calculated z position of the features based on T i,i+1   (0)  as opposed to merely an average of z position. Least squares estimator  1120  estimates new angular velocities ω x , ω y , and ω z  and new translational velocities v x   trans , v y   trans , and v z   trans  which may be expressed as a transformation matrix T i+i+1   (1)  for each of the relevant time intervals. Again, in some implementations of the invention, a cumulative transformation matrix corresponding to the frame to frame time interval may be determined. 
       FIG.  13    illustrates a block diagram of a configuration of processing system  160  that may be used during a third phase of the first processing stage to estimate a trajectory of target  190  according to various implementations of the invention. In some implementations of the invention, during the third phase of the first stage, new transformation matrices (referred to herein as T i,i+1   (2) ) are determined from various estimates of the motion aspects of target  190 . As illustrated, lidar subsystem  130  provides range, Doppler velocity, azimuth, elevation and time for at each of the points as input to a least squares estimator  1110  to estimate angular velocities ω x  and ω y  and translational velocity v z   trans  over each of a series of time intervals in a manner similar to that described above during the first phase. In this phase, calculated values of v x  and v y  for each point based on T i,i+1   (1)  as determined during the prior phase are input into least squares estimator  1120  as opposed to the feature measurements used above. 
     The primary difference between the third phase and the second phase is that least squares estimator  1120  uses T i,i+1   (1)  to describe motion between the relevant frames  155 . Least squares estimators  1110 ,  1120  estimate new angular velocities ω x , ω y , and ω z  and new translational velocities v x   trans , v y   trans , and v z   trans  which may be expressed as a transformation matrix T i,i+1   (2)  for each of the relevant time intervals. Again, in some implementations of the invention, a cumulative transformation matrix corresponding to the frame to frame time interval may be determined. 
     In various implementations of the invention, any of the phases of the first processing stage may be iterated any number of times as additional information is gained regarding motion of target  190 . For example, as the transformation matrices are improved, each point  810  may be better expressed at a given reference time in relation to its measurement time. 
     During the first processing stage, the translational velocities of each point (not otherwise available from the lidar measurements) may be estimated using features from the frames  155 . Once all velocity components are known or estimated for each point, transformation matrices may be determined without using the feature measurements as illustrated in  FIG.  13   . 
       FIG.  14    illustrates a block diagram of a configuration of processing system  160  that may be used during a first phase of the second processing stage to refine a trajectory of target  190  according to various implementations of the invention. The first processing stage provides transformation matrices sufficient to enable images  155  to be mapped onto images at any of the frame times. Once so mapped, differences in pixels themselves (as opposed to features in images  155 ) from different images transformed to the same frame time may be used to further refine the trajectory of target  190 . In some implementations, various multi-frame corrections may be determined, which in turn through the least square estimator can be used for obtaining offsets between the consecutive images Δx i,i+1 , Δy i,i+1 , Δθz i,i+1 . These corrections may be used to refine the transformation matrices (for example, matrices T i,i+1   (2)  to T i,i+1   (3) ). In some implementations of the invention, during the first phase of the second processing stage, a series of new transformation matrices (referred to herein as T i,i+1   (3) ) are refinements of T i,i+1   (2)  on the basis of the offsets between image I i  and an image I j , namely, Δx i,j , Δy i,j , Δθz i,j . As illustrated, an estimator  1410  determines a difference between an image L and an image L using the appropriate transformation matrix T i,j   (2)  to express the images at the same frame time. 
       FIG.  15    illustrates a block diagram of a configuration of processing system  160  that may be used during a second phase of the second processing stage to further refine a trajectory of target  190  according to various implementations of the invention. To the extent that additional accuracy is necessary, the transformation matrices from the first phase of the second processing stage (e.g., T i,i+1   (3) ) may be used in connection with the measurements from lidar subsystem  130  to further refine the transformation matrices (referred to herein as T i,i+1   (4) ). In some implementations of the invention, during this phase, measurements from lidar subsystem  130  that occur within any overlap regions  360 ,  450  are used. These measurements correspond to multiple measurements taken for the same point (or substantially the same point) at different times. This phase is based on the premise that coordinate measurements of the same point on target  190  taken different times should transform precisely to one another with a perfect transformation matrix, i. e. points measured at the different times that have the same x and y coordinates should have the same z coordinate. In other words, all coordinates of the underlying points should directly map to one another at the different points in time. The differences in z coordinates (i.e., corrections) between the measurements at different times can be extracted and input to the least square estimator. Through the least square estimator, the corrections to the transformation parameters between the consecutive frames may obtained and may be expressed as Δz i,i+1 , Δθx i,i+1 , and Δθy i,i+1 . These corrections may be used to refine the the transformation matrices T i,i+1   (3)  to T i,i+1   (4) . In some implementations of the invention, multiple measurements corresponding to use of an overscan beam may be used in a similar manner. 
     As described herein, obtaining accurate transformation parameters allows on one hand the six degrees-of-freedom (“6DOF”) tracking of targets, and on the other hand generation of the motion compensated 3D image of the target that can be reproduced at any frame time during the scanning. 
     Stochastic errors present in each set of transformation parameters may be accumulated upon successive transformations. Such stochastic errors in the transformation parameters is due to the stochastic error of the respective video and lidar measurements. 
     According to various implementations of the invention, the 3D images may be refined by reducing the stochastic components in the transformation parameters (i.e., v x , v y , v z , ω x , ω y  and ω z ) between video frame times. In some implementations, the 3D images may be refined on the basis of matching the transformed coordinates and pixels positions. In other words, the stochastic component for the set of transformation parameters may be reduced, which ultimately leads to the 3D image refinement. As described with respect to  FIG.  11    above, during the first processing stage, the transformation parameters may be calculated self-consistently from the Doppler velocity measurements made by the lidar subsystem  130  (e.g., v z , ω x , ω y  and referred to as the lidar transformation parameters or lidar parameters) and the measurements obtained from the series of 2D video images from video subsystem  150  (e.g., v x , v y , ω z  and referred to as the video transformation parameters or video parameters). In some implementations, the processing system  160  may refine the transformation parameters by first, refining the lidar parameters and then refining the video parameters, although the order here is not important. 
     In some implementations, refinement of the lidar parameters may be based on the assumption that—for the motion compensated 3D image transformed to a particular frame time, the z offset of the subsequent overscan measurements (from a fifth beam  112 , for example) from the line scan measurements (from one or more of the four beams using various scanning patterns described herein) should be the same. Refinement of the lidar parameters may be accomplished by transforming the stationary image (e.g., point cloud) to a frame time associated with the frames used during the first stage. Next, the overscan lidar measurements taken during the inter-frame time intervals immediately before a given frame time and immediately after the given frame time may be gathered/selected/identified for each of the frames, and may be referred to respectively as pre and post over-scan measurements. By definition, these pre and post over-scan measurements cross the multiple line scans of the other beams (i.e., beams other than that/those used for the overscan) in the scan pattern. For each scan line and at each frame time, processing system  160  may determine Δz offsets between the points (i.e., lidar measurements) in the line scan relative to each of the pre over-scan measurements (i.e., also “points”) and the post over-scan measurements. If the stochastic measurement errors were “0” and the transformation parameters ideal, the Δz offsets (both for pre and post overscan measurements) would be zero. However, these Δz offsets are typically not zero. In some implementations of the invention, when the lidar transformation parameters may be updated by imposing the condition that pre and post over-scan measurements should be offset from the same line scan measurements by the same amount. To correct the lidar transformation parameters, processing system  160  may use a least-squares optimization to minimize the difference between the Δz offsets relative to pre and post over-scan measurements over the respective set of frame times as would be appreciated. This results in a set of equations where all inter-frame transformation parameters are inter-related, such as the same subset of parameters (except for the first and last ones) enter explicitly in one equation from the pre over-scan measurements and in another equation as the post over-scan measurements. This process may be applied to all overscan measurements at their respective times as would be appreciated. In addition, in some implementations, this procedure can be iterated, since in general the least square optimization favors the iterations. 
     In some implementations, processing system  160  may refine the video transformation parameters utilizing similar principles. Using the transformation parameters, not only can any 3D point be transformed to any frame time, but any video image (more appropriately, any pixel in any image) may be transformed to any frame time (and any other time as well). For video images, if the measurement errors were “0” and the transformation parameters ideal, a video image transformed to a different frame time should coincide with the video image originally captured at that frame time (i.e., the only untransformed video image at that frame time). By comparing pixel intensity of the transformed video image(s) and the original image at a given frame time, corrections to the transformation parameters may be derived. In some implementations of the invention, the video image at frame time n, the video image at frame time n−1 transformed to frame time n (the “transformed pre-video image”) and the video image at frame time n+1 transformed to frame time n (the “transformed post-video image”) may be compared to one another. In other words, three video images captured at consecutive frame times n−1, n, and n+1, respectively, are each transformed to frame time n (clearly, no transformation of the original video image at frame time n is necessary) using the transformation parameters. Then, the pixel intensities of the transformed pre-video image may be compared with those of the video image, the pixel intensities of the video image may be compared with those of the transformed post-video image, and the pixel intensities of the transformed pre-video image may be compared with those of the transformed post-video image. The first combination determines the correction to the video transformation parameters for the frame interval between n−1 and n (i.e., dv x   (n-1) , dv y   (n−1) , dω z   (n-1) ), the second combination determines the correction to the video transformation parameters for the frame interval between n and n+1 (dv x   (n) , dv y   (n) , dω z   (n) ) and the third combination determines a correction to the video transformation parameters of both intervals. Thus, for each image frame time the pre and post frame time corrections may be determined, making it an inter-related system, which again may be solved using the least square optimization. While described above as a single frame time n approach, this process may be generalized to any number of the video images and corresponding frame times as would be appreciated. However, in some applications/implementations, a tradeoff may exist between increasing number of frames used and changing light conditions (e.g., rapid changes in lighting may favor fewer number of frames for the optimization whereas stable lighting conditions may favor more number of frames for the optimization). 
     The following equations set forth the refinement process described above in accordance with various implementations of the invention.
 
 T   ij   I   j   =I   i   (j)  
 
 T   ik   I   j   =I   i   (k)  
 
Where
 
     T ij  is the initial transformation that transforms from frame i to frame j, I i  and I j  are the original images i and j, I i   (j)  is the original image j transformed to the time frame of image i. In the absence of errors, I i   (j) =I i   (k) . The transformation correction DT jk   (i)  that matches two transformed images may be defined as:
 
 DT   jk   (i)   I   i   (k)   =I   i   (j)   Eq. (3)
 
     Then, the corrected transformation may be:
 
 T   jk   (c)   =T   ij   −1   DT   jk   (i)T   ik .  Eq. (4)
 
     An equation relating two consecutive transformation corrections may have the form:
 
 DT   jm   (p)   =T   pi   DT   jk   (i)   T   is   DT   km   (s)   T   sp   . Eq. (5)
 
     If all transformation corrections are taken at the same frame time, p=i=s, equation (5) may be simplified as:
 
 DT   jm   (p)   =DT   jk   (p)   DT   km   (p)   . Eq. (6)
 
     Equation (6) is the fundamental equation for multi-frame image refinement. Equation (6) shows that operating in the same frame time and in the linear approximation the corrections become additive. Particularly, it shows that dx, dy, and dθz are related through the multi-frame equation as:
 
 dx   (i)   i−1,i+1   =dx   (i)   i−1,i   +dx   (i)   i,i+1 ,  Eq. (7)
 
 dy   (i)   i−1,i+1   =dy   (i)   i−1,i   +dy   (i)   i,i+1 ,  Eq. (8)
 
 dθ   z   (i)   i−1,i+1   =dθ   z   (i)   i−1,   i+dθ   z   (i)   i,i+1 .  Eq. (9)
 
     In the single frame approach, the corrected transformation T i,i+1   (c)  can be calculated either through the i th  or (i th +1) reference frame:
 
 T   i,i+1   (c)   =T   i,i+1   DT   i,i+1   (i+1)   =DT   i,i+1   (i)   T   i,i+1   Eq. (10)
 
     For practical purposes, the geometric mean may be used as follows:
 
 T   i,i+1   (c) =( T   i,i+1   DT   i,i+1   (i+1)   DT   i,i+1   (i)   T   i,i+1 ) 1/2 ,  Eq. (11)
 
     which accounts for both forward and backward correction to the image transformation. 
     The forward and backward corrections may be determined from the least-square minimization. The forward and backward corrections may be defined as the ones that have their own multi-frame correction term:
 
 dx   i   (F)   =dx   (i)   i−1,i +α i   (F) ,  Eq. (12)
 
 dx   i   (B)   =dx   (i)   i,i+1 +α i   (B) ,  Eq. (13)
 
 dx   i   (2)   =dx   (i)   i−1,i+1 +α i   (2) ,  Eq. (14)
 
which obey to the multi-frame equation:
 
 dx   i   (2)   =dx   i   (F)   +dx   i   (B) .  Eq. (15)
 
     Minimizing the functional of the multi-frame corrections:
 
 X =Σ[(α (F) ) 2 +(α (B) ) 2 +(α (2) ) 2 ],  Eq. (16)
 
     the least-square multi-frame corrections may be obtained as follows:
 
α i   (F) =α i   (B)   =dx   (i)   i−1,i+1   −dx   (i)   i−1,i   −dx   (i)   i,i+1 )/3.  Eq. (17)
 
     In the similar way, in such formulation the multi-frame corrections for dy and dθ z  may be obtained. The multi-frame corrections (17) define the multi-frame lateral and angular offsets (10-12), which in turn defines the transformation correction: 
     
       
         
           
             
               
                 
                   
                     
                       ( 
                       141 
                       ) 
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     DT 
                   
                   = 
                   
                     
                       
                         1 
                       
                       
                         
                           
                             - 
                             d 
                           
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           θ 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           z 
                         
                       
                       
                         0 
                       
                       
                         dx 
                       
                     
                     
                       
                         
                           d 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           θ 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           z 
                         
                       
                       
                         1 
                       
                       
                         0 
                       
                       
                         dy 
                       
                     
                     
                       
                         0 
                       
                       
                         0 
                       
                       
                         1 
                       
                       
                         0 
                       
                     
                     
                       
                         0 
                       
                       
                         0 
                       
                       
                         0 
                       
                       
                         1 
                       
                     
                   
                 
               
               
                 
                   Eq 
                   . 
                   
                       
                   
                   ⁢ 
                   
                     ( 
                     18 
                     ) 
                   
                 
               
             
           
         
       
     
     This via Eq. (11) produces the corrected transformation matrix. 
       FIG.  16    illustrates a flowchart depicting example operations performed by a system for refining a three dimensional image, according to various aspects of the invention. In some implementations, the described operations may be accomplished using one or more of the components described herein. In some implementations, various operations may be performed in different sequences. In other implementations, additional operations may be performed along with some or all of the operations shown in  FIG.  16   . In yet other implementations, one or more operations may be performed simultaneously. In yet other implementations, one or more operations may not be performed. Accordingly, the operations described in  FIG.  16    and other drawing figures are exemplary in nature and, as such, should not be viewed as limiting. 
     In some implementations, in an operation  1602 , process  1600  may receive a first set of line scan 3D measurements and a second set of overscan 3D measurements (from lidar subsystem  130 , for example) for a plurality of points on a target, and at least one video frame of the target (from video subsystem  150 , for example). 
     In some implementations, in an operation  1604 , process  1600  may determine, at each frame time, the multiple Δz offsets between the pre over-scan measurements and the line scans (referred to as pre Δz offsets) and the post over-scan measurements and the line scans (referred to as post Δz offsets). 
     In some implementations, in an operation  1606 , process  1600  may adjust or update the lidar transformation parameters based on determined pre and post Δz offsets by using the least square optimization. 
       FIG.  17    illustrates a flowchart depicting example operations performed by a system for refining a three dimensional image, according to various aspects of the invention. In some implementations, in an operation  1702 , process  1700  may receive a plurality of 3D measurements (from lidar subsystem  130 , for example) for a plurality of points on a target, and the video frames of the target (from video subsystem  150 , for example). 
     In some implementations, in an operation  1704 , process  1700  may transform a video frame to a particular frame time to generate transformed video frame (for example, a video frame at frame time n−1 or n+1 transformed to frame time n). In some implementations, process  1700  may transform each video frame to a next and previous video frame. In some implementations, in an operation  1706 , process  1700  may compare the pixel intensities of the transformed video frames and determine the offsets between the transformed video frames. In some implementations, in an operation  1708 , process  1700  may determine corrections to the video transformation parameters on the basis of the multi-frame least square optimization and may update the transformation parameters. 
     While the invention has been described herein in terms of various implementations, it is not so limited and is limited only by the scope of the following claims, as would be apparent to one skilled in the art. These and other implementations of the invention will become apparent upon consideration of the disclosure provided above and the accompanying figures. In addition, various components and features described with respect to one implementation of the invention may be used in other implementations as well.